Information processing system

Abstract

The purpose of the present invention is to provide an information processing system that automatically cleans annotation data necessary for deep learning. An information processing system that executes a cleaning process of a data set, which is a set including data to be classified and annotation data of a class corresponding to the data to be classified, is provided with: a learning processing unit that generates a model by executing a learning process of deep learning using the data set; and a cleaning processing unit that performs cleaning processing of annotation data in the data set, the cleaning processing unit outputting a first class corresponding to data to be classified in first annotation data of the first data set, inputting the data to be classified into a first model generated by the learning processing unit using the first data set, and outputting a second class, a cleaning process of the first annotation data in the first data set is performed by comparing the first class with the second class.

Description

Technical Field

[0001] The present invention relates to an information processing system for automatically cleaning annotation data required for deep learning. Background Art

[0002] In recent years, a data processing method called deep learning has been used for various purposes. For example, it is used to determine the type of object shown in image data. As an example, it is used for the following purposes: by inputting image data obtained by photographing a display shelf of a store into a model based on deep learning (learned model, learned network), the goods displayed on the display shelf shown in the image data are identified. In addition, it is used for various other purposes.

[0003] Deep learning has attracted much attention in recent years due to its ability to classify various types of data.

[0004] On the other hand, when using deep learning, it is necessary to use annotation data in advance to learn the model. Annotation data is data that shows which class (category) a certain data is classified into. For example, it is necessary to show that the class of the image data of "dog" is "dog". In this way, the situation of associating a certain data with the class of the data is called annotation.

[0005] In deep learning, the model is learned by performing a prescribed learning process using multiple annotated data. At this time, if the annotation data with high annotation accuracy is used for learning, the accuracy of the model will also increase. In other words, the accuracy of data classification based on deep learning depends on the quality of the annotation.

[0006] Therefore, we need to seek annotation data with good accuracy. Regarding annotation, there are cases where annotation is done manually and cases where annotation is generated based on the data left in the system. In either case, the results contain some misjudgments or fluctuations, and in order to improve accuracy, human confirmation and correction are often required. The operation of correcting annotation errors in annotation data or deleting erroneous data is called cleaning.

[0007] In deep learning, classification can be performed even if the number of classes is more than several thousand. However, when the number of classes is so large, manual cleaning requires people with rich knowledge to do the work. However, in this case, the work time becomes longer, and errors occur due to decreased attention, so it costs a lot to obtain high-precision annotation data. In addition, the accuracy is limited.

[0008] Therefore, it is desired to automatically perform cleaning without manual labor. An example of this is disclosed in Patent Document 1 listed below.

[0009] Prior art literature

[0010] Patent Literature

[0011] Patent Document 1: Japanese Patent Application Publication No. 2021-157497 Summary of the invention

[0012] Problem that the invention aims to solve

[0013] Patent document 1 discloses a method for automatically cleaning annotation data in deep learning. Specifically, an image set is segmented, and a portion of the segmented image set is used to overlearn CNN (Convolutional Neural Networks), and the overlearned CNN is used to delete and reclassify (re-annotate) the image in the image classification result, and the above process is repeated until the specified qualification criteria are met.

[0014] However, it is known that the accuracy of data classification decreases if over-learning is performed, and there is no structure disclosed to show the reason why the accuracy of data classification is generally improved by the method disclosed in Patent Document 1. It is also unclear whether the accuracy of data classification is actually improved. In addition, since over-learning, re-classification (re-annotation) and other processes are repeatedly performed, the processing becomes complicated, and the processing time and processing burden increase.

[0015] Solutions for solving problems

[0016] In view of the above problems, the inventors have invented an information processing system that automatically cleans annotation data in deep learning in a simpler way.

[0017] The first invention is an information processing system that performs cleaning processing of annotation data in a data set, wherein the data set is a collection of annotation data including classification object data and a class corresponding to the classification object data, and the information processing system comprises: a learning processing unit that performs deep learning learning processing using the data set to generate a model; and a cleaning processing unit that performs cleaning processing of the annotation data in the data set, wherein the cleaning processing unit outputs a first class corresponding to the classification object data in first annotation data of a first data set, inputs the classification object data into a first model generated by the learning processing unit using the first data set to output a second class, and performs the cleaning processing of the first annotation data in the first data set by comparing the first class with the second class.

[0018] By configuring as in the present invention, it is possible to perform cleaning processing of annotation data in deep learning in a simpler method than the prior art described in Patent Document 1. In addition, the cleaning processing in the information processing system of the present invention refers to the processing of excluding erroneous annotation data from a data set (exclusion type cleaning processing).

[0019] In the above invention, the cleaning processing unit can be constructed as the following information processing system: outputting a first category corresponding to the classification object data in the first annotation data of the first data set, inputting the classification object data into the first model generated by the learning processing unit using the first data set to output a second category, and if the first category is compared with the second category and the specified conditions are not met, excluding the first annotation data from the first data set, and outputting the second data set using the annotation data in the first annotation data of the first data set that has not been excluded.

[0020] The cleaning process of the annotation data can be performed as in the present invention.

[0021] In the above invention, the information processing system may be configured such that the learning processing unit generates the second model using a data set that is a collection of annotation data on which the cleaning process has been performed.

[0022] Models for deep learning are preferably generated as described in these inventions.

[0023] In the above invention, the information processing system may include an operation processing unit configured to input analysis target data as a processing target to the second model to execute deep learning processing.

[0024] The model of the present invention performs a cleaning process on the annotation data, so that learning is performed using the annotation data with high accuracy, and the accuracy of deep learning is also improved.

[0025] In the above-mentioned invention, it can be constructed as the following information processing system: the information processing system has an initial annotation processing unit, which performs prescribed classification processing on the classification object data, and outputs the first category corresponding to the classification object data, and outputs the first data set which is a collection of first annotation data obtained by associating the classification object data with the first category.

[0026] Outputting the data set as a collection of first annotation data can be performed by various methods, but may also be performed automatically as in the present invention.

[0027] In the above invention, the information processing system can be configured such that the classification process in the initial annotation data processing unit and the classification process in the cleaning processing unit are different classification processes.

[0028] For example, it is preferable to use different classification processes in a manner such as using a classification process using local feature quantities as a first classification process for a data set outputting first annotation data, and using a classification process using deep learning as a second classification process for a data set outputting second annotation data. Thus, the accuracy of the data set outputted by the cleaning process can be improved.

[0029] In the above invention, it is possible to configure an information processing system in which the classification target data is data obtained by cutting out a part of original data.

[0030] When the raw data includes a plurality of objects, it is preferable to perform processing on data obtained by cutting out a part of the raw data as the classification target data as in the present invention.

[0031] The eighth invention is an information processing system that performs cleaning processing of annotation data in a data set, wherein the data set is a collection of annotation data including image data of a product and product identification information, and the information processing system comprises: a learning processing unit that uses the data set to perform deep learning learning processing to generate a model; and a cleaning processing unit that performs cleaning processing of the annotation data in the data set, wherein the cleaning processing unit outputs first product identification information corresponding to the image data of the product in the first annotation data of the first data set, inputs the image data of the product into the first model generated by the learning processing unit using the first data set to output second product identification information, and performs the cleaning processing of the first annotation data in the first data set by comparing the first product identification information with the second product identification information, and the learning processing unit generates a second model using the data set that is a collection of annotation data on which the cleaning processing has been performed.

[0032] The information processing system of the first invention is preferably used for cleaning the annotation data used for learning the model used in the deep learning process for identifying the commodity identification information of the commodity shown in the image data as in the present invention. This is because: in the case of the process of identifying the commodity identification information of the commodity shown in the image data as in the present invention, there are many similar commodities, so the accuracy of identification is required.

[0033] In the above invention, the following information processing system is constructed: the information processing system has an application processing unit, and the application processing unit inputs the image data of the goods displayed on the display shelf into the second model based on the image data obtained by photographing the display shelf to perform deep learning processing, thereby identifying the product identification information of the goods.

[0034] The information processing system of the first invention can be realized by causing a computer to read the program of the present invention and execute the program. That is, the computer is made to function as the following components: a learning processing unit that performs deep learning learning processing to generate a model using a data set that is a collection of annotation data including classification object data and classes corresponding to the classification object data; and a cleaning processing unit that performs cleaning processing of the annotation data in the data set, wherein the cleaning processing unit outputs a first class corresponding to the classification object data in the first annotation data of the first data set, inputs the classification object data to the first model generated by the learning processing unit using the first data set to output a second class, and performs the cleaning processing of the first annotation data in the first data set by comparing the first class with the second class.

[0035] The information processing system of the eighth invention can be realized by causing a computer to read the program of the present invention and execute the program. That is, the computer is made to function as the following components: a learning processing unit that performs deep learning learning processing using a set of annotation data including image data of commodities and commodity identification information, i.e., a data set, to generate a model; and a cleaning processing unit that performs cleaning processing of the annotation data in the data set, wherein the cleaning processing unit outputs first commodity identification information corresponding to the image data of the commodity in the first annotation data of the first data set, inputs the image data of the commodity to the first model generated by the learning processing unit using the first data set to output second commodity identification information, performs the cleaning processing of the first annotation data in the first data set by comparing the first commodity identification information with the second commodity identification information, and generates a second model using the data set as a set of annotation data on which the cleaning processing is performed.

[0036] Effects of the Invention

[0037] By using the information processing system of the present invention, annotation data in deep learning can be automatically cleaned in a simpler way. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a block diagram schematically showing an example of the configuration of the information processing system of the present invention.

[0039] Figure 2This is a block diagram schematically showing an example of the hardware configuration of a computer used in the information processing system of the present invention.

[0040] Figure 3 This is a conceptual diagram schematically showing an example of the overall processing of the information processing system of the present invention.

[0041] Figure 4 This is a conceptual diagram showing the flow of data utilization in the overall processing of the information processing system of the present invention.

[0042] Figure 5 This is a flowchart showing an example of the overall processing procedure of the information processing system of the present invention.

[0043] Figure 6 This is a flowchart showing an example of the processing procedure of initial annotation data processing by the information processing system of the present invention.

[0044] Figure 7 This is a flowchart showing an example of a processing procedure of a cleaning process of the information processing system of the present invention.

[0045] Figure 8 This is a block diagram schematically showing an example of the configuration of the information processing system according to the second embodiment.

[0046] Fig. 9 This is a conceptual diagram schematically showing an example of overall processing of the information processing system of the second embodiment.

[0047] Fig.10 This is a conceptual diagram showing the flow of data utilization in the overall processing of the information processing system of the second embodiment.

[0048] Fig.11 This is a flowchart showing an example of a processing procedure of initial annotation data processing in the information processing system of the second embodiment.

[0049] Fig.12 This is a conceptual diagram schematically showing an example of overall processing of the information processing system of the third embodiment.

[0050] Fig.13 This is a diagram showing an example of photographic image data in Example 3.

[0051] Fig.14 It is shown that according to Fig.13 A diagram showing an example of a state in which a row surface area is determined and cut out in a rack area determined by photographic image data. DETAILED DESCRIPTION

[0052] exist Figure 1An example of the configuration of the information processing system 1 of the present invention is shown in a block diagram. The information processing system 1 is implemented using a computer used by an organization such as a company operating the information processing system 1 or a third party's computer or the like.

[0053] An example of the hardware structure of a computer used in the information processing system 1 is schematically shown. The computer includes: a CPU or other computing device 70 that executes computing processing of a program; a storage device 71 such as a RAM or a hard disk that stores information; a display device 72 such as a monitor that displays information; an input device 73 such as a keyboard or a mouse that can input information; and a communication device 74 that sends and receives processing results of the computing device 70 and information stored in the storage device 71 via a network such as the Internet or a LAN.

[0054] When the computer includes a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited thereto.

[0055] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 based on the fact that input can be directly performed on the display using a predetermined input device (a pen for a touch panel, etc.), a finger, etc.

[0056] The components in the present invention are only logically distinguished in their functions, and they may form the same area physically or in fact. The processing of the components of the present invention may also be appropriately changed in the processing order. In addition, part of the processing may also be omitted. In addition, all or part of the functions of the information processing system 1 may also be implemented by cloud computing.

[0057] The information processing system 1 includes an initial annotation processing unit 20 , a learning processing unit 21 , a cleaning processing unit 22 , and an operation processing unit 23 .

[0058] The initial annotation processing unit 20 has a raw data input acceptance processing unit 201, a first classification processing unit 202, and a first data set output processing unit 203, and uses the accepted input raw data to generate and output a data set DS0 of initial annotation data (initial annotation data). Annotation data is data obtained by associating data that will become a classification object (classification object data) with a class. As classification object data, the following description will explain the case where the classification object data is image data, but if it is sound data, text data, biometric information data (fingerprints, voiceprints, irises, veins, etc. caused by the human body) and other data that become classification objects in deep learning, it can be any data. In addition, regarding the record of image, it can be replaced with sound, text, biometric information, etc.

[0059] The raw data input accepting processing unit 201 accepts input of image data (raw data) that is the source of initial annotation data. The raw data accepted as input may be data of a quantity sufficient to form annotation data, for example, more than ten data, but may also be input of several hundred or several thousand data. The annotation data of the data set DS1 output after the cleaning process performed by the cleaning processing unit 22 described later may be data of a quantity sufficient to generate the model 1.

[0060] The first classification processing unit 202 performs a predetermined classification process, such as an image classification process, on the classification object data, such as each input raw data or a part of the data accepted by the raw data input acceptance processing unit 201, thereby performing a process of associating the classification object data with the class C0. As to which class the classification object data belongs to, it can be performed by performing a predetermined classification process, such as a classification process using a local feature quantity, a classification process using deep learning, or any other process. As the classification process in the first classification processing unit 202, when multiple classes are output in descending order of reliability, it is preferred that the class with the highest reliability be set as the class C0 for the classification object data, but multiple classes may also be set as the class C0. In this case, it is also possible to set the class C0 as a predetermined number or a predetermined reliability range from the class with the highest reliability. In this case, multiple classes are associated as the class C0 for the classification object data. The classification process in the first classification processing unit 202 is preferably a classification process different from the classification process of the second classification processing unit 220 described later. For example, as a result of performing image classification processing in the first classification processing unit 202 using classification target data (raw data showing “dog”) as input values, the class “dog” is determined and output.

[0061] The first data set output processing unit 203 associates the classification object data input as input values ​​to the first classification processing unit 202 with the class C0 output by the classification processing to form initial annotation data. Then, a set of initial annotation data including each classification object data and the class C0 corresponding to the classification object data is output as a data set DS0.

[0062] The learning processing unit 21 performs a learning process based on deep learning using a data set that is a collection of annotation data to generate a model. For example, a learning process is performed using the data set DS0 output by the initial annotation processing unit 20 to generate a model 0. In addition, as described later, a learning process is performed using the data set DS1 output by the cleaning processing unit 22 to generate a model 1. The learning process can be performed by associating image data in the data set with classes and inputting them as correct answer data, similar to the well-known learning process in deep learning.

[0063] In addition, the learning processing unit 21 only needs to perform a well-known learning process in deep learning, and therefore, the annotation data in the data set is classified into learning data and test data for learning processing at a predetermined ratio, and the learning process is performed. The ratio at this time can be arbitrary. For example, a model is generated by a well-known learning process using the learning data, and test data is input to the model. Then, the learning process is performed on the result of the output test data until the predetermined reliability is ensured.

[0064] The cleaning processing unit 22 includes a second classification processing unit 220, a comparison processing unit 221 and a second data set output processing unit 222, which excludes annotation data in the data set DS0 output by the initial annotation processing unit 20 that meets specified conditions from the data set DS0, and outputs the remaining set of annotation data as a data set DS1.

[0065] The second classification processing unit 220 inputs the classification object data in the dataset DS0 of the initial annotation data as input values ​​to the model 0 generated by the learning processing unit 21 using the dataset DS0 for learning, and performs processing based on deep learning, and outputs the class C1 for the classification object data as an output value. It is preferable to perform this processing on the learning data classified by the learning processing unit 21 in the dataset DS0, but it is also possible to input the classification object data classified as test data in the dataset DS0 as input values ​​to the model 0, and perform processing based on deep learning, and output the class C1 for the classification object data as an output value. As the classification processing in the second classification processing unit 220, when multiple classes are output in order from high to low reliability, it is preferable to set the class with the highest reliability as the class C1 for the classification object data, but multiple classes may also be set as the class C1. In this case, it is also possible to set the classes within a specified number or a specified reliability range from the class with the highest reliability as the class C1. In this case, multiple classes are associated as the class C1 for the classification object data.

[0066] The comparison processing unit 221 compares the class C0 corresponding to the classification object data in the data set DS0 with the class C1 outputted by the second classification processing unit 220 for the classification object data to determine whether the prescribed condition is satisfied. That is, the class C0 obtained by the classification processing by the first classification processing unit 202 and the class C1 obtained by the classification processing by the second classification processing unit 220, which are attached to the same classification object data, are compared to determine whether the prescribed condition, for example, is consistent. Then, the inconsistent annotation data is excluded from the data set DS0. In addition, when the class C1 obtained by the classification processing by the second classification processing unit 220 contains multiple classes, it is also possible to determine that the annotation data is consistent when a class in the class C1 contains a class in the class C0 obtained by the classification processing by the first classification processing unit 202. In addition, as a specified condition, in addition to the situation where class C0 is consistent with class C1 as described above, class C0 obtained by the first classification processing unit 202 and class C1 obtained by the second classification processing unit 220 may be added with conditions such as whether their respective reliabilities exceed a specified threshold, and whether the reliability between the output results of the second class differs by more than a specified value, etc., and the annotation data may be determined to be consistent when the added conditions are met.

[0067] When the classification object data classified as test data is input into the model 0 in the second classification processing unit 220, and the class C0 of the classification object data is inconsistent with the class C1, the comparison processing unit 221 may exclude the inconsistent annotation data from the data set DS0, or may not exclude the inconsistent annotation data classified as test data from the data set DS0.

[0068] The second data set output processing section 222 outputs the data set DS1 as a collection of annotation data excluding the annotation data excluded in the comparison processing section 221 among the annotation data of the data set DS0 .

[0069] The application processing unit 23 inputs the data set as the actual processing object (processing object data) as an input value to the model 1 generated by learning using the data set DS1, performs processing based on deep learning, and outputs the class C2 for the processing object data as an output value. In addition, the reliability of the class C2 can also be output at this time. In addition, the processing in the application processing unit 23 can be a known deep learning-based image classification processing using the model 1.

[0070] In addition, in the present specification, as a classification processing based on deep learning, a model (model 0, model 1) in which the weighting coefficients between neurons in each layer of a neural network whose intermediate layer is composed of multiple layers are optimized is input with classification object data or processing object data as an input value, and the class of its output value (class C1, class C2) is output.

[0071] In addition, instead of automatically annotating the classification target data and outputting the data set DS0 in the initial annotation processing unit 20 , for example, a predetermined operator may manually annotate each classification target data and output the data set DS0 . In this case, the initial annotation processing unit 20 may not be provided.

[0072] Example 1

[0073] Next, use Figure 3 and Figure 4 Concept map of Figures 5 to 7 An example of the processing procedure of the information processing system 1 of the present invention is described with reference to a flowchart. In the following description, a case where the classification target data is image data and the image classification process is performed on the image data is described.

[0074] First, the information processing system 1 of the present invention executes initial annotation data processing ( S100 ) in order to learn a model used in deep learning.

[0075] Specifically, a sufficient amount of image data (raw data) for learning, for example, ten or more images per class, is input to the information processing system 1, and the raw data input accepting processing unit 201 of the initial annotation processing unit 20 accepts the input (S200).

[0076] Then, the first classification processing unit 202 performs image classification processing on each received input raw data, i.e., image data (classification target data) (S210), and outputs class C0 for each raw data. The first data set output processing unit 203 associates each raw data with class C0 to form initial annotation data, and outputs data set DS0 as a collection of the initial annotation data (S220).

[0077] As described above, the initial annotation processing section 20 outputs the data set DS0 of initial annotation data based on the original data.

[0078] Then, the learning processing unit 21 classifies the dataset DS0 of the initial annotation data into learning data for learning processing and test data for model testing, and performs deep learning learning processing to generate model 0 (S110). In addition, image data and corresponding classes are associated with each annotation data classified as learning data and test data.

[0079] The cleaning processing unit 22 performs a cleaning process on the initial annotation data in the data set DS0 using the model 0 generated in this way ( S120 ).

[0080] That is, the second classification processing unit 220 in the cleaning processing unit 22 inputs the image data of the learning data in the data set DS0, or the image data of the learning data and the test data, as input values ​​to the model 0 to perform image classification processing based on deep learning (S300), and outputs the class C1 for the input data (learning data, or learning data and test data). The classification processing unit preferably performs classification processing on all the data of the learning data, or the learning data and the test data in the data set DS0.

[0081] Then, the comparison processing unit 221 compares the class C0 corresponding to the image data (classification target data) as the original data in the data set DS0 input to the second classification processing unit 220 with the class C1 which is the output result for the image data output by the second classification processing unit 220 (S310), to determine whether the class C0 is consistent with the class C1. If the class C0 is inconsistent with the class C1, the annotation data is excluded from the data set DS0 (S320).

[0082] After the comparison processing by the comparison processing unit 221, the second data set output processing unit 222 outputs the set of annotation data that is not excluded as a data set DS1 (S330). In addition, at this time, the image data of the non-excluded display area and the annotation data of the commodity identification information (class C1) corresponding to the image data can also be determined, and the set can be output as the data set DS1.

[0083] When the classes in the class C0 obtained by the classification processing in the first classification processing unit 202 and the classes in the class C1 obtained by the classification processing in the second classification processing unit 220 are respectively consistent, it is estimated that the annotation accuracy of the annotation data is high. On the other hand, in the case of inconsistency, there may be errors in the annotation of the annotation data. Therefore, the annotation data with inconsistent classes is excluded from the data set, thereby being able to output the data set DS1 of annotation data with good accuracy.

[0084] Then, the learning processing unit 21 executes a learning process of deep learning using each annotation data of the data set DS1 output after the cleaning process by the cleaning processing unit 22, thereby generating a new model 1 (S130).

[0085] The application processing of deep learning is performed using the model 1 generated as described above in the application processing unit 23 (S140). That is, the image data (processing target data) set as the actual processing target is input as an input value to the model 1, and the processing based on deep learning is performed, and the class C2 for the image data input as the input value is output as an output value.

[0086] Furthermore, in the image classification processing of the first classification processing unit 202 and the second classification processing unit 220, in addition to the output class, the reliability of the class (how reliable the output class can be) is also output, and the allocation can be performed according to a certain threshold. It can also be set as follows: for example, for the output result (reliable class) whose reliability is above the specified threshold (greater than the specified threshold), it is directly determined, and for the output result (unreliable class) whose reliability is less than the specified threshold (below the specified threshold), the recognition result is shown to the operator, and in the case of an error, the correction input of the correct class is accepted or the data itself is excluded. In this way, it is also possible to automatically determine whether it is necessary to manually correct the annotation data that should be corrected. It can also be set as follows: for example, for the class associated with the image data as the classification target data in the second classification processing unit 220, when the output result of the first class and the output result of the second class are below the specified value or less than the specified value, the recognition result is shown to the operator.

[0087] Example 2

[0088] When the raw data is image data, multiple objects may be reflected in the image data. When a process is performed to identify commodities displayed on a display rack based on image information obtained by photographing a display rack, for example, multiple commodities are often reflected. It is preferable to detect an object based on the raw data and perform a process of setting the image data containing the object as classification target data and processing target data. This is explained below.

[0089] exist Figure 8 An example of the overall structure of the information processing system 1 in this embodiment is schematically shown in FIG. The information processing system 1 in this embodiment further includes an object detection processing unit 24. Fig. 9 A conceptual diagram showing an example of the overall processing of the second embodiment is shown in Fig.10 The conceptual diagram shows the process of data utilization.

[0090] The object detection processing unit 24 detects the object based on the image data and cuts out the area containing the object. The so-called cutting out the area containing the object can be to actually separate or copy the area containing the object from the image data and set it as other image data, or it can be that the area containing the object is specified and the processing is performed on the area without separating or copying it from the image data. In addition, it can be cut along the outline of the object, or it can be cut into a rectangular area containing the object or an area of ​​any shape containing the outline of the object. In addition, the image data obtained by cutting out the area containing the object is called local image data. The local image data can be other image data obtained by actually separating or copying the area containing the object from the image data as described above, or it can be data obtained by specifying the area containing the object without separating or copying it from the image data. In this case, the local image data becomes the classification object data in Example 1, and is associated with the class in the annotation data.

[0091] The object detection processing unit 24 can use various well-known methods to cut out the object. For example, it can cut out the object by performing pattern matching with the target object, or it can perform processing based on deep learning using a model that detects the object from image data.

[0092] Next, use Figure 5 , Figure 7 as well as Fig.11 An example of the processing procedure of this embodiment is described with reference to the flowchart of FIG.

[0093] First, the information processing system 1 of the present invention executes initial annotation data processing ( S100 ) in order to learn a model used in deep learning.

[0094] Specifically, a sufficient amount of image data (raw data) for learning, for example, ten or more images per class, is input to the information processing system 1, and the raw data input accepting processing unit 201 of the initial annotation processing unit 20 accepts the input (S200).

[0095] Then, the object area is detected based on each received input raw data (S205), and the local image data (classification target data) obtained by cutting out the object area is cut out. When multiple objects can be detected in the raw data, multiple local image data are cut out from the raw data.

[0096] Then, the first classification processing unit 202 performs image classification processing on each local image data (S210), and outputs class C0 for each local image data. The first data set output processing unit 203 associates each local image data with class C0 to form initial annotation data, and outputs data set DS0 as a collection of the initial annotation data (S220).

[0097] As described above, the initial annotation processing section 20 outputs the data set DS0 of initial annotation data based on the partial image data (classification target data) in the original data.

[0098] Then, similarly to the first embodiment, the learning processing unit 21 classifies the dataset DS0 of the initial annotation data into learning data for learning processing and test data for model testing, performs a learning process of deep learning, and generates a model 0 ( S110 ).

[0099] The cleaning processing unit 22 performs a cleaning process on the initial annotation data in the data set DS0 using the model 0 generated in this way ( S120 ).

[0100] That is, the second classification processing unit 220 in the cleaning processing unit 22 inputs the local image data (classification target data) in the data set DS0, that is, the learning data, or the learning data and the test data as input values ​​to the model 0 to perform image classification processing based on deep learning (S300), and outputs the class C1 for the input local image data (learning data, or the learning data and the test data). The classification processing unit preferably performs classification processing on all the learning data, or the learning data and the test data, which are the local image data (classification target data) in the data set DS0, but may also perform classification processing on a part of the data.

[0101] Then, the comparison processing unit 221 compares the class C0 corresponding to the partial image data in the data set DS0 input to the second classification processing unit 220 with the class C1 outputted from the second classification processing unit 220 for the partial image data (S310), to determine whether the class C0 is consistent with the class C1. If the class C0 is inconsistent with the class C1, the annotation data is excluded from the data set DS0 (S320).

[0102] After the comparison processing by the comparison processing unit 221 , the second data set output processing unit 222 outputs the set of annotation data that is not excluded as a data set DS1 ( S330 ).

[0103] The learning processing unit 21 performs a learning process of deep learning using each annotation data of the data set DS1 output after the cleaning process by the cleaning processing unit 22, thereby generating a new model 1 (S130).

[0104] The application processing unit 23 uses the model 1 generated as described above to perform deep learning application processing (S140). That is, the object detection processing unit 24 performs object detection processing on the image data set as the actual processing object, detects the area of ​​the object based on the received input image data, and cuts out the local image data (processing object data) obtained by cutting out the area of ​​the object. Then, the cut-out local image data is input to the model 1 as an input value to perform processing based on deep learning, and the class C2 for the image data input as the input value is output as an output value.

[0105] Example 3

[0106] Next, a case where the information processing system 1 of the present invention is used in processing for identifying commodities based on image data (photographic image data) obtained by photographing a display shelf in a store is described. In this embodiment, a deep learning-based processing is performed on commodities displayed on a display shelf shown in the image data obtained by photographing the display shelf to identify commodity identification information such as commodity names and predetermined codes (JAN codes, etc.).

[0107] In the information processing system 1 of this embodiment, the operation processing unit 23 has a processing function for identifying commodities based on photographic image data obtained by photographing a display rack. The operation processing unit 23 includes a photographic image input receiving processing unit 230, an image data processing unit 231, and a commodity identification processing unit 232. Fig.12 2 is a block diagram showing an example of the configuration of the information processing system 1 of the present embodiment.

[0108] The photographic image input acceptance processing unit 230 accepts input of photographic image data to be processed by executing a process of identifying commodities displayed on a display shelf using a model based on deep learning.

[0109] The image data processing unit 231 performs correction processing such as keystone correction processing, distortion correction processing, etc. on the captured image data so as to achieve a straight-on state. The correction processing in the image data processing unit 231 can use various known methods.

[0110] The product identification processing unit 232 takes the image data of the area (display area) where products are displayed, which is extracted from the photographic image data by the object detection processing unit 24, as an input value, and inputs it into the model 1 obtained by learning using the data set DS1 output by the cleaning processing unit 22, and outputs the product identification information reflected in the display area as a class for identification.

[0111] Next, a description will be given of a process of identifying commodities based on photographic image data obtained by photographing a display shelf in a store using the information processing system 1 of the present invention.

[0112] First, a process of generating a model (model 1) used in deep learning for identifying commodities will be described.

[0113] In order to generate a model for identifying products through deep learning, image data obtained by photographing display shelves (photographic image data for learning. Original data) is prepared in an amount sufficient for deep learning processing, for example, more than ten for each category, and input into the information processing system 1.

[0114] The raw data input receiving processing unit 201 of the initial annotation processing unit 20 receives input of photographic image data for learning as raw data. Then, the area of ​​the merchandise displayed on the display shelf is detected based on each received input photographic image data, and the area is cut out as image data of the display area (classification target data).

[0115] Specifically, the photographic image data is set to Fig.13 Like that. A plurality of display shelves are reflected in the photographic image data, and a plurality of commodities are displayed on each display shelf. After the photographic image data obtained by photographing the display shelf is subjected to known correction processing such as trapezoidal correction processing and distortion correction processing in the image data processing unit 231, the area of ​​the display shelf where the commodities are displayed (display shelf area) is determined by a known method. Then, the object detection processing unit 24 detects the area containing the commodities (display surface area) from the display shelf area by a known method, and captures the image data of the display surface area (classification object data). Fig.14 It is shown in Fig.13 An example of a state in which a display area is determined and cut out from a shed area determined by photographic image data. The display area includes commodities displayed on a display shelf. In addition to determining a shed area and cutting out a display area from the shed area, the display area can also be cut out from the photographic image data.

[0116] The image data of the array area cut out in this way is input to the first classification processing unit 202 to perform image classification processing, identify the commodity identification information (class C0) and output it. As the image classification processing in the first classification processing unit 202, for example, the image data of each commodity as a sample or the data (specimen data) based on the local feature quantity of the image data is pre-stored, and the commodity identification information is identified by comparing the input image data of the array area with the specimen data. Alternatively, the commodity identification information can also be identified by deep learning, etc. Through the image classification processing performed by the first classification processing unit 202, the commodity identification information (class C0) of the commodity reflected in the image data of each array area can be identified. Then, the first data set output processing unit 203 associates the image data of the array area with the identified commodity identification information (class C0) to form annotation data. In addition, the output data set DS0 is a collection of initial annotation data obtained by associating the image data of the array area with the identified commodity identification information (class C0) of the commodity.

[0117] Through the above-described processing, the initial annotation processing unit 20 outputs the data set DS0.

[0118] Then, the learning processing unit 21 classifies the data set DS0 of the initial annotation data into learning data for learning processing and test data for model testing, and performs a learning process of deep learning, thereby generating a model 0 .

[0119] The cleaning processing unit 22 performs cleaning processing on the initial annotation data in the data set DS0 using the model 0 generated in this way.

[0120] That is, the second classification processing unit 220 in the cleaning processing unit 22 inputs the image data (learning data, or learning data and test data) of each display area in the data set DS0 as input values ​​into the model 0 to perform image classification processing based on deep learning, and outputs the product identification information (class C1) for the image data of the display area.

[0121] Then, the comparison processing unit 221 compares the commodity identification information (class C0) corresponding to the image data of the array area in the data set DS0 input to the second classification processing unit 220 with the commodity identification information (class C1) outputted from the second classification processing unit 220 for the image data, to determine whether they are the same commodity identification information. Furthermore, if they are not the same commodity identification information, the image data of the array area and the annotation data of the commodity identification information (class C0) corresponding to the image data are excluded from the data set DS0.

[0122] After the comparison processing by the comparison processing unit 221, the second data set output processing unit 222 outputs the image data of the display area that is not excluded and the annotation data of the commodity identification information (class C0) corresponding to the image data as a data set DS1.

[0123] When the data set DS1 is output from the cleaning processing unit 22 as described above, the learning processing unit 21 performs a learning process of deep learning using each annotation data of the data set DS1 , thereby generating a new model 1 .

[0124] Through the above-described processing, the model 1 for identifying a commodity can be generated.

[0125] Next, a description will be given of a process (operation process) of identifying commodities displayed on a display shelf that is shown in photographic image data obtained by photographing a display shelf in a store.

[0126] First, when the photographic image data to be processed is input to the information processing system 1, the photographic image input receiving processing unit 230 of the application processing unit 23 receives the input. Then, the image data processing unit 231 performs correction processing on the image information based on the received input photographic image data, and the object detection processing unit 24 determines the display shelf area for the photographic image data after the correction processing by a known method. Then, the object detection processing unit 24 intercepts the image data of the display surface area including the commodity from the display shelf area by a known method. In addition to determining the shelf grid area and intercepting the display surface area from the shelf grid area, the display surface area can also be intercepted from the photographic image data.

[0127] The product identification processing unit 232 inputs the image data of the display area cut out in this way as an input value to the model 1 generated as described above, performs deep learning-based processing, and outputs the product identification information (class C2) for the display area as an output value.

[0128] By executing the above-described processing, it is possible to accurately identify the commodity identification information of the commodity displayed on the display shelf that is reflected in the photographic image data that is the processing target.

[0129] Example 4

[0130] In the third embodiment, the information processing system 1 of the present invention is described as performing identification processing on commodities displayed on a display shelf, but the present invention is not limited to this and can be widely applied to performing image classification based on image data to be processed by deep learning.

[0131] In the above-mentioned embodiments 1 to 4, the case of performing deep learning learning and application processing using image data as an example is described, but it is not limited to image data, and can also be sound data or text data. In this case, the same processing can be performed by replacing the description of "image" with "sound" or "text".

[0132] In addition, when processing "images", "sounds", and "texts", the description of "image data", "sound data", "text data", etc., even if it is not the image data, sound data, or text data itself, it can be data that refers to the image, sound, or text. For example, it also includes data of feature quantities such as local feature quantities, data indicating the location of data, and other data that has some kind of correlation.

[0133] Industrial Applicability

[0134] By using the information processing system 1 of the present invention, annotation data in deep learning can be automatically cleaned in a simpler way.

[0135] Description of Reference Numerals

[0136] 1: information processing system; 20: initial annotation processing unit; 21: learning processing unit; 22: cleaning processing unit; 23: application processing unit; 24: object detection processing unit; 70: computing device; 71: storage device; 72: display device; 73: input device; 74: communication device; 201: raw data input acceptance processing unit; 202: first classification processing unit; 203: first data set output processing unit; 220: second classification processing unit; 221: comparison processing unit; 222: second data set output processing unit; 230: photographic image input acceptance processing unit; 231: image data processing unit; 232: commodity identification processing unit.

Claims

1. An information processing system for performing cleaning processing on annotation data in a data set, wherein the data set is a set of classification object data and annotation data of a class corresponding to the classification object data, The information processing system is characterized by having: a learning processing unit that performs a deep learning learning process using the data set to generate a model; and a cleaning processing unit, which performs cleaning processing on the annotation data in the data set, in, The cleaning processing unit outputs a first category corresponding to the classified object data in the first annotation data of the first data set, inputs the classified object data into the first model generated by the learning processing unit using the first data set to output a second category, and performs the cleaning processing of the first annotation data in the first data set by comparing the first category with the second category.

2. The information processing system according to claim 1, characterized in that The cleaning treatment unit performs the following treatments: outputting a first category corresponding to the classification target data in the first annotation data of the first data set, inputting the classification target data into a first model generated by the learning processing unit using the first data set to output a second category, If the first category is compared with the second category and the prescribed condition is not satisfied, excluding the first annotation data from the first data set, A second data set is outputted using the non-excluded annotation data of the first annotation data of the first data set.

3. The information processing system according to claim 1, characterized in that The learning processing unit generates a second model using a data set that is a collection of annotation data on which the cleaning process has been performed.

4. The information processing system according to claim 3, characterized in that: The information processing system includes an operation processing unit that inputs analysis target data as a processing target to the second model to execute deep learning processing.

5. The information processing system according to claim 1, characterized in that: The information processing system includes an initial annotation processing unit, which performs prescribed classification processing on the classification object data, outputs the first category corresponding to the classification object data, and outputs the first data set which is a collection of first annotation data obtained by associating the classification object data with the first category.

6. The information processing system according to claim 5, characterized in that: The classification process in the initial annotation data processing unit and the classification process in the cleaning processing unit are different classification processes.

7. The information processing system according to any one of claims 1 to 6, characterized in that: The classification target data is data obtained by cutting out a part of original data.

8. An information processing system for performing cleaning processing on annotation data in a data set, wherein the data set is a collection of annotation data including image data of a commodity and commodity identification information, The information processing system is characterized by having: a learning processing unit that performs a deep learning learning process using the data set to generate a model; and a cleaning processing unit, which performs cleaning processing on the annotation data in the data set, in, The cleaning processing unit outputs first commodity identification information corresponding to the image data of the commodity in the first annotation data of the first data set, inputs the image data of the commodity to the first model generated by the learning processing unit using the first data set to output second commodity identification information, and performs the cleaning processing of the first annotation data in the first data set by comparing the first commodity identification information with the second commodity identification information. The learning processing unit generates a second model using a data set that is a collection of annotation data on which the cleaning process has been performed.

9. The information processing system according to claim 8, characterized in that: The information processing system includes an application processing unit, which inputs image data of commodities displayed on the display shelf into the second model based on image data obtained by photographing the display shelf to perform deep learning processing, thereby identifying commodity identification information of the commodities.

10. An information processing program that enables a computer to function as: a learning processing unit that generates a model by performing a learning process of deep learning using a data set that is a collection of classification target data and annotation data of a class corresponding to the classification target data; and a cleaning processing unit, which performs cleaning processing on the annotation data in the data set, The information processing program is characterized in that The cleaning processing unit outputs a first category corresponding to the classified object data in the first annotation data of the first data set, inputs the classified object data into the first model generated by the learning processing unit using the first data set to output a second category, and performs the cleaning processing of the first annotation data in the first data set by comparing the first category with the second category.

11. An information processing program that enables a computer to function as: a learning processing unit that generates a model by performing a learning process of deep learning using a dataset that is a collection of annotation data including image data of commodities and commodity identification information; and a cleaning processing unit, which performs cleaning processing on the annotation data in the data set, The information processing program is characterized in that The cleaning processing unit outputs first commodity identification information corresponding to the image data of the commodity in the first annotation data of the first data set, inputs the image data of the commodity to the first model generated by the learning processing unit using the first data set to output second commodity identification information, and performs the cleaning processing of the first annotation data in the first data set by comparing the first commodity identification information with the second commodity identification information. The learning processing unit generates a second model using a data set that is a collection of annotation data on which the cleaning process has been performed.

Images