Information Processing Systems

Abstract

The purpose of the present invention is to provide an information processing system that automatically performs cleansing of annotation data required when performing learning for deep learning.　This information processing system executes a cleansing process on a dataset that is a collection of annotation data including to-be-categorized data to be categorized and the class corresponding thereto, the system comprising: a learning processing unit that generates a model by executing learning processing for deep learning by using a dataset; and a cleansing processing unit that executes a cleansing process for the annotation data in the dataset. The cleansing processing unit executes a cleansing process for first annotation data in a first dataset by outputting a first class which corresponds to the to-be-categorized data in the first annotation data in the first dataset, and outputting a second class by inputting the to-be-categorized data into a first model, which was generated by the learning processing unit as a result of using the first dataset, and comparing the first class and the second class.

Description

[Technical Field]

[0001] The present invention relates to an information processing system that automatically cleans annotation data required for deep learning. [Background technology]

[0002] In recent years, a data processing method called deep learning has come to be used for a variety of purposes. For example, it is used to determine the type of object captured in image data. For example, image data of store shelves is input into a deep learning model (trained model, trained network) to identify the products displayed on the shelves captured in the image data. However, it is also used for a variety of other purposes.

[0003] Deep learning has been attracting attention in recent years because it can classify a variety of data.

[0004] On the other hand, when using deep learning, it is necessary to train the model in advance using annotation data. Annotation data is data that indicates the class (type) into which a certain piece of data is classified. For example, for image data of "dog," it is necessary to indicate that the class of that image data is "dog." This process of associating a certain data with its class is called annotation.

[0005] In deep learning, a model is trained by executing a predetermined learning process using a large amount of annotation data. In this case, if the model is trained using annotation data with high annotation accuracy, the accuracy of the model will also be high. In other words, the accuracy of data classification using deep learning depends on the quality of the annotation.

[0006] This calls for highly accurate annotation data. Annotations can be done manually or generated from data left in the system, but in either case, the result is likely to contain some misjudgment or inconsistency, and in most cases, manual review and correction is required to improve accuracy. The process of correcting annotation errors in annotation data or deleting incorrect data is called cleansing.

[0007] Deep learning can classify even thousands of classes, but when the number of classes is so large, manual cleansing requires the work of a person with extensive knowledge. However, this requires a long work time and errors can occur due to a decrease in attention, so obtaining highly accurate annotation data is expensive. In addition, there are limits to the accuracy.

[0008] Therefore, there is a demand for an automatic cleansing system that does not require human intervention. One such system is disclosed in Patent Document 1 below. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Patent Publication No. 2021-157497 Summary of the Invention [Problem to be solved by the invention]

[0010] Patent Document 1 discloses a method for automatically cleansing annotation data in deep learning. Specifically, an image set is divided, a part of the divided image set is used to overtrain a convolutional neural network (CNN), and the overtrained CNN is used to classify images, deleting and reclassifying (re-annotating) the images, and this process is repeated until a predetermined pass criterion is met.

[0011] However, it is known that overlearning reduces the accuracy of data classification, and the method disclosed in Patent Document 1 does not disclose any configuration that shows why the accuracy of data classification improves in general cases. It is also unclear whether the accuracy of data classification actually improves. Furthermore, since processes such as overlearning and reclassification (reannotation) are repeated, the process becomes more complex, and the processing time and processing load increase. [Means for solving the problem]

[0012] In view of the above-mentioned problems, the present inventors have invented an information processing system that automatically cleanses annotation data in deep learning using a simpler method.

[0013] A first invention is an information processing system that performs a cleansing process on annotation data in a dataset, which is a collection of annotation data including data to be classified and its corresponding classes, the information processing system including: a learning processing unit that performs a learning process of deep learning using the dataset to generate a model; and a cleansing processing unit that performs a cleansing process on the annotation data in the dataset, wherein the cleansing processing unit: a second class output by inputting the classification target data of the first annotation data into a first model generated by the learning processing unit using the classification target data of the first annotation data in the first dataset and a first class corresponding thereto; and a first class corresponding to the classification target data of the first annotation data input into the first model. and performs the cleansing process on the first annotation data in the first dataset by comparing the first annotation data with the first annotation data.

[0014] The configuration of the present invention makes it possible to perform cleansing processing of annotation data in deep learning in a simpler manner than the conventional technology described in Patent Document 1. Note that the cleansing processing in the information processing system of the present invention refers to processing of excluding erroneous annotation data from a dataset (exclusion-type cleansing processing).

[0015] In the above-mentioned invention, the cleansing processing unit can be configured as an information processing system that inputs a first class corresponding to data to be classified in first annotation data in a first dataset and the data to be classified into a first model generated by the learning processing unit using the first dataset, outputs a second class, compares the first class with the second class, and if a predetermined condition is not satisfied, excludes the first annotation data from the first dataset, and outputs a second dataset using annotation data that has not been excluded from the first annotation data in the first dataset.

[0016] The cleansing process of the annotation data can be performed as in the present invention.

[0017] In the above-described invention, the learning processing unit can be configured as an information processing system that generates a second model using a dataset of a set of annotation data that has been subjected to the cleansing process.

[0018] It is preferable that the models used in deep learning are generated as in these inventions.

[0019] In the above-mentioned invention, the information processing system can be configured as an information processing system having an operation processing unit that inputs data to be analyzed to be processed for the second model and performs deep learning processing.

[0020] Since the model of the present invention has undergone a cleansing process on the annotation data, it learns using highly accurate annotation data, which also increases the accuracy of deep learning.

[0021] In the above-mentioned invention, the information processing system can be configured as an information processing system having an initial annotation processing unit that performs a predetermined classification process on the data to be classified, outputs the first class corresponding to the data to be classified, and outputs the first dataset which is a set of first annotation data that corresponds the data to be classified to the first class.

[0022] A dataset, which is a collection of first annotation data, can be output by various methods, but may also be output automatically, as in the present invention.

[0023] In the above invention, Initial annotation data processing section The classification process in the image processing unit and the classification process in the cleansing processing unit are different classification processes, and the information processing system can be configured as such.

[0024] For example, different classification processes may be used, such as a classification process using local features as the first classification process for outputting a dataset of the first annotation data, and a classification process using deep learning as the second classification process for outputting a dataset of the second annotation data, thereby improving the accuracy of the datasets output by the cleansing process.

[0025] In the above invention, the classification target data can be configured as an information processing system in which a part of the original data is extracted.

[0026] When the original data includes a plurality of objects, it is preferable to perform processing using data obtained by cutting out a portion of the original data as data to be classified, as in the present invention.

[0027] An eighth invention is an information processing system that performs a cleansing process on annotation data in a dataset that is a collection of annotation data including image data of products and product identification information, the information processing system including: a learning processing unit that performs a learning process of deep learning using the dataset to generate a model; and a cleansing processing unit that performs a cleansing process on the annotation data in the dataset, wherein the cleansing processing unit is configured to: Image data of a product in the first annotation data and corresponding first product identification information in The first model was generated using of the first annotation data Enter product image data The output was Second product identification information; first product identification information corresponding to image data of the product of the first annotation data input to the first model; , and the learning processing unit performs the cleansing process on the first annotation data in the first dataset by comparing the first annotation data and the first annotation data, and generates a second model using a dataset of the set of annotation data on which the cleansing process has been performed.

[0028] The information processing system of the first invention may be used for cleansing annotation data to learn a model used in deep learning processing to identify product identification information of products captured in image data, as in the present invention. This is because, in the case of processing to identify product identification information from products captured in image data, as in the present invention, there are many similar products, and therefore high identification accuracy is required.

[0029] In the above-mentioned invention, the information processing system can be configured as an information processing system having an operation processing unit that inputs image data of products displayed on a display shelf from image data of the display shelf, performs deep learning processing on the second model, and identifies product identification information of the products.

[0030] The information processing system of the first invention can be realized by loading and executing the program of the present invention into a computer. That is, the information processing program causes a computer to function as a learning processing unit that generates a model by executing a deep learning learning process using a dataset that is a collection of annotation data including data to be classified and its corresponding classes, and a cleansing processing unit that executes a cleansing process on the annotation data in the dataset, wherein the cleansing processing unit: a second class output by inputting the classification target data of the first annotation data into a first model generated by the learning processing unit using the classification target data of the first annotation data in the first dataset and a first class corresponding thereto; and a first class corresponding to the classification target data of the first annotation data input into the first model. and executes the cleansing process on the first annotation data in the first dataset by comparing the first annotation data with the first annotation data.

[0031] An information processing system according to an eighth aspect of the present invention can be realized by loading and executing the program of the present invention into a computer. That is, the information processing program causes a computer to function as a learning processing unit that executes a deep learning learning process using a dataset that is a collection of annotation data including image data of products and product identification information to generate a model, and a cleansing processing unit that executes a cleansing process on the annotation data in the dataset, wherein the cleansing processing unit performs the following steps: Image data of a product in the first annotation data and corresponding first product identification information in The first model was generated using of the first annotation data Enter product image data The output was Second product identification information; first product identification information corresponding to image data of the product of the first annotation data input to the first model; , and the learning processing unit performs the cleansing process on the first annotation data in the first dataset by comparing the first annotation data and the first annotation data, and generates a second model using a dataset of the set of annotation data on which the cleansing process has been performed. [Effects of the Invention]

[0032] By using the information processing system of the present invention, it becomes possible to automatically cleanse annotation data in deep learning in a simpler manner. [Brief explanation of the drawings]

[0033] [Figure 1] 1 is a block diagram schematically illustrating an example of a configuration of an information processing system according to the present invention. [Figure 2] FIG. 2 is a block diagram schematically illustrating an example of a hardware configuration of a computer used in the information processing system of the present invention. [Figure 3] FIG. 1 is a conceptual diagram illustrating an example of the overall processing in an information processing system according to the present invention. [Figure 4] FIG. 2 is a conceptual diagram showing the flow of data usage in the overall processing in the information processing system of the present invention. [Figure 5] 3 is a flowchart showing an example of the overall processing process in the information processing system of the present invention. [Figure 6] 10 is a flowchart showing an example of a processing process for initial annotation data processing in the information processing system of the present invention. [Figure 7] 10 is a flowchart showing an example of a processing procedure for cleansing processing in the information processing system of the present invention. [Figure 8] FIG. 10 is a block diagram illustrating an example of the configuration of an information processing system according to a second embodiment. [Figure 9] FIG. 10 is a conceptual diagram illustrating an example of the overall processing in the information processing system according to the second embodiment. [Figure 10] FIG. 10 is a conceptual diagram showing the flow of data usage in the overall processing in the information processing system according to the second embodiment. [Figure 11] 10 is a flowchart showing an example of a processing process of initial annotation data processing in the information processing system according to the second embodiment. [Figure 12] FIG. 11 is a conceptual diagram illustrating an example of the overall processing in an information processing system according to a third embodiment. [Figure 13] FIG. 11 is a diagram showing an example of photographed image data in the third embodiment. [Figure 14] 14 is a diagram showing an example of a state in which a face area is identified and cut out in a shelf area identified from the photographed image data of FIG. 13. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0034] An example of the configuration of an information processing system 1 of the present invention is shown in block diagram in Figure 1. The information processing system 1 is realized using a computer used by an organization such as a company that operates the information processing system 1, or a computer of another third party.

[0035] 1 shows a schematic diagram of an example of the hardware configuration of a computer used in the information processing system 1. The computer has a calculation device 70 such as a CPU that executes the arithmetic processing of a program, a storage device 71 such as a RAM or hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or mouse that can input information, and a communication device 74 that transmits and receives the processing results of the calculation device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0036] If the computer is equipped with a touch panel display, the display device 72 may be integrated with the input device 73. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited to these.

[0037] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

[0038] The functions of the various means in the present invention are only logically distinct, and may be physically or practically in the same area. The order of the processes performed by the various means in the present invention may be changed as appropriate. Some of the processes may also be omitted. All or some of the functions of the information processing system 1 may be realized by cloud computing.

[0039] The information processing system 1 includes an initial annotation processing unit 20, a learning processing unit 21, a cleansing processing unit 22, and an operation processing unit 23.

[0040] The initial annotation processing unit 20 has an original data input reception processing unit 201, a first classification processing unit 202, and a first dataset output processing unit 203, and generates and outputs a dataset DS0 of initial annotation data (initial annotation data) using the received input original data. Annotation data is data in which data to be classified (classification target data) is associated with a class. In the following description, the classification target data will be image data, but any data that can be classified by deep learning, such as sound data, text data, or biometric information data (data originating from the human body, such as fingerprints, voiceprints, irises, and veins), can be used. Furthermore, the term "image" can be hereinafter replaced with "sound," "text," or "biometric information," respectively.

[0041] The original data input reception processing unit 201 receives input of image data (original data) that will be the source of the initial annotation data. The original data to be received as input may be a sufficient number of data to be used as annotation data, for example, ten or more, but may also receive input of several hundred or several thousand. It is sufficient that a sufficient number of data can be secured to generate model 1 as annotation data of dataset DS1 that is output after cleansing processing by the cleansing processing unit 22 (described later).

[0042] The first classification processing unit 202 performs a predetermined classification process, such as an image classification process, on data to be classified, e.g., each piece of raw data received by the raw data input receiving processing unit 201, or a portion of the data, thereby associating a class C0 with the data to be classified. The class to which the data to be classified belongs can be determined by performing any process, such as a predetermined classification process using local features or deep learning. When the classification process in the first classification processing unit 202 outputs multiple classes in descending order of reliability, it is preferable to select the most reliable class as class C0 for the data to be classified. However, multiple classes may also be selected as class C0. In this case, a predetermined number of classes from the most reliable class or classes within a predetermined reliability range may be selected as class C0. In this case, multiple classes are associated with the data to be classified as class C0. The classification process in the first classification processing unit 202 is preferably different from the classification process in the second classification processing unit 220, which will be described later. For example, the classification target data (original data showing a "dog") is used as an input value, and as a result of image classification processing in the first classification processing unit 202, a class of "dog" is determined and output.

[0043] The first dataset output processing unit 203 associates the classification target data input as input values ​​to the first classification processing unit 202 with the class C0 output by the classification process to generate initial annotation data, and outputs a set of initial annotation data including each classification target data and its corresponding class C0 as dataset DS0.

[0044] The learning processing unit 21 executes a learning process by deep learning using a dataset, which is a collection of annotation data, to generate a model. For example, the learning processing unit 21 executes the learning process using dataset DS0 output by the initial annotation processing unit 20 to generate model 0. Furthermore, as will be described later, the learning processing unit 21 executes the learning process using dataset DS1 output by the cleansing processing unit 22 to generate model 1. The learning processing is similar to well-known learning processing in deep learning, and the learning processing can be executed by associating image data and classes in the dataset and inputting them as correct answer data.

[0045] The learning processing unit 21 only needs to execute a known learning process in deep learning, and classifies the annotation data in the dataset into learning data and test data to be used in the learning process at a predetermined ratio, and executes the learning process. The ratio may be arbitrary. For example, a model is generated using the learning data through a known learning process, and test data is input to the model. The learning process is then executed until a predetermined reliability is ensured for the results of the output test data.

[0046] The cleansing processing unit 22 has a second classification processing unit 220, a comparison processing unit 221, and a second dataset output processing unit 222, and excludes from the dataset DS0 annotation data output by the initial annotation processing unit 20 annotation data that meets specified conditions, and outputs the set of remaining annotation data as dataset DS1.

[0047] The second classification processing unit 220 inputs the classification target data in the dataset DS0 as input values ​​to the model 0 generated by the learning processing unit 21 through learning using the dataset DS0 of the initial annotation data, performs processing using deep learning, and outputs a class C1 for the classification target data as an output value. This processing is preferably performed on the training data classified by the learning processing unit 21 from the dataset DS0, but it may also be possible to input the classification target data classified as test data in the dataset DS0 as input values ​​to the model 0, perform processing using deep learning, and output a class C1 for the classification target data as an output value. When the classification processing in the second classification processing unit 220 outputs multiple classes in descending order of reliability, it is preferable to designate the most reliable class as the class C1 for the classification target data, but multiple classes may also be designated as class C1. In this case, a predetermined number of classes from the most reliable class or classes within a predetermined reliability range may be designated as class C1. In this case, multiple classes are associated with the classification target data as class C1.

[0048] The comparison processor 221 compares the class C0 corresponding to the data to be classified in the data set DS0 with the class C1 of the output result for that data to be classified by the second classification processor 220, and determines whether a predetermined condition is satisfied. That is, the comparison processor 221 compares the class C0 of the classification process by the first classification processor 202 with the class C1 of the classification process by the second classification processor 220, both of which are assigned to the same data to be classified, and determines whether they satisfy a predetermined condition, for example, whether they match. Then, the comparison processor 221 excludes any mismatching annotation data from the data set DS0. Furthermore, if the class C1 of the classification process by the second classification processor 220 includes multiple classes, the annotation data may be determined to match if the class of the class C0 of the classification process by the first classification processor 202 is included in any of the classes C1. In addition to the above-mentioned predetermined condition that class C0 and class C1 match, further conditions may be added such as whether the reliability of class C0 from the first classification processing unit 202 and class C1 from the second classification processing unit 220 exceeds a predetermined threshold, or whether the reliability of each class differs from the output result of the second class by more than a predetermined value, and if the added condition is met, the annotation data may be determined to be matching.

[0049] In the case where the second classification processing unit 220 has also input the classification target data classified as test data into model 0 and the class C0 and class C1 do not match, the comparison processing unit 221 may exclude the mismatched annotation data from the dataset DS0, or may not exclude the mismatched annotation data classified as test data from the dataset DS0.

[0050] The second dataset output processor 222 outputs the dataset DS1 as a collection of annotation data other than the annotation data excluded by the comparison processor 221 from the annotation data of the dataset DS0.

[0051] The operational processing unit 23 inputs data to be actually processed (processing target data) as an input value to the model 1 generated by learning using the data set DS1, executes processing by deep learning, and outputs a class C2 for the processing target data as an output value. At this time, the reliability of the class C2 may also be output. The processing in the operational processing unit 23 may be image classification processing by well-known deep learning using the model 1.

[0052] In this specification, classification processing using deep learning involves inputting data to be classified or processed as input values ​​to a model (model 0, model 1) in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and outputting the class of the output value (class C1, class C2).

[0053] Furthermore, instead of automatically annotating the data to be classified in the initial annotation processor 20 and outputting the data set DS0, a predetermined operator may manually annotate each piece of data to be classified and output the data set DS0. In this case, the initial annotation processor 20 may not be provided. [Example]

[0054] Next, an example of the processing process of the information processing system 1 of the present invention will be described with reference to the conceptual diagrams of Figures 3 and 4 and the flowcharts of Figures 5 to 7. In the following description, the data to be classified is image data, and image classification processing is performed on the image data.

[0055] First, the information processing system 1 of the present invention executes initial annotation data processing in order to learn a model to be used in deep learning (S100).

[0056] Specifically, a sufficient amount of image data (original data) for use in learning, for example, ten or more per class, is input into the information processing system 1, and the input is accepted by the original data input acceptance processing unit 201 of the initial annotation processing unit 20 (S200).

[0057] The first classification processing unit 202 then executes image classification processing for each of the input image data (data to be classified) that is the original data (S210), and outputs a class C0 for each of the original data. The first dataset output processing unit 203 associates each of the original data with the class C0, respectively, for each of the original data, to create initial annotation data, and outputs a dataset DS0 of the set (S220).

[0058] In this manner, the initial annotation processing unit 20 outputs the data set DS0 of the initial annotation data based on the original data.

[0059] Then, the learning processing unit 21 classifies the data set DS0 of the initial annotation data into learning data used in the learning process and test data used to test the model, and executes the learning process in deep learning to generate model 0 (S110). Each piece of annotation data classified into learning data and test data is associated with image data and its corresponding class.

[0060] Using the model 0 thus generated, the cleansing processing unit 22 executes the cleansing process on the initial annotation data in the data set DS0 (S120).

[0061] That is, the second classification processing unit 220 in the cleansing processing unit 22 inputs image data of the training data in the dataset DS0 or image data of the training data and test data as input values ​​to the model 0, performs image classification processing by deep learning (S300), and outputs a class C1 for the input data (training data or training data and test data). This classification processing unit is preferably performed on all data of the training data in the dataset DS0, or the training data and test data.

[0062] The comparison processing unit 221 then compares the class C0 corresponding to the image data (data to be classified) that is the original data in the data set DS0 input to the second classification processing unit 220 with the class C1 that is the output result for that image data by the second classification processing unit 220 (S310), and determines whether the class C0 and the class C1 match. If the class C0 and the class C1 do not match, the comparison processing unit 221 excludes the annotation data from the data set DS0 (S320).

[0063] After the comparison process in the comparison processing unit 221, the second data set output processing unit 222 outputs a set of annotation data that has not been excluded as a data set DS1 (S330). At this time, the image data of the face area that has not been excluded and the corresponding annotation data of the product identification information (class C1) may be identified, and this set may be output as the data set DS1.

[0064] If the class C0 obtained by the classification process in the first classification processing unit 202 matches the class C1 obtained by the classification process in the second classification processing unit 220, it is assumed that the annotation accuracy of the annotation data is high. On the other hand, if there is a mismatch, there is a possibility that there is an error in the annotation of the annotation data. Therefore, by excluding annotation data with mismatched classes from the dataset, it is possible to output a dataset DS1 of highly accurate annotation data.

[0065] Then, the learning processing unit 21 executes a learning process in deep learning using each annotation data of the dataset DS1 that has been cleansed and output by the cleansing processing unit 22, and generates a new model 1 (S130).

[0066] Using the model 1 generated as described above, the operation processing unit 23 executes deep learning operation processing (S140). That is, image data to be actually processed (processing target data) is input as an input value to the model 1, deep learning processing is executed, and class C2 for the image data input as an input value is output as an output value.

[0067] In the image classification process performed by the first classification processing unit 202 and the second classification processing unit 220, if the reliability of each class (how reliable the output class is) is output in addition to the class, classification may be performed based on a certain threshold. For example, output results with reliability equal to or greater than a predetermined threshold (reliable class) may be accepted as is, while output results with reliability less than the predetermined threshold (unreliable class) may be shown to the operator. If the recognition result is incorrect, the operator may be prompted to correct the class, or the data itself may be excluded. This allows automatic determination of whether or not annotation data that should be manually corrected needs to be corrected. For example, if the output results of the first-ranked class and the second-ranked class associated with the image data to be classified in the second classification processing unit 220 are equal to or less than a predetermined value, the recognition result may be shown to the operator. [Example]

[0068] When the original data is image data, the image data may contain multiple objects. For example, when executing a process to identify products displayed on a display shelf from image information obtained by photographing the display shelf, multiple images of the products are often included. In such cases, it is advisable to detect the objects from the original data and execute a process to treat the image data containing the objects as data to be classified and processed. This will be explained below.

[0069] An example of the overall configuration of the information processing system 1 in this embodiment is shown schematically in Fig. 8. The information processing system 1 in this embodiment further includes an object detection processing unit 24. A conceptual diagram of an example of the overall processing in this embodiment 2 is shown in Fig. 9, and a conceptual diagram of the flow of data usage is shown in Fig. 10.

[0070] The object detection processing unit 24 detects an object from the image data and cuts out a region including the object. Cutting out a region including the object may involve actually separating or copying the region including the object from the image data to create separate image data, or it may involve specifying the region including the object without separating or copying it from the image data and enabling processing to be performed on that region. The region may be cut out along the contour of the object, or it may be cut out as a rectangular region including the object or a region of any shape including the contour of the object. Image data from which a region including the object has been cut out is called partial image data. As described above, partial image data may be separate image data obtained by separating or copying the region including the object from the image data, or it may be data specifying a region including the object without separating or copying it from the image data. In this case, the partial image data becomes classification target data in the first embodiment and is associated with a class in the annotation data.

[0071] The object detection processing unit 24 can use various known methods to extract objects, for example, by pattern matching with the target object, or by performing deep learning processing using a model that detects objects from image data.

[0072] Next, an example of the processing process in this embodiment will be described with reference to the flowcharts of FIGS.

[0073] First, the information processing system 1 of the present invention executes initial annotation data processing in order to learn a model to be used in deep learning (S100).

[0074] Specifically, a sufficient amount of image data (original data) for use in learning, for example, ten or more per class, is input into the information processing system 1, and the input is accepted by the original data input acceptance processing unit 201 of the initial annotation processing unit 20 (S200).

[0075] Then, the object area is detected from each of the received original data (S205), and partial image data (data to be classified) is extracted from the object area. If multiple objects are detected in the original data, multiple partial image data are extracted from the original data.

[0076] Then, the first classification processing unit 202 executes image classification processing for each partial image data (S210) and outputs a class C0 for each partial image data. The first dataset output processing unit 203 associates each partial image data with the class C0 for each partial image data and outputs a dataset DS0 of the set as initial annotation data (S220).

[0077] In this manner, the initial annotation processing unit 20 outputs the data set DS0 of the initial annotation data based on the partial image data (data to be classified) in the original data.

[0078] Then, as in Example 1, the learning processing unit 21 classifies the data set DS0 of the initial annotation data into learning data to be used in the learning process and test data to be used for testing the model, and performs the learning process in deep learning to generate model 0 (S110).

[0079] Using the model 0 thus generated, the cleansing processing unit 22 executes the cleansing process on the initial annotation data in the data set DS0 (S120).

[0080] That is, the second classification processing unit 220 in the cleansing processing unit 22 inputs partial image data (data to be classified) in the data set DS0, that is, the training data or the training data and the test data, as input values ​​to the model 0, executes image classification processing by deep learning (S300), and outputs a class C1 for the input partial image data (training data or the training data and the test data). This classification processing unit is preferably performed on all of the training data, which is the partial image data (data to be classified) in the data set DS0, or the training data and the test data, but may also perform processing on only a part of it.

[0081] The comparison processing unit 221 then compares the class C0 corresponding to the partial image data in the data set DS0 input to the second classification processing unit 220 with the class C1 of the output result for that partial image data by the second classification processing unit 220 (S310), and determines whether the class C0 and the class C1 match. If the class C0 and the class C1 do not match, the comparison processing unit 221 excludes the annotation data from the data set DS0 (S320).

[0082] After the comparison process in the comparison processing unit 221, the second dataset output processing unit 222 outputs the set of annotation data that has not been excluded as a dataset DS1 (S330).

[0083] The learning processing unit 21 executes a learning process in deep learning using each annotation data of the dataset DS1 that has been cleansed and output by the cleansing processing unit 22, and generates a new model 1 (S130).

[0084] Using the model 1 generated as described above, the operation processing unit 23 executes deep learning operation processing (S140). That is, the object detection processing unit 24 executes object detection processing on the image data to be actually processed, detects an object area from the image data that has been received as input, and extracts partial image data (data to be processed) that extracts the object area. The extracted partial image data is then input as an input value to the model 1, deep learning processing is executed, and class C2 for the image data that has been input as an input value is output as an output value. [Example]

[0085] Next, a case will be described in which the information processing system 1 of the present invention is used in a process of identifying products from image data (photographed image data) of a store's display shelves. In this example, deep learning processing is performed to identify product identification information such as product names and predetermined codes (JAN codes, etc.) of products displayed on the display shelves that are shown in the image data of the display shelves.

[0086] In the information processing system 1 of this embodiment, the operation processing unit 23 has a processing function for identifying products from photographed image data of display shelves. The operation processing unit 23 has a photographed image input reception processing unit 230, an image data processing unit 231, and a product identification processing unit 232. A block diagram of an example of the configuration of the information processing system 1 of this embodiment is shown in Fig. 12.

[0087] The captured image input reception processing unit 230 receives input of captured image data to be processed to execute a process for identifying products displayed on the shelves using a deep learning model.

[0088] The image data processing unit 231 performs correction processing on the captured image data to make it face the camera, such as keystone correction processing, distortion correction processing, etc. The correction processing in the image data processing unit 231 can use various known methods.

[0089] The product identification processing unit 232 inputs the image data of the area where the products are displayed (face area) cut out from the captured image data by the object detection processing unit 24 as input values ​​into model 1 trained using dataset DS1 output by the cleansing processing unit 22, and outputs the product identification information captured in the face area as a class for identification.

[0090] Next, a process of identifying a commodity from photographed image data of a display shelf in a store using the information processing system 1 of the present invention will be described.

[0091] First, we will explain the process of generating a model (Model 1) used in deep learning to identify products.

[0092] In order to generate a model for identifying products using deep learning, image data of display shelves (photographed image data for learning; original data) is prepared in a quantity sufficient for the deep learning learning process, for example, ten or more per class, and input into information processing system 1.

[0093] The raw data input reception processing unit 201 of the initial annotation processing unit 20 receives input of photographed image data for learning as raw data. Then, from each of the received photographed image data, the area of ​​the product displayed on the display shelf is detected, and that area is extracted as image data of the face area (data to be classified).

[0094] Specifically, assume that the captured image data is that shown in FIG. 13. The captured image data shows multiple display shelves, each with multiple products displayed on it. After performing known correction processes such as keystone correction and distortion correction in the image data processing unit 231 on the captured image data of the display shelves, the area of ​​the display shelves where the products are displayed (display shelf area) is identified using a known method. The object detection processing unit 24 then detects an area (face area) including the products from the display shelf area using a known method, and cuts out image data of the face area (data to be classified). FIG. 14 shows an example of the state in which a face area has been identified and cut out from the shelf area identified from the captured image data of FIG. 13. The face area includes the products displayed on the display shelves. In addition to identifying the shelf area and cutting out the face area from there, the face area may also be cut out from the captured image data.

[0095] The image data of the face region thus extracted is input to the first classification processing unit 202, which performs image classification processing to identify and output product identification information (class C0). The image classification processing in the first classification processing unit 202 involves, for example, storing sample image data for each product or data (sample data) of local features based on the sample image data in advance, and comparing the input image data of the face region with the sample data to identify product identification information. Alternatively, product identification information may be identified by deep learning or the like. The image classification processing by the first classification processing unit 202 can identify the product identification information (class C0) of the product depicted in the image data of each face region. The first dataset output processing unit 203 then associates the image data of the face region with the identified product identification information (class C0) to generate annotation data. Furthermore, it outputs a dataset DS0, which is a collection of initial annotation data in which the image data of the face region is associated with the product identification information (class C0) of the identified product.

[0096] Through the above-described processing, the initial annotation processing unit 20 outputs the data set DS0.

[0097] The learning processing unit 21 then classifies the data set DS0 of the initial annotation data into learning data to be used in the learning process and test data to be used for testing the model, and executes the learning process in deep learning to generate model 0.

[0098] Using the model 0 thus generated, the cleansing processing unit 22 executes the cleansing process on the initial annotation data in the data set DS0.

[0099] That is, the second classification processing unit 220 in the cleansing processing unit 22 inputs the image data (learning data or learning data and test data) of each face area in the dataset DS0 as input values ​​to model 0, performs image classification processing using deep learning, and outputs product identification information (class C1) for the image data of the face area.

[0100] Then, the comparison processing unit 221 compares the product identification information (class C0) corresponding to the image data of the face region in the data set DS0 input to the second classification processing unit 220 with the product identification information (class C1) of the output result for that image data by the second classification processing unit 220, and determines whether they are the same product identification information. If they are not the same product identification information, the image data of the face region and the annotation data of the corresponding product identification information (class C0) are excluded from the data set DS0.

[0101] After the comparison process in the comparison processing unit 221, the second data set output processing unit 222 outputs the image data of the face region that has not been excluded and the corresponding annotation data of the product identification information (class C0) as a data set DS1.

[0102] When the cleansing processing unit 22 outputs the dataset DS1 as described above, the learning processing unit 21 executes a learning process in deep learning using each annotation data of the dataset DS1 to generate a new model 1.

[0103] By the above processing, a model 1 for identifying a product can be generated.

[0104] Next, a process (operation process) for identifying products displayed on a store's display shelves that are captured in photographed image data of the display shelves will be described.

[0105] First, when photographed image data to be processed is input to the information processing system 1, the photographed image input reception processing unit 230 of the operation processing unit 23 receives the input. Then, the image data processing unit 231 performs correction processing on the image information from the received input photographed image data, and the object detection processing unit 24 identifies a display shelf area for the photographed image data after correction processing using a known method. Then, the object detection processing unit 24 cuts out image data of a face area including a product from the display shelf area using a known method. Note that in addition to identifying a shelf area and cutting out a face area from it, the face area may also be cut out from the photographed image data.

[0106] The product identification processing unit 232 inputs the image data of the face area cut out in this manner as an input value to the model 1 generated as described above, performs processing using deep learning, and outputs product identification information (class C2) for the face area as an output value.

[0107] By performing the above-described processing, it is possible to accurately identify the product identification information of the products displayed on the display shelves that are captured in the photographed image data to be processed. [Example]

[0108] Although the information processing system 1 of the present invention has been described in the third embodiment as an example of identifying products displayed on a display shelf, the present invention is not limited to this example and can be widely applied to cases where image classification is performed using deep learning on image data to be processed.

[0109] In the above-described first to fourth embodiments, the deep learning learning process and operational process are described using image data as an example, but the data is not limited to image data, and may be sound data or text data. In such cases, the term "image" may be replaced with "sound" or "text" to perform the same process.

[0110] Furthermore, in the case of processing of "images," "sound," and "text," descriptions such as "image data," "sound data," and "text data" do not necessarily refer to image data, sound data, or text data themselves, but may refer to data that represents the image, sound, or text, and may also include data that is somehow related, such as data on features such as local features, or data indicating the location of data. [Industrial Applicability]

[0111] By using the information processing system 1 of the present invention, it becomes possible to automatically cleanse annotation data in deep learning in a simpler manner. [Explanation of symbols]

[0112] 1: Information processing system 20: Initial annotation processing section 21: Learning processing unit 22: Cleansing processing section 23: Operation processing unit 24: Object detection processing unit 70: Arithmetic device 71:Storage device 72:Display device 73: Input device 74:Communication equipment 201: Original data input reception processing unit 202: First classification processing unit 203: First dataset output processing unit 220: Second classification processing unit 221: Comparison processing unit 222: Second dataset output processing unit 230: Photographed image input reception processing unit 231: Image data processing unit 232: Product identification processing unit

Claims

1. An information processing system that performs a cleansing process on annotation data in a dataset that is a collection of annotation data including classification target data and corresponding classes, The information processing system includes: a learning processing unit that performs a deep learning learning process using the dataset to generate a model; a cleansing processing unit that performs a cleansing process on the annotation data in the dataset, The cleansing processing unit includes: the learning processing unit executes the cleansing process on the first annotation data in the first dataset by comparing a second class output by inputting the classification target data of the first annotation data into a first model generated by the learning processing unit using the classification target data of the first annotation data in the first dataset and a first class corresponding to the data, with the first class corresponding to the classification target data of the first annotation data input into the first model; An information processing system comprising:

2. The cleansing processing unit includes: inputting a first class corresponding to data to be classified in first annotation data in a first dataset and the data to be classified into a first model generated by the learning processing unit using the first dataset, and outputting a second class; comparing the first class with the second class and, if a predetermined condition is not satisfied, excluding the first annotation data from the first data set; outputting a second dataset using annotation data that has not been excluded from the first annotation data in the first dataset; 2. The information processing system according to claim 1, wherein:

3. The learning processing unit generating a second model using a dataset of the set of annotation data that has been subjected to the cleansing process; 2. The information processing system according to claim 1, wherein:

4. The information processing system includes: an operation processing unit that inputs data to be analyzed as a processing target and executes deep learning processing on the second model; 4. The information processing system according to claim 3, further comprising:

5. The information processing system includes: an initial annotation processing unit that executes a predetermined classification process on the data to be classified, outputs the first class corresponding to the data to be classified, and outputs the first dataset that is a set of first annotation data in which the data to be classified and the first class are associated with each other; 2. The information processing system according to claim 1, further comprising:

6. the classification process in the initial annotation data processing unit and the classification process in the cleansing processing unit are different classification processes.

6. The information processing system according to claim 5.

7. The classification target data is This is data extracted from a portion of the original data.

7. The information processing system according to claim 1, wherein:

8. An information processing system that performs a cleansing process on annotation data in a dataset that is a collection of annotation data including image data of products and product identification information, The information processing system includes: a learning processing unit that performs a deep learning learning process using the dataset to generate a model; a cleansing processing unit that performs a cleansing process on the annotation data in the dataset, The cleansing processing unit includes: the learning processing unit generates a first model using image data of a commodity in first annotation data in a first data set and first commodity identification information corresponding thereto, and outputs second commodity identification information by inputting the image data of the commodity in the first annotation data into the first model, and compares the second commodity identification information with first commodity identification information corresponding to the image data of the commodity in the first annotation data input into the first model; The learning processing unit generating a second model using a dataset of the set of annotation data that has been subjected to the cleansing process; An information processing system comprising:

9. The information processing system includes: an operation processing unit that inputs image data of products displayed on the display shelves from image data obtained by photographing the display shelves and executes deep learning processing on the second model to identify product identification information of the products; 9. The information processing system according to claim 8, further comprising:

10. Computer, a learning processing unit that generates a model by performing a deep learning learning process using a dataset that is a collection of annotation data including data to be classified and corresponding classes; an information processing program that functions as a cleansing processing unit that executes a cleansing process on annotation data in the dataset, The cleansing processing unit includes: the learning processing unit executes the cleansing process on the first annotation data in the first dataset by inputting the classification target data of the first annotation data into a first model generated by the learning processing unit using the classification target data of the first annotation data in the first dataset and a first class corresponding to the data, and comparing a second class output by the first model with the first class corresponding to the classification target data of the first annotation data input into the first model; An information processing program characterized by:

11. Computer, a learning processing unit that generates a model by performing a deep learning learning process using a dataset that is a collection of annotation data including image data of products and product identification information; an information processing program that functions as a cleansing processing unit that executes a cleansing process on annotation data in the dataset, The cleansing processing unit includes: the learning processing unit generates a first model using image data of a commodity in first annotation data in a first data set and first commodity identification information corresponding thereto, and outputs second commodity identification information by inputting the image data of the commodity in the first annotation data into the first model, and compares the second commodity identification information with first commodity identification information corresponding to the image data of the commodity in the first annotation data input into the first model; The learning processing unit generating a second model using a dataset of the set of annotation data that has been subjected to the cleansing process; An information processing program characterized by:

Images