Information processing system

Abstract

The purpose is to provide an information processing system for efficiently collecting learning data used in machine learning, such as deep learning (deep learning).　The information processing system extracts image data for learning of machine learning from a data group of unlabeled image data. The information processing system comprises: a network processing unit that receives input of image data in a data group and query image data, and outputs a feature quantity of image data input from a predetermined layer in an intermediate layer of a network of machine learning; and an extraction processing unit that extracts image data to serve as learning data from image data that has been output according to a criterion calculated using a feature quantity of the query image data and a feature quantity of the image data in the data group.

Description

Information Processing Systems

[0001] The present invention relates to an information processing system for efficiently collecting learning data used in machine learning such as deep learning. CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority from Patent Application No. 2023-173897, filed October 5, 2023, the entire contents of which are incorporated herein by reference.

[0003] Machine learning, such as deep learning, is a computer system that uses a network trained with training data to perform a specific process, such as image classification. Image classification using machine learning networks has high classification accuracy and has been used for a variety of purposes in recent years. In machine learning, particularly deep learning, image data is input into a learning model (network) in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized, and a specific output is produced.

[0004] On the other hand, it is usually not possible to input unlearned image data as a processing target. Even if it is input, it is difficult to determine that it is an unlearned target, and it will be determined to be in a different category with extremely poor judgment accuracy.

[0005] Therefore, when processing untrained image data, it is necessary to perform machine learning again using training data. In this case, the training data to be used for training must be collected appropriately, but conventionally, this training data has been collected by humans visually selecting from a large amount of image data.

[0006] Systems for collecting learning data are disclosed in the following Patent Documents 1 and 2.

[0007] JP 2020-071811 A JP 2023-104040 A

[0008] The collection method of Patent Document 1 merely describes how to generate learning data, and is not capable of efficiently collecting image data to be used as learning data.

[0009] Furthermore, in the collection method of Patent Document 2, learning data is collected using an inference process, so unless the inference accuracy is high, it is not possible to collect appropriate image data. If appropriate image data cannot be collected, it is not possible to improve the accuracy of machine learning, and it is not easy to use it in actual machine learning.

[0010] For example, machine learning may be used to recognize products displayed on store shelves from image data of the shelves. In this case, image data of the products to be processed is trained in advance as training data, and the trained network is used to perform product recognition processing.

[0011] Furthermore, if it is desired to recognize new products or products that have not been recognized before, it is necessary to generate a machine learning network by training the network with new image data of the products to be recognized as training data.When training using image data of new products to be recognized, it is possible to obtain image data in a clear state (noise-free, such as no light or shadow) as a sample from the manufacturer selling the product, etc., and use this as training data.

[0012] However, there is a problem that the accuracy of image classification (image recognition) decreases when clean image data acquired as a sample is used as training data. That is, the input images are taken of actual store shelves, and the products in that state generally contain noise such as light and shadow. Therefore, even if clean image data is used as training data, there is a limit to the improvement in accuracy.

[0013] One possible solution is to use image data of product areas extracted from image data of actual store shelves as training data, but this image data is generally not labeled, so it is not possible to search for image data of new products to be processed by product name, etc.

[0014] In the past, humans would visually identify image data from a large amount of unlabeled image data and use it as training data, but this involved a significant workload.

[0015] Therefore, there is a need to efficiently collect image data of products that are the target of new processing from a large amount of unlabeled image data.

[0016] In view of the above-mentioned problems, an information processing system has been disclosed that efficiently collects image data to be used as learning data for machine learning from a data set of unlabeled image data.

[0017] An information processing system according to one aspect of the present disclosure is an information processing system that extracts image data for machine learning training from a data group of unlabeled image data, and the information processing system has a network processing unit that accepts input of image data in the data group and query image data, and outputs features of the input image data from a predetermined layer in the intermediate layer of a machine learning network, and an extraction processing unit that extracts image data to be used as training data from the output image data based on criteria calculated using the features of the query image data and the features of the image data in the data group.

[0018] By using the information processing system described above, it is possible to efficiently collect image data of the target product to be newly processed from a large amount of unlabeled image data.

[0019] In the information processing system according to one aspect described above, the information processing system has a similarity processing unit that calculates similarity using features of the query image data and features of image data in the data group, and the extraction processing unit can be configured to accept selection of image data to be used as learning data from the image data output based on the similarity, and extract the selected image data as the learning data.

[0020] In the information processing system according to the above-described aspect, the network processing unit can be configured to extract a feature vector of the input image data from a predetermined layer in the intermediate layer of the machine learning network, and the similarity processing unit can be configured to calculate the distance between the feature vector of the query image data and the feature vector of the image data in the data group as the similarity, sort the image data in the data group using the calculated similarity, and output image data up to a predetermined rank.

[0021] The sorted and output image data may include image data that is not the target of machine learning. Therefore, as disclosed above, by sorting the image data in the data group in order of similarity to the image data to be the target of machine learning, it becomes possible to output image data based on similarity. This makes it easier to select image data to be the target of machine learning. In particular, when an operator makes a selection, the image data is displayed in a group, making it easier to select image data to be the target of machine learning.

[0022] In the information processing system according to the above-described aspect, the similarity processing unit can be configured to calculate the cosine similarity between the feature vector of the query image data and the feature vector of the image data in the data group as the similarity.

[0023] In the above-mentioned information processing system, by using cosine similarity as a calculation method for similarity determination, even when there are hundreds of thousands of image data to be compared and calculations are performed in a brute-force manner, sufficient processing speed can be ensured, thereby reducing processing time.

[0024] In the information processing system according to the above aspect, the information processing system can be configured to include an image input processing unit that inputs image data in the data group and the query image data.

[0025] In the information processing system according to one aspect described above, if the number of learning data extracted by the extraction processing unit does not satisfy a predetermined condition, the image input processing unit can be configured to further input the extracted learning data to the machine learning network as new query image data, the network processing unit accepts the input of the new query image data and outputs features of the input new query image data from a predetermined layer in the intermediate layer of the machine learning network, and the extraction processing unit can be configured to further extract image data to be used as learning data from the image data output according to a criterion calculated using the features of the new query image data and the features of image data in the data group.

[0026] If a group of unlabeled image data contains only a small amount of image data to be used in machine learning, it may not be possible to extract a sufficient number of image data to be used in machine learning in a single process. Therefore, by inputting the extracted learning data into the network and performing the same process, it is possible to extract image data to be used in machine learning in a chain reaction.

[0027] In the information processing system according to the above aspect, the predetermined layer may be configured to be one layer and / or two layers before the final layer in the intermediate layer.

[0028] According to experiments conducted by the present inventor, it has been found that image data can be extracted with high accuracy by performing processing using the feature amounts output from these layers.

[0029] This can be realized by loading and executing an information processing program according to the present disclosure into a computer. That is, the information processing program causes the computer to function as a network processing unit that accepts input of image data in a data group of unlabeled image data and query image data, and outputs feature quantities of the input image data from a predetermined intermediate layer of a machine learning network, and an extraction processing unit that extracts image data to be used as learning data from the output image data according to a criterion calculated using the feature quantities of the query image data and the feature quantities of the image data in the data group.

[0030] By using the information processing system of the present disclosure, it is possible to efficiently collect image data to be used as learning data for machine learning from a data group of unlabeled image data.

[0031] FIG. 1 is a diagram schematically showing an example of the overall concept of an information processing system of the present disclosure. FIG. 2 is a block diagram schematically showing an example of the configuration of an information processing system of the present disclosure. FIG. 3 is a block diagram schematically showing an example of the hardware configuration of a computer used in the information processing system of the present disclosure. FIG. 4 is a flowchart showing an example of the overall processing in the information processing system of the present disclosure. FIG. 5 is a diagram schematically showing an example of a state in which unlearned image data similar to query image data has been sorted. FIG. 6 is a diagram schematically showing processing in which selected learning data is used as new query image data.

[0032] An example of the overall concept of the information processing system 1 of the present disclosure is shown schematically in Fig. 1. An example of the overall processing function of the information processing system 1 of the present disclosure is shown in a block diagram in Fig. 2. The information processing system 1 uses a management terminal 2 and an operator terminal 3. The management terminal 2 is a computer used by an organization such as a company that operates the information processing system 1. The operator terminal 3 is a computer operated by an operator.

[0033] The management terminal 2 and the operator terminal 3 in the information processing system 1 are realized using a computer. An example of the hardware configuration of a computer is shown in Figure 3. The computer has a calculation device 70 such as a CPU that executes the calculation processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or a mouse that can input information, and a communication device 74 that sends and receives the processing results of the calculation device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0034] If the computer is equipped with a touch panel display, the display device 72 may be integrally configured with the input device 73. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited to these.

[0035] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

[0036] The operator terminal 3 may be equipped with a photographing device such as a camera in addition to the above-mentioned devices. A portable communication terminal such as a mobile phone, a smartphone, or a tablet computer can also be used as the operator terminal 3. The operator terminal 3 photographs a predetermined subject and inputs image data of the photographed subject (photographed image data) to the management terminal 2. For example, the operator terminal 3 photographs a display shelf in a store and inputs image data of the display shelf (photographed image data) to the management terminal 2. In this case, products are displayed on the display shelf, and the display shelf is photographed.

[0037] The functions of the means in the present disclosure are only logically distinct, and may be physically or practically in the same area. The order of the processes in the means in the present disclosure may be changed as appropriate. Also, some of the processes may be omitted. For example, the normalization process described below may be omitted. In that case, the process may be performed on image data that has not been normalized.

[0038] The management terminal 2 of the information processing system 1 has an image input processing unit 20 , a network processing unit 21 , a similarity processing unit 22 , an extraction processing unit 23 , an unlearned image storage unit 24 , and a learning image storage unit 25 .

[0039] The image input processing unit 20 inputs image data to a machine learning network in the network processing unit 21 (described later). For example, it inputs unlearned image data (unlabeled image data) stored in the unlearned image storage unit 24 (described later) to the network, and inputs query image data to be used as a learning target for machine learning to the network. The query image data is image data of an object to be used as a learning target for machine learning, and is used as image data to be input to the network processing unit 21. The query image data initially input to the network in the network processing unit 21 is often clear image data (noise-free sample image data) of the object to be used as a learning target for machine learning, but is not limited to this. Furthermore, as described later, as query image data to be input to the network of the network processing unit 21 from the second time onwards, some or all of the image data extracted by the extraction processing unit 23 (learning data extracted as learning data) can be used as new query image data.

[0040] The network processing unit 21 executes machine learning processing. Image data received from the image input processing unit 20 is input to a network used in the machine learning processing (a network for image classification that has been trained in advance), and the machine learning processing is executed. The network in the network processing unit 21 is a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized. For the image data received as input, the network processing unit 21 executes normal image classification processing in the network at each layer, i.e., the input layer, intermediate layer, and output layer. Note that the output results output from the output layer are not used and are therefore discarded.

[0041] The network processing unit 21 outputs the processing results of a predetermined intermediate layer, rather than the output layer of the network, and stores them as features of the input image data, such as a feature vector. As an example, if the trained network used in the network processing unit 21 is mobilenetv2, the results of the Bottleneck residual block layer, which is the intermediate layer one layer before and / or two layers before the final layer, are output and stored as a feature vector of the input image data. The intermediate layer used for output can be any layer, and is not limited to the Bottleneck residual block layer described above.

[0042] This is performed for each piece of unlearned image data, and the feature vectors of each piece of unlearned image data are output.

[0043] Similarly, the network processing unit 21 inputs query image data into the network and executes machine learning processing. Then, for the received query image data, the network processing unit 21 executes normal image classification processing in the network at each layer, i.e., the input layer, intermediate layer, and output layer, outputs the processing result of a predetermined intermediate layer, and stores it as a feature vector of the input query image data. Note that the output result output from the output layer is not used and is therefore discarded. For example, if the trained network used by the network processing unit 21 is mobilenetv2, the result of the Bottleneck residual block layer is output and stored as a feature vector of the query image data.

[0044] The similarity processing unit 22 calculates the similarity between the query image data and each piece of image data output by the network processing unit 21. As an example of the similarity calculation process, the similarity processing unit 22 calculates the distance (e.g., cosine similarity) between the feature amount (e.g., feature vector) of the query image data and the feature amount (e.g., feature vector) of the image data output by the network processing unit 21. Then, for each piece of image data, the similarity processing unit 22 calculates the cosine similarity with the query image data, sorts the image data in descending order from highest to lowest distance (e.g., cosine similarity), and outputs the image data in a predetermined order, for example, up to 300th place. This allows image data similar to the query image data to be sorted in descending order. The similarity processing unit 22 then displays the image data on the display device 72 in the sorted order. Note that the similarity processing unit 22 preferably displays the image data in the sorted order because this facilitates processing by the extraction processing unit 23, which will be described later, but the display in the sorted order is not necessarily required.

[0045] The extraction processing unit 23 accepts the selection of image data to be used as learning data from the image data displayed on the display device 72, extracts the selected image data as learning data, and stores it in the learning image storage unit 25 (described later). Machine learning is performed on the image data stored here as learning data. This machine learning may be performed on the network used by the network processing unit 21 or on a separate network.

[0046] The unlearned image storage unit 24 is a storage area for storing unlearned image data.

[0047] The learning image storage unit 25 is a storage area that stores image data (learning data used in machine learning) selected as a learning target for machine learning from unlearned image data.

[0048] Next, an example of processing of the information processing system 1 according to an embodiment of the present disclosure will be described with reference to the flowchart of FIG.

[0049] The network processing unit 21 already has annotated image data, and executes a process of classifying images using a network trained by machine learning. Furthermore, when unlearned data is input to this network as a processing target, if the confidence level in the output result output from the network is lower than a predetermined threshold, the image data is determined to be unclassified, and the image data determined to be unclassified is preferably stored in the unlearned image storage unit 24. Unlearned image data, for example, image data of more than a thousand images, for example, image data of several thousand to tens of thousands or hundreds of thousands of images, is stored in the unlearned image storage unit 24 in advance, and from this image data, data is extracted as learning data to be used in the machine learning network and stored in the learning image storage unit 25.

[0050] In the following description of this specification, the network in the network processing unit 21 will be described as a network that recognizes product identification information (image classification), such as product names and JAN codes, when the image data of products displayed on the shelves, extracted from image data of the shelves, depicts a product that has been previously machine-learned. Furthermore, the data group of unlearned image data stored in the unlearned image storage unit 24 will be image data of products displayed on the shelves, extracted from image data of the shelves, that has not yet been machine-learned by the network. For example, image data of products that have not yet been subjected to image classification is one example. Image data of objects to be trained can be image data of a new training object to be trained, such as image data of a new product. Therefore, the purpose of the processing of the information processing system 1 of the present disclosure is to extract image data of the same product as the training object from the data group of unlearned product image data stored in the unlearned image storage unit 24 and store it in the training image storage unit 25.

[0051] It should be noted that this specification is not limited to the above-mentioned examples, but rather any network or object may be used as long as it is a network that performs some kind of image classification and is used to extract image data of an object to be learned from a data group of unlearned image data as learning data, and there are no restrictions on these.

[0052] The image input reception processing unit inputs image data (unlearned image data) of various products stored in the unlearned image storage unit 24 to the network in the network processing unit 21 (S100). The image input reception processing unit also receives input of query image data of products to be learned, and inputs the query image data to the network in the network processing unit 21 (S100).

[0053] The network processing unit 21 executes processing in a machine learning network for each image data of various products and query image data received from the image input reception processing unit. Then, the processing results of a predetermined intermediate layer of the network are output as feature vectors for each input image data. That is, the network processing unit 21 outputs the feature vectors for each unlearned image data and the feature vectors for the query image data in a predetermined intermediate layer of the network (S110).

[0054] The similarity processing unit 22 then calculates the distance between the feature vector of the query image data and the feature vector of each unlearned image data as the similarity between the image data (S120). For example, it calculates the cosine similarity between the feature vector of the query image data and the feature vector of each unlearned image data. This distance calculation is performed between the query image data and each unlearned image data.

[0055] After calculating the similarity between the query image data and each piece of unlearned image data, the similarity processing unit 22 sorts the unlearned image data in descending order of similarity, and outputs the unlearned image data in a predetermined order, for example, up to 300th place, to the display device 72 of the operator terminal 3 (S130). That is, in this state, the unlearned image data are displayed in descending order of similarity to the query image data. An example of this state is schematically shown in FIG. 5. FIG. 5(a) shows an example of query image data of an object to be learned, i.e., a new product to be machine-learned, and FIG. 5(b) shows an example of unlearned image data sorted by similarity.

[0056] The operator then selects image data from the unlearned image data displayed on the display device 72 that shows the object to be learned, in this example, a new product for which machine learning is desired. At this time, to improve the accuracy of machine learning, it is advisable to select a variety of image data of the same object. For example, it is advisable to select image data showing the product in various orientations, shading, etc.

[0057] When the selection receiving processor receives this selection (S140), the image data for which the selection has been received is stored as learning data in the learning image storage unit 25 (S150).

[0058] If there is not enough training data for machine learning (S160), some or all of the image data selected in S140 (image data selected as training data from among untrained image data) is input to the network in the network processing unit 21 as new query image data, and the processes from S100 onwards are executed again (S170). That is, the processes from S100 to S170 are repeated until a sufficient number of image data for training data is obtained for machine learning. This process is schematically shown in FIG. 6.

[0059] By performing the above processing, image data of the object to be learned can be extracted from a large amount of unlearned image data, and stored in the learning image storage unit 25 as learning data.

[0060] In addition to storing the extracted learning data in the learning image storage unit 25, the extracted unlearned image data may be flagged as learning data, so that it can be recognized as learning data.

[0061] The learning data extracted and stored in this way can be used for additional learning in the network, or for transfer learning in another network.

[0062] By using the information processing system 1 of the present disclosure, it is possible to efficiently collect image data to be used as learning data for machine learning from a data group of unlabeled image data.

[0063] 1: Information processing system 2: Management terminal 3: Operator terminal 20: Image input processing unit 21: Network processing unit 22: Similarity processing unit 23: Extraction processing unit 24: Unlearned image storage unit 25: Learning image storage unit 70: Arithmetic unit 71: Storage device 72: Display device 73: Input device 74: Communication device

Claims

1. An information processing system that extracts image data for machine learning training from a data group of unlabeled image data, the information processing system having: a network processing unit that accepts input of image data and query image data in the data group, and outputs features of the input image data from a specified layer in an intermediate layer of a machine learning network; and an extraction processing unit that extracts image data to be used as training data from the output image data based on criteria calculated using the features of the query image data and the features of the image data in the data group.

2. The information processing system of claim 1, further comprising a similarity processing unit that calculates a similarity using features of the query image data and features of image data in the data group, and the extraction processing unit that accepts a selection of image data to be used as learning data from the image data output based on the similarity, and extracts the selected image data as the learning data.

3. The information processing system of claim 2, wherein the network processing unit extracts a feature vector of the input image data from a predetermined layer in the intermediate layers of the machine learning network, and the similarity processing unit calculates the distance between the feature vector of the query image data and the feature vector of image data in the data group as the similarity, and sorts the image data in the data group using the calculated similarity, and outputs image data up to a predetermined rank.

4. The information processing system according to claim 3, wherein the similarity processing unit calculates, as the similarity, a cosine similarity between a feature vector of the query image data and a feature vector of image data in the data group.

5. The information processing system according to claim 1, further comprising: an image input processing unit that inputs image data in the data group and the query image data.

6. The information processing system of claim 5, wherein the image input processing unit, if the number of learning data extracted by the extraction processing unit does not satisfy a predetermined condition, further inputs the extracted learning data to the machine learning network as new query image data; the network processing unit accepts the input of the new query image data and outputs features of the input new query image data from a predetermined layer in an intermediate layer of the machine learning network; and the extraction processing unit further extracts image data to be used as learning data from the output image data based on a criterion calculated using the features of the new query image data and the features of image data in the data group.

7. An information processing system according to any one of claims 1 to 6, wherein the predetermined layer is a layer one layer and / or two layers before a final layer in the intermediate layer.

8. An information processing program that causes a computer to function as: a network processing unit that accepts input of image data in a data group of unlabeled image data and query image data, and outputs features of the input image data from a predetermined layer in the intermediate layer of a machine learning network; and an extraction processing unit that extracts image data to be used as learning data from the output image data based on criteria calculated using the features of the query image data and the features of image data in the data group.

Images