Information processing system

Abstract

An object of the present invention is to provide an information processing system related to generation of a learning model (network) in machine learning. 【Means for solving the problem】 An information processing system that executes processing related to a learning model used in machine learning, using a first learning model generated using object image information and provisional identification information associated with the object image information, a first processing unit that identifies object identification information corresponding to the provisional identification information; a second processing unit that performs verification processing on an output result of object identification information corresponding to the image information input to the first learning model; and a second model generation processing unit that executes learning processing of machine learning using the learning data verified by the second processing unit to generate a second learning model.

Description

Technical Field

[0001] The present invention relates to an information processing system for generating a learning model (network) in machine learning.

Background Art

[0002] In recent years, machine learning has been widely used as image analysis processing using a computer. In machine learning, a learning dataset (learning data) is created in advance, and the computer is made to read it and execute predetermined learning processing.

[0003] However, preparing the learning data in advance itself has a large workload. Therefore, systems as disclosed in Patent Document 1 and Patent Document 2 below are disclosed.

[0004] In the invention of Patent Document 1, a product image is photographed, and the code attached to the product is read to perform product image and its labeling, thereby generating learning data.

[0005] Also, in the invention of Patent Document 2, from an image database storing image data including an object whose information related to classification has been labeled in advance, the classification of the object included in the image data is selected, and the object is extracted from the image data to generate a template image, thereby generating learning data with the template image and labeling.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] In the case of the invention of Patent Document 1, learning data is generated by preparing a product for shooting and reading the code attached to the product with a code reader, which imposes a heavy workload on the operator. When there are more products to be read as learning data, the workload becomes enormous.

[0008] In the case of the invention of Patent Document 2, the process cannot be performed unless an image database storing image data including an object pre-labeled with information regarding classification is prepared.

[0009] In a retail store or the like, recognizing the products displayed on the display shelves and grasping the display situation thereof are very important as a corporate marketing strategy. Therefore, there is a desire to grasp the situation in which the company's own products or the products of competing companies are displayed on the display shelves of retail stores.

[0010] Therefore, there is a system that photographs the display shelves of a retail store and identifies the products displayed on the display shelves, and machine learning is used for this product identification. Also in this machine learning, as described above, learning processing is executed using learning data, but the products displayed on the display shelves are diverse. Also in this case, conventionally, a learning model has been generated using learning data in which the displayed products and their identification information are associated as labels.

[0011] However, even in this case, the burden of preparing the learning data is large, and the burden required for generating the learning model is large.

Means for Solving the Problems

[0012] In view of the above problems, the present inventor has invented an information processing system that efficiently generates a learning model used in machine learning.

[0013] The first invention is an information processing system that executes processing related to a learning model used in machine learning. The information processing system includes a first processing unit that identifies object identification information corresponding to the temporary identification information using a first learning model generated using object image information and temporary identification information associated with the object image information; a second processing unit that performs verification processing on an output result of the object identification information corresponding to the image information input to the first learning model; and a second model generation processing unit that generates a second learning model by executing learning processing of machine learning using the learning data verified by the second processing unit.

[0014] By configuring as in the present invention, a learning model used in machine learning can be efficiently generated.

[0015] In the above invention, the second processing unit may be configured as an information processing system including an image information reception processing unit that receives an input of image information, a second recognition processing unit that inputs the received image information to the first learning model and outputs at least one piece of temporary identification information corresponding to the image information, a second output processing unit that outputs object identification information corresponding to the output temporary identification information, a verification processing unit that extracts the image information of the output object identification information from a predetermined storage area, and performs verification processing on the object identification information of the input image information using the extracted image information and its object identification information, and uses the verified object identification information and the image information as learning data.

[0016] As in the present invention, since not only the first processing but also the verification of the output result is performed in the second processing, a learning model with higher accuracy can be efficiently generated.

[0017] In the above invention, the second recognition processing unit may be configured as an information processing system that outputs temporary identification information corresponding to the image information and a recognition coefficient, the second output processing unit outputs object identification information corresponding to the output temporary identification information, and sorts the object identification information based on the recognition coefficient.

[0018] Sorting the output results from the first learning model enables rapid verification processing.

[0019] In the above invention, the verification processing unit can be configured as an information processing system that performs the verification processing by displaying the extracted object identification information and its image information, and receiving, from a computer used by a verifier, a selection of the correct result of the image information input to the first learning model.

[0020] When the verification processing is performed by having a verifier check the output results, the accuracy is enhanced.

[0021] In the above invention, the verification processing unit can be configured as an information processing system that performs the verification processing by comparing the image information corresponding to the extracted object identification information with the image information stored in a predetermined storage area.

[0022] In the above invention, the verification processing unit can be configured as an information processing system that performs verification processing to determine the object identification information of the sample information determined to have the highest similarity as the object identification information of the image information input to the first learning model by comparing the image information corresponding to the extracted object identification information with the sample information stored in a predetermined storage area.

[0023] When a verifier performs the verification processing, a burden is incurred. Therefore, it may be automated as in the present invention.

[0024] In the above invention, the first processing unit can be configured as an information processing system having a classification processing unit that classifies image information of a plurality of objects, a provisional identification information processing unit that associates the classified group with provisional identification information, a first model generation processing unit that executes a learning process of machine learning using learning data including the image information of the objects included in the group and the provisional identification information to generate the first learning model, a first recognition processing unit that outputs provisional identification information by inputting the received sample information into the first learning model, and a first output processing unit that associates the output provisional identification information with the object identification information corresponding to the sample information.

[0025] By configuring as in the present invention, a learning model used in machine learning can be efficiently generated.

[0026] In the above invention, the information processing system can be configured as an information processing system that outputs, as an output value, object identification information corresponding to the provisional identification information output by the learning model by inputting the image information to be the target of the identification process into the learning model.

[0027] By configuring as in the present invention, when image information of an object to be identified is input to the learning model, object identification information can be output.

[0028] In the above invention, the information processing system can be configured as an information processing system having a cleansing processing unit that executes a cleansing process of excluding object image information that may contain errors from the image information of the objects included in the classified group.

[0029] By executing the cleansing process as in the present invention, it is possible to exclude object image information that may contain errors from the object image information included in the group, and improve the accuracy of image classification.

[0030] In the above invention, the cleansing processing unit can be configured as an information processing system that determines the image information of the object to be excluded by using the information distance from the reference value calculated using the index value of the image information of the objects included in the group.

[0031] In the above invention, the cleansing processing unit calculates a reference value using the index value of the image information of the objects included in the group, calculates an information distance using the reference value and the index value of the image information of the object, and when the information distance has a deviation of a predetermined threshold value or more or a predetermined ratio or more, determines the image information of the object as the image information of the object to be excluded, and can be configured as an information processing system.

[0032] In the above invention, the cleansing processing unit can be configured as an information processing system that determines the image information of the object to be excluded by using the similarity of the image information of the objects included in the group.

[0033] In the above invention, the cleansing processing unit calculates the similarity between the image information of the objects included in the group and the image information of other objects included in the group, and when the predetermined conditions using the similarity are satisfied, determines the image information of the object as the image information of the object to be excluded, and can be configured as an information processing system.

[0034] The cleansing process can be executed as in these inventions.

[0035] In the above invention, the object is a product displayed on a display shelf, the classification processing unit cuts out the image information of the products displayed from the image information of the display shelf and extracts it as object image information, compares the similarity of the object image information of adjacent products, and when the similarity satisfies certain conditions, classifies the object image information of adjacent products into the same group, and can be configured as an information processing system.

[0036] On display shelves, products of the same type are often displayed adjacent to each other. Therefore, if the object is a product displayed on a display shelf, the object image information of the adjacent products may be classifiable into the same type, that is, the same group. Thus, it is preferable to compare the similarity of the object image information of the adjacent products and classify them into the same group if the condition is satisfied.

[0037] In the above invention, the object is a product displayed on a display shelf, and the first recognition processing unit inputs the specimen information of the product corresponding to the received sales amount into the learning model, and outputs the provisional identification information corresponding to the specimen information received by the learning model, and can be configured as an information processing system.

[0038] In the case of a learning model for identifying products displayed on a display shelf, the number of products may be in the thousands to tens of thousands. In that case, it is a heavy burden to input the specimen information of all products.

[0039] Generally, a small number of best-selling products account for most of the sales. And there is a high possibility that best-selling products are displayed on the display shelf. Therefore, if the best-selling products with a high possibility of being the identification target of the product are input as specimen information, even if all the products displayed on the display shelf cannot be identified, many products can be recognized based on the sales amount, and a practically sufficient system can be constructed.

[0040] In the above invention, the object is a product displayed on a display shelf, and the first recognition processing unit inputs the specimen information of the product for each organization received into the learning model, and outputs the provisional identification information corresponding to the specimen information received by the learning model, and can be configured as an information processing system.

[0041] When the present invention is used for marketing, there may be a case where a specific organization such as a certain company wants to grasp the display status of products of its own company or competing companies. Therefore, if the sample information of products for each organization is input as in the present invention, even if it is not possible to identify all the products displayed on the display shelves, important marketing information can be obtained for that organization, and a practically sufficient system can be constructed.

[0042] The first invention can be realized by causing a computer to read and execute the program of the present invention. That is, the computer is caused to use a first learning model generated using object image information and provisional identification information associated with the object image information to specify object identification information corresponding to the provisional identification information, a second processing unit that performs verification processing on the output result of the object identification information corresponding to the image information input to the first learning model, and a second model generation processing unit that executes learning processing of machine learning using the learning data verified by the second processing unit to generate a second learning model, and it is an information processing program that functions as such.

Effect of the Invention

[0043] By using the information processing system of the present invention, a learning model used in machine learning can be efficiently generated.

Brief Explanation of the Drawings

[0044]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Mode for Carrying Out the Invention

[0045] An example of a conceptual diagram of the processing of the information processing system 1 of the present invention is shown in FIGS. 1 to 4. Further, an example of the processing function of the information processing system 1 of the present invention is shown in a block diagram in FIG. 5. The information processing system 1 is a system that generates a model for recognizing an object such as a product displayed on a display shelf from image information using a machine learning model. As an example of the object, the case of a product displayed on a display shelf will be described, but it is not limited thereto.

[0046] The management terminal 2 in the information processing system 1 is realized using a computer. An example of the hardware configuration of the computer is schematically shown in FIG. 6. The computer includes a CPU that executes arithmetic processing of programs, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or a mouse through which information can be input, and a communication device 74 that transmits and receives the processing results of the arithmetic device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0047] When the computer is equipped with a touch panel display, the display device 72 and the input device 73 may be integrally configured. The touch panel display is often used in portable communication terminals such as tablet computers and smartphones, but is not limited thereto.

[0048] The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that input can be directly performed on the display using a predetermined input device (such as a pen for the touch panel) or a finger.

[0049] Each means in the present invention only has its functions logically distinguished, and may physically or de facto form the same area. The processing order in each means of the present invention can be appropriately changed. Also, a part of the processing may be omitted.

[0050] The management terminal 2 includes a first processing unit 20, a first learning model storage unit 21, an image information storage unit 22, a second processing unit 23, a learning information storage unit 24, a second model generation processing unit 25, and a second learning model storage unit 26.

[0051] The first processing unit 20 generates a learning model (first learning model) using, for example, the image information of an object (object image information) shown in the image information such as an image of a display shelf, etc., and temporary identification information (described later) associated with the object image information, and specifies the object identification information corresponding to the temporary identification information. The first processing unit 20 includes a classification processing unit 200, a temporary identification information processing unit 201, a first model generation processing unit 202, a specimen information reception processing unit 203, a first recognition processing unit 204, and a first output processing unit 205.

[0052] The classification processing unit 200 classifies the object image information by means of clustering processing or the like from the image information of an object (object image information) shown in the image information such as an image of a display shelf, etc. As the object image information, in the case where the object is a product displayed on a display shelf, it may be the image information of the product. For example, from the image information showing the display shelf, it may be the image information obtained by cutting out the area including the displayed product that is the object, whether it is cut out in a rectangle at this time or the image information cut out in an arbitrary shape according to the outer shape of the product.

[0053] The identification information of the object (object identification information) shown in the object image information, for example, in the case where the object is a product, the product name or codes such as JAN codes do not have to be known.

[0054] The classification processing unit 200 executes clustering processing on a plurality of object image information by a known method to classify the image information into a plurality of groups.

[0055] Note that although the case where the classification processing unit 200 classifies the object image information using clustering processing is described, the object image information may be classified by a method other than clustering processing.

[0056] The temporary identification information processing unit 201 associates temporary identification information (temporary ID information) with each group classified by the classification processing unit 200. The temporary ID information may be associated with automatically generated temporary ID information, or may be associated with the input temporary ID information through a predetermined operation. Then, the temporary ID information associated with the object image information classified into each group is associated as a label to form a dataset, which is used as learning data.

[0057] The first model generation processing unit 202 performs machine learning by a known method using the learning data to generate a learning model. That is, since temporary ID information is associated with each group classified in the object image information, a learning model (first learning model) is generated using the learning data that uses the object image information and its label as the temporary ID information as a dataset.

[0058] The first model generation processing unit 202 stores the generated learning model (first learning model) in the first learning model storage unit 21.

[0059] The specimen information reception processing unit 203 receives the input of the image information to be input to the first model generation processing unit 202. The image information received here is object image information for which the object identification information for identifying the object is known. For example, it is image information of a product that may be displayed on a display shelf, and the product identification information (object identification information) such as the product name and the code such as the JAN code corresponding to the product is known. The image information input here is referred to as specimen information.

[0060] The first recognition processing unit 204 inputs the image information (specimen information) received by the specimen information reception processing unit 203 into the first learning model stored in the first learning model storage unit 21 and outputs temporary ID information. In this case, when the specimen information is input to a learning model in which the weighting coefficients between the neurons of each layer of a neural network having multiple intermediate layers are optimized, the temporary ID information is output as an output value. When a plurality of output values are output together with their recognition coefficients (reliability), the output value with the highest recognition coefficient (reliability) is adopted.

[0061] The first output processing unit 205 associates the provisional identification information, which is the output value output by the first recognition processing unit 204, with the object identification information of the sample information used for the input. Then, the first output processing unit 205 outputs the object identification information associated with the provisional identification information output by the first recognition processing unit 204 as the recognition result of the first recognition processing unit 204.

[0062] Note that instead of outputting the object identification information associated with the provisional identification information output by the first recognition processing unit 204, the first output processing unit 205 may replace the object identification information with the provisional identification information associated with the group output by the classification processing unit 200, and cause the first model generation processing unit 202 to execute the learning process of machine learning again.

[0063] Also, when the recognition coefficient (confidence level) of the provisional identification information output by the first recognition processing unit 204 is equal to or less than a predetermined threshold value, the processing of the first output processing unit 205 may not be executed.

[0064] By executing the above-described processing in the first processing unit 20, a learning model (first learning model) used in machine learning can be efficiently generated.

[0065] The image information storage unit 22 stores the image information and the object identification information corresponding thereto. For example, it stores the object image information shown in the image information such as the image information obtained by photographing a display shelf, and the object identification information corresponding thereto. Note that the image information storage unit 22 may store provisional identification information in addition to the object identification information. The image information storage unit 22 may store the sample information received by the sample information reception processing unit 203.

[0066] The second processing unit 23 inputs the image information to the first learning model stored in the first learning model storage unit 21, verifies the output value for the input, and uses it as learning data for generating a second learning model in a second model generation processing unit 25 described later. The second processing unit 23 includes an image information reception processing unit 230, a second recognition processing unit 231, a second output processing unit 232, and a verification processing unit 233.

[0067] The image information reception processing unit 230 receives the input of the image information to be input to the first learning model stored in the first learning model storage unit 21. Here, the image information for which the input is received may be the object image information not used for the classification process in the classification processing unit 200, or may be other image information. For example, it may be the object image information shown in the image information such as the image information obtained by photographing a display shelf. The image information stored in the image information storage unit 22 may be the image information used in the process of the second processing unit 23.

[0068] The second recognition processing unit 231 inputs the image information received by the image information reception processing unit 230 to the first learning model stored in the first learning model storage unit 21. The first learning model outputs at least one or more provisional identification information and its recognition coefficient (reliability) as an output value for the input image information. In this case, when image information is input to the learning model (first learning model) in which the weighting coefficients between the neurons of each layer of the neural network having a large number of intermediate layers are optimized, provisional identification information and its recognition coefficient (reliability) are output as output values.

[0069] The second output processing unit 232 associates the corresponding object identification information with the provisional identification information that is the output value output by the second recognition processing unit 231. Then, the second output processing unit 232 outputs the object identification information associated with the provisional identification information output by the second recognition processing unit 231 and the recognition coefficient (reliability) as the recognition result of the second recognition processing unit 231.

[0070] Also, when the recognition coefficient (reliability) of the provisional identification information output by the second recognition processing unit 231 is equal to or less than a predetermined threshold value, the process of the second output processing unit 232 may not be executed. Further, the second recognition processing unit 231 or the second output processing unit 232 may output the provisional identification information or the object identification information having a recognition coefficient (reliability) equal to or more than a predetermined recognition coefficient (reliability).

[0071] Furthermore, the second output processing unit 232 sorts and outputs in descending order of the recognition coefficient (reliability) based on the recognition coefficient (reliability).

[0072] For example, when the second recognition processing unit 231 inputs the image information received by the image information reception processing unit 230 into the first learning model, and the output values are the provisional identification information "XXX", "XXY", "XXZ", and the recognition coefficients are "0.58423", "0.32014", and "0.03293" respectively, the second output processing unit 232 specifies the object identification information "Product X" corresponding to the provisional identification information "XXX", the object identification information "Product Y" corresponding to the provisional identification information "XXY", and the object identification information "Product Z" corresponding to the provisional identification information "XXZ" (the correspondence between the provisional identification information and the object identification information is performed by the first processing unit 20), and sorts and outputs the provisional identification information and / or the object identification information based on the respective recognition coefficients.

[0073] The verification processing unit 233 specifies the image information stored in the image information storage unit 22 based on the provisional identification information and / or the object identification information output by the second output processing unit 232. Then, a list of the output results is displayed, and the verifier selects the object identification information corresponding to the input image information. For example, when the second output processing unit outputs the object identification information "Product X", "Product Y", "Product Z" and sorts them according to the recognition coefficients (reliability), the verification processing unit 233 extracts the image information corresponding to "Product X", "Product Y", "Product Z" from the image information storage unit 22. Then, the extracted image information and its object identification information are displayed on a predetermined computer for the verifier to use as a list in a list.

[0074] The verification processing unit 233 receives the selection of the correct output result for the input image information from the list. For example, if among "Product X", "Product Y", "Product Z", the verifier determines that "Product X" is the correct output result for the input image information, the input image information and the object identification information "Product X" are associated and stored in the learning information storage unit as a data set of learning data.

[0075] As described above, in the second processing unit 23, each piece of image information is input into the first learning model stored in the first learning model storage unit 21, and the object identification information corresponding to the output value is verified by the verifier. By accepting the input, a learning dataset (learning dataset) associating the image information with the object identification information can be stored in the learning information storage unit. Then, learning data having a plurality of learning datasets can be generated.

[0076] In addition, the learning data stored in the learning information storage unit has been verified by the verifier, and only requires inputting the image information for verification. Therefore, highly accurate learning data can be generated with a simple operation.

[0077] The learning information storage unit 24 stores learning data having a learning dataset that stores the image information and the object identification information in association with each other. Note that the image information is preferably the image information of the object.

[0078] The second model generation processing unit 25 performs machine learning by a known method using the learning data having the learning dataset stored in the learning information storage unit 24, and generates a learning model. That is, a learning model (second learning model) is generated using the image information in the learning dataset and the learning data with the label thereof being the object identification information.

[0079] The second model generation processing unit 25 stores the generated learning model (second learning model) in the second learning model storage unit 26.

[0080] The second learning model stored in the second learning model storage unit has performed machine learning using the learning data in which the image information and the object identification information are associated with each other. Therefore, when the image information is input, the corresponding object identification information will be output.

Example

[0081] Next, an example of the processing of the information processing system 1 of the present invention will be described with reference to the flowcharts of FIGS. 7 and 8. FIG. 7 is a flowchart showing a process of generating a first learning model using the first processing unit 20 and specifying object identification information corresponding to the provisional identification information. FIG. 8 is a flowchart showing a process of inputting image information into the first learning model using the second processing unit, performing verification using the output result thereof, and then generating learning data for generating a second learning model.

[0082] In the following description, a case where products displayed on a display shelf are identified from image information obtained by photographing the display shelf will be described. In this case, the object is the product displayed on the display shelf, and the object identification information is the product identification information.

[0083] First, correction processing is performed on the image information obtained by photographing the display shelf on which products are displayed in a store or the like so that the products are in a position facing directly. As the correction processing in this case, various known methods can be used. For example, trapezoidal correction processing or the like can be used. If the image information obtained by photographing the display shelf is the image information obtained by photographing from a position facing directly, the correction processing may not be performed. At this time, the image information may be the image information showing the entire display shelf, or may be the image information showing a part of the display shelf, for example, the area of one or two or more shelf levels. FIG. 9 shows an example of the image information of the shelf level area of the display shelf corrected to a position facing directly.

[0084] Then, from the corrected image information or the image information obtained by photographing the display shelf (the image information to be processed), the area of the displayed product is cut out by a known method and extracted as object image information. For example, from the corrected image information or the image information obtained by photographing the display shelf, a thin and narrow shadow generated between products is identified, a repeating pattern of the image is identified, a step at the upper edge of the package is identified, and the separation position is identified based on constraints such as the same product width, thereby identifying the area of the product. As another method, machine learning such as deep learning may be used to identify the area of the product. In this case, for a learning model in which the weighting coefficients between neurons in each layer of a neural network having a large number of intermediate layers are optimized, image information of the area to be processed, for example, the display shelf or the shelf step area, is input, and based on the output value, the area of the product may be identified. As the learning model, one in which the area of the product is given as correct data for image information of various areas to be processed, for example, the display shelf and the shelf step area, can be used.

[0085] An example of the image information obtained by extracting the object image information from the image information of the shelf step area of the display shelf in FIG. 9 is shown in FIG. 10. At this point, since only the object image information has been extracted from the image information of the photographed display shelf, the product identification information of the product, which is the object shown in the object image information, does not have to be known.

[0086] Then, the classification processing unit 200 groups similar image information together and classifies it into a plurality of groups (S100) by executing a known clustering process or the like using the object image information. FIG. 11 schematically shows this state.

[0087] The temporary identification information processing unit 201 associates temporary identification information for each group classified by the classification processing unit 200 (S110). For example, arbitrary temporary identification information such as "a123", "b456", "c789", "d012" is associated with each group. Fig. 12 schematically shows this state. As a result, for the object image information of each group, temporary identification information is associated as a label. Then, a dataset using these object image information and the labels of the temporary identification information is used as learning data.

[0088] When temporary identification information is associated for each group classified by the classification processing unit 200, the first model generation processing unit 202 performs machine learning by a known method using the learning data (S120), and generates a learning model (first learning model) (S130). The generated learning model (first learning model) is stored in the first learning model storage unit.

[0089] Next, the specimen information reception processing unit 203 receives the input of specimen information, which is image information for which the object identification information is known, and the first recognition processing unit 204 inputs the received specimen information into the learning model (first learning model) generated in S130 (S140). The first recognition processing unit 204 receives the input of the specimen information and outputs the output result of machine learning using the learning model (first learning model). As this output result, temporary identification information is output (S150). Fig. 13 schematically shows this.

[0090] The specimen information is image information for which object identification information such as product identification information is known in advance. Therefore, it is considered that the temporary identification information output by the first recognition processing unit 204 corresponds to the object identification information of the specimen information input to the learning model. Therefore, the first output processing unit 205 associates the output temporary identification information with the object identification information of the input specimen information (S160). For example, as shown in Fig. 14, the temporary identification information and the object identification information are associated and stored in a predetermined storage area.

[0091] By this association, the association between the temporary identification information and the object identification information can be established. By memorizing this association, if the temporary identification information output by the first recognition processing unit 204 is replaced with the corresponding object identification information and output, when the image information for performing the identification process is input to the learning model, the object identification information can be output as the output value of the learning model.

[0092] Next, the image information reception processing unit 230 of the second processing unit 23 receives the input of the image information to be processed (S200), and the second recognition processing unit 231 inputs the received image information to the learning model (the first learning model) stored in the first learning model storage unit 21 (S210).

[0093] The second recognition processing unit 231 outputs the output result of machine learning using the learning model (the first learning model). As this output result, one or more pieces of temporary identification information corresponding to the input image information and its recognition coefficient (reliability) are output.

[0094] The second output processing unit 232 associates the corresponding object identification information with the temporary identification information that is the output value output by the second recognition processing unit 231. Then, the second output processing unit 232 outputs the object identification information associated with the temporary identification information output by the second recognition processing unit 231 and the recognition coefficient (reliability) as the recognition result of the second recognition processing unit 231 (S220).

[0095] Also, the second output processing unit 232 sorts in descending order of the recognition coefficient (reliability) based on the recognition coefficient (reliability) (S230).

[0096] Based on the object identification information output by the second output processing unit 232, the verification processing unit 233 identifies the image information stored in the image information storage unit 22 (S240), displays a list of the output results, and executes the verification process by allowing the verifier to select the object identification information corresponding to the input image information (S250). The verification processing unit 233 causes a predetermined computer used by the verifier to display the object identification information corresponding to the output result of the learning model (first learning model) and the image information as a list of the list, and accepts the selection of the correct object identification information as the object shown in the input image information. The verification processing unit 233 may display the input image information together with the list.

[0097] Then, the selected object identification information and the input image information are associated with each other and stored in the learning information storage unit 24 as a learning data set (S260).

[0098] By performing the above-described processing on a plurality of pieces of image information in the second processing unit 23, learning data having a learning data set in which the image information and the object identification information are associated with each other can be stored in the learning information storage unit 24.

[0099] At a predetermined timing, the second model generation processing unit 25 performs machine learning by a known method using the learning data having the learning data set stored in the learning information storage unit 24 (S270), and generates a learning model (second learning model) (S280). That is, a learning model (second learning model) is generated using the image information in the learning data set and the learning data having the object identification information as its label. The learning model (second learning model) generated by the second model generation processing unit 25 is stored in the second learning model storage unit 26.

[0100] By the above-described processing, a learning model (second learning model) machine-learned using the learning data in which the image information and the object identification information are associated with each other can be generated.

Example

[0101] The classification process performed by the classification processing unit 200 may be less accurate than the classification process performed by humans. Therefore, errors may occur. Thus, a cleansing processing unit 26 may be provided that performs a cleansing process on the image information (target image information) of each group classified by the classification processing unit 200, and excludes object image information that may contain errors from the image information classified into the group, thereby reducing the object image information used as learning data for the first model generation processing unit 202. An example of the configuration of the information processing system 1 in this case is shown in FIG. 15.

[0102] That is, when the classification processing unit 200 classifies object image information for each group, index values such as feature amounts indicating the features of the image information in that group are calculated. Then, in that group, the information distance (a value calculated by a predetermined calculation formula for the deviation from the reference value such as the median or average value of the index values for each object image information to the index value of the image information) from the reference value is calculated, and object image information with an information distance deviated by a certain value or a certain ratio or more may be excluded from the group.

[0103] In addition to using the information distance as the cleansing process, the similarity of the image information may be used. In this case, the similarity between the image information in the group is quantified, and if the number of pieces of image information for which a similarity of a certain level or more is calculated is equal to or more than a predetermined value, it is left as object image information, and if it is less than the predetermined value, it is excluded from the object image information.

[0104] For example, if there are five pieces of target image information (image information 1 to image information 5) in a certain group, for these five pieces of image information, the similarity with each other image information is calculated. That is, the similarity of image information 1 with image information 2 to image information 5, image information 2 with image information 3 to image information 5, image information 3 with image information 4 to image information 5, and image information 4 with image information 5 is calculated. Thereby, the similarity between each piece of image information can be calculated.

[0105] Regarding Image Information 1, the similarity with Image Information 2 is 0.9, the similarity with Image Information 3 is 0.95, the similarity with Image Information 4 is 0.3, and the similarity with Image Information 5 is 0.8. Regarding Image Information 2, the similarity with Image Information 3 is 0.8, the similarity with Image Information 4 is 0.2, the similarity with Image Information 5 is 0.6. Regarding Image Information 3, the similarity with Image Information 4 is 0.4, the similarity with Image Information 5 is 0.9. Regarding Image Information 4, the similarity with Image Information 5 is 0.5.

[0106] Here, when the similarity of the reference image information is 0.75 and the number of image information below the reference degree is 3, for Image Information 1, the number of image information below the reference similarity is 1 for Image Information 4; for Image Information 2, the number of image information below the reference similarity is 2 for Image Information 4 and Image Information 5; for Image Information 3, the number of image information below the reference similarity is 1 for Image Information 4; for Image Information 4, the number of image information below the reference similarity is 4 for Image Information 1, Image Information 2, Image Information 3, and Image Information 5; for Image Information 5, the number of image information below the reference similarity is 2 for Image Information 2 and Image Information 4.

[0107] Therefore, the cleansing processing unit 26 excludes Image Information 4 for which the number of image information below the reference similarity is 3 or more from the group, and sets the target object image information of the group to 4 for Image Information 1, Image Information 2, Image Information 3, and Image Information 5, and causes the temporary identification information processing unit 201 to execute the processing.

[0108] By excluding the deviated image information from the grouped target object image information as in this embodiment, the grouped target object image information can be narrowed down, leading to an improvement in accuracy.

Embodiment

[0109] When extracting target object image information by cutting out the products displayed on the display shelf, generally, the same products are often adjacent to each other in one of the up, down, left, or right directions.

[0110] Therefore, after extracting the target object image information from the image information of the display shelf, the classification processing unit 200 compares the similarity of the target object image information of the adjacent products. If it meets certain conditions, for example, if the similarity is equal to or greater than a predetermined threshold value, it determines that the target objects are of the same type and may classify the target object image information of the adjacent products into the same group.

Embodiment

[0111] When the products displayed on the display shelf are used as target objects, it is preferable to generate a learning model of the products that may be displayed in advance. However, in that case, the target products may range from several hundred to several thousand, and in some cases, tens of thousands. In this case, if a learning model that can recognize a learning model in the thousands to tens of thousands is used, the sample information input by the sample information reception processing unit 203 will be in the thousands to tens of thousands, and although the workload is reduced compared to the conventional learning model, the workload is still large.

[0112] On the other hand, it is known in marketing that generally about 10% of the product types account for about 90% of the sales of that classification. Therefore, products with high sales may be used as the sample information input to the sample information reception processing unit 203. In this case, information provided by a marketing company or the like may be used.

[0113] As a result, about 90% of the products can be recognized based on the sales volume, and practically, it is acceptable as the output of the target object identification information. If a product with little sales is input into the learning model, since there is no association between the target object identification information and the provisional identification information, the first output processing unit 205 will output the provisional identification information as it is. When the first output processing unit 205 outputs the provisional identification information without the association between the provisional identification information and the target object identification information, a predetermined display such as "No corresponding information" may be performed.

[0114] Also, if it is only necessary to recognize the products of an organization such as a specific company, the image information of the products of that company may be input into the sample information reception processing unit 203 as sample information. In this way, for the products of a certain company, object identification information can be output. On the other hand, when products of a company other than a certain company are input, as in the above case, since the object identification information and the temporary identification information are not associated, the first output processing unit 205 will output the temporary identification information as it is. When the first output processing unit 205 outputs temporary identification information for which there is no association between the temporary identification information and the object identification information, a predetermined display such as "no corresponding information" may be performed.

Example

[0115] For the learning model (first learning model) generated as in Examples 1 to 4, for the purpose of improving accuracy, the learning model may be reconstructed again. In this case, new sample information may be input so that the object identification information is newly associated with the temporary identification information for which the object identification information is not associated, or alternatively, the classification process in the classification processing unit 200 may be executed using the object image information of a new object so that the new object can be recognized.

Example

[0116] As another example of Examples 1 to 5, instead of the verifier performing the verification process in the verification processing unit 233 in the second processing unit 23, it may be determined by a computer. For example, the verification processing unit 233 compares the image information specified from the image information storage unit 22 with the image information stored in another storage area, for example, the image information stored as sample information, and the second recognition processing unit 231 may determine that the object identification information of the sample information determined to have the highest similarity is the object identification information of the image information input to the first learning model. Also, instead of sample information, comparison may be made with image information on the Internet, etc.

[0117] When performing such processing, output of recognition coefficients, sorting processing, etc. may not be provided.

Example

[0118] In the above-described Examples 1 to 6, the case of generating a learning model (first learning model) for identifying products displayed on a display shelf from image information of the photographed display shelf was described, but it can also be applied to other cases. In particular, the present invention is useful for automating the identification of multiple types of objects from image information.

[0119] As an example, the object may be an animal. For example, it can be applied to a case where a plurality of types of animals such as seals and otters inhabit in large numbers in a place where people do not approach, and the types and numbers of inhabitants are efficiently grasped. In this case, the habitat is photographed from above by a drone or the like, object image information is cut out for each individual, and the object image information for each individual is subjected to image classification processing by the classification processing unit 200 and grouped. Then, the first model generation processing unit 202 performs machine learning using the learning data generated by associating the temporary identification information for each group by the temporary identification information processing unit 201, and generates a learning model (first learning model). Then, the specimen information reception processing unit 203 receives the input of specimen information by type, and the first recognition processing unit 204 inputs the received specimen information as an input value to the learning model (first learning model) to output the temporary identification information. As a result, the temporary identification information is associated with the corresponding object identification information (scientific name, name, etc. of the animal) of the input specimen information, and the first output processing unit 205 can output the object identification information.

[0120] Then, the second recognition processing unit 231 inputs the image information received by the image information reception processing unit 230 to the learning model (first learning model), and the verification processing unit 233 performs verification processing using the output result of the second output processing unit 232, whereby learning data having a learning dataset can be generated. Then, using the learning data, the second model generation processing unit 25 can generate a new learning model (second learning model).

[0121] Similarly to the above, birds or plants may be used as the object. Even in the case of birds or plants, the same processing can be performed as for the animals described above, and the processing can be executed by reading "animals" as "birds" or "plants".

Industrial Applicability

[0122] By using the information processing system 1 of the present invention, a learning model used in machine learning can be efficiently generated.

Explanation of Signs

[0123] 1: Information processing system 2: Management terminal 20: First processing unit 21: First learning model storage unit 22: Image information storage unit 23: Second processing unit 24: Learning information storage unit 25: Second model generation processing unit 26: Second learning model storage unit 70: Arithmetic unit 71: Storage device 72: Display device 73: Input device 74: Communication device 200: Classification processing unit 201: Temporary identification information processing unit 202: First model generation processing unit 203: Specimen information reception processing unit 204: First recognition processing unit 205: First output processing unit 206: Cleaning processing unit 230: Image information reception processing unit 231: Second recognition processing unit 232: Second output processing unit 233: Verification processing unit

Claims

1. An information processing system that executes processing related to a learning model used in machine learning, a first processing unit that identifies object identification information corresponding to the temporary identification information using a first learning model generated using object image information and temporary identification information associated with the object image information; a second processing unit that performs verification processing on an output result of object identification information corresponding to the image information input to the first learning model; a second model generation processing unit that executes learning processing of machine learning using the learning data verified by the second processing unit to generate a second learning model; An information processing system characterized by comprising the above.

2. The second processing unit, an image information reception processing unit that receives input of image information; a second recognition processing unit that inputs the received image information to the first learning model and outputs at least one or more pieces of temporary identification information corresponding to the image information; a second output processing unit that outputs object identification information corresponding to the output temporary identification information; a verification processing unit that extracts the image information of the output object identification information from a predetermined storage area, and performs verification processing on the object identification information of the input image information using the extracted image information and its object identification information, and uses the verified object identification information and the image information as learning data; The information processing system according to claim 1, characterized by comprising the above.

3. The second recognition processing unit, outputs temporary identification information corresponding to the image information and a recognition coefficient, The second output processing unit, outputs object identification information corresponding to the output temporary identification information, and sorts the object identification information based on the recognition coefficient. The information processing system according to claim 2, characterized by the above.

4. The verification processing unit, displays the extracted object identification information and its image information, and performs the verification processing by receiving, from a computer used by a verifier, a selection of a correct result of the image information input to the first learning model. The information processing system according to claim 2 or claim 3, characterized by the above.

5. The verification processing unit, performs the verification processing by comparing the image information corresponding to the extracted object identification information with the image information stored in a predetermined storage area. The information processing system according to claim 2 or claim 3, characterized by the above.

6. The verification processing unit, Compare the extracted object identification information with the sample information stored in a predetermined storage area, and perform a verification process of determining that the object identification information of the sample information determined to have the highest similarity is the object identification information of the image information input to the first learning model. The information processing system according to claim 5, characterized in that.

7. The first processing unit is A classification processing unit that classifies image information of a plurality of objects, A provisional identification information processing unit that associates the classified group with provisional identification information, A first model generation processing unit that executes a learning process of machine learning using learning data including the image information of the objects included in the group and the provisional identification information to generate the first learning model, A first recognition processing unit that outputs provisional identification information by inputting the received sample information into the first learning model, A first output processing unit that associates the output provisional identification information with the object identification information corresponding to the sample information, The information processing system according to claim 1, characterized in that it has.

8. The information processing system is By inputting the image information to be subjected to the identification process into the first learning model, the object identification information corresponding to the provisional identification information output by the learning model is output as an output value. The information processing system according to claim 7, characterized in that.

9. The information processing system is A cleansing processing unit that executes a cleansing process of excluding object image information that may contain errors from the image information of the objects included in the classified group, The information processing system according to claim 7, characterized in that it has.

10. The cleansing processing unit is Using the information distance from the reference value calculated using the index value of the image information of the objects included in the group, determine the image information of the objects to be excluded. The information processing system according to claim 9, characterized in that.

11. The cleansing processing unit is Calculate a reference value using the index value of the image information of the objects included in the group, Calculate the information distance using the reference value and the index value of the image information of the object, When the information distance is equal to or greater than a predetermined threshold value or there is a deviation of a predetermined ratio or more, determine the image information of the object as the image information of the object to be excluded. The information processing system according to claim 10, characterized in that.

12. The cleansing processing unit is Determining the image information of the object to be excluded by using the similarity of the image information of the objects included in the group The information processing system according to claim 9, characterized in that

13. The cleaning processing unit Calculates the similarity between the image information of the objects included in the group and the image information of other objects included in the group, When a predetermined condition using the similarity is satisfied, determines the image information of the object as the image information of the object to be excluded The information processing system according to claim 12, characterized in that

14. The object is a product displayed on a display shelf, The classification processing unit Extracts the image information of the products displayed from the image information of the display shelf as object image information, Compares the similarity of the object image information of adjacent products, When the similarity satisfies a certain condition, classifies the object image information of adjacent products into the same group The information processing system according to claim 7, characterized in that

15. The object is a product displayed on a display shelf, The first recognition processing unit Inputs the sample information of the product corresponding to the received sales amount into the learning model, and outputs the temporary identification information corresponding to the sample information received by the learning model The information processing system according to claim 7, characterized in that

16. The object is a product displayed on a display shelf, The first recognition processing unit Inputs the sample information of the product for each organization received into the learning model, and outputs the temporary identification information corresponding to the sample information received by the learning model The information processing system according to claim 7, characterized in that

17. A computer A first processing unit that specifies the object identification information corresponding to the temporary identification information by using the first learning model generated by using the object image information and the temporary identification information associated with the object image information, A second processing unit that performs a verification process on the output result of the object identification information corresponding to the image information input to the first learning model, A second model generation processing unit that executes a learning process of machine learning by using the learning data verified by the second processing unit to generate a second learning model An information processing program characterized by functioning as

Images