Information processing system

Abstract

The purpose of the present disclosure is to provide an information processing system related to the generation of a learning model (network) in machine learning.　This information processing system executes processing related to a learning model used in machine learning. The information processing system includes: a first processing unit that, by using a first learning model generated using object image information and provisional identification information associated with the object image information, specifies object identification information corresponding to the provisional identification information; a second processing unit that performs verification processing of an output result of object identification information corresponding to image information input to the first learning model; and a second model generation processing unit that executes learning processing of machine learning by using learning data verified by the second processing unit, thereby generating a second learning model.

Description

Information Processing Systems

[0001] The present disclosure relates to an information processing system for generating a learning model (network) in machine learning. CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority from patent application Ser. No. 2024-103395, filed Jun. 26, 2024, the entire contents of which are incorporated herein by reference.

[0003] In recent years, machine learning has become widely used as a computer-based image analysis method. In machine learning, a learning dataset (learning data) is created in advance, and then the computer is loaded with the data to perform a predetermined learning process.

[0004] However, preparing learning data in advance itself is a heavy workload, so systems such as those shown in Patent Documents 1 and 2 below have been disclosed.

[0005] In the disclosure of Patent Document 1, a product image is photographed, and a code attached to the product is read, whereby the product image and its label are created, thereby generating learning data.

[0006] Furthermore, in the disclosure of Patent Document 2, a classification of an object contained in image data is selected from an image database storing image data including objects that have been labeled with information relating to the classification in advance, and the object is extracted from the image data to generate a template image, thereby generating learning data that is labeled with the template image.

[0007] JP 2020-095537 A JP 2022-076296 A

[0008] In the case of the disclosure of Patent Document 1, the learning data is generated by preparing and photographing products and reading codes attached to the products with a code reader, which places a heavy workload on the worker, and the amount of work becomes enormous when there are many products to be read as learning data.

[0009] In the case of the disclosure of Patent Document 2, the processing cannot be performed unless an image database is prepared in which image data including objects labeled with information relating to their classification is stored in advance.

[0010] In retail stores and other places, recognizing products displayed on shelves and understanding their display status is extremely important for a company's marketing strategy. Therefore, there is a demand for knowing how a company's own products, or products of competitors, are displayed on shelves in retail stores.

[0011] Therefore, there are systems that take photos of retail store shelves and identify the products displayed on the shelves, and machine learning is used to identify the products. In this machine learning system, as in the above, a learning process is performed using training data, but a wide variety of products are displayed on the shelves. In this case, too, a learning model has conventionally been generated using training data in which the displayed products are associated with their identification information as labels.

[0012] However, even in this case, the burden of preparing the learning data is large, and the burden of generating the learning model is also large.

[0013] In view of the above-mentioned problems, an information processing system for efficiently generating a learning model used in machine learning has been disclosed.

[0014] An information processing system of one embodiment of the present disclosure is an information processing system that performs processing related to a learning model used in machine learning, and includes: a first processing unit that identifies object identification information corresponding to tentative identification information using a first learning model generated using object image information and tentative identification information associated with the object image information; a second processing unit that performs verification processing of the output result of the object identification information corresponding to the image information input to the first learning model; and a second model generation processing unit that performs machine learning learning processing using the learning data verified in the second processing unit to generate a second learning model.

[0015] By configuring as disclosed herein, it is possible to efficiently generate learning models used in machine learning.

[0016] In the above disclosure, the second processing unit can be configured as an information processing system having an image information reception processing unit that receives input of image information, a second recognition processing unit that inputs the received image information into the first learning model and outputs at least one piece of tentative identification information corresponding to the image information, a second output processing unit that outputs object identification information corresponding to the output tentative identification information, and a verification processing unit that extracts image information of the output object identification information from a predetermined memory area, uses the extracted image information and the object identification information to perform a verification process for the object identification information of the input image information, and uses the verified object identification information and the image information as learning data.

[0017] As disclosed herein, the output results are verified not only in the first process but also in the second process, so that a more accurate learning model can be generated efficiently.

[0018] In the above disclosure, the second recognition processing unit can be configured as an information processing system in which the second recognition processing unit outputs provisional identification information and a recognition coefficient corresponding to the image information, and the second output processing unit outputs object identification information corresponding to the output provisional identification information and sorts the object identification information based on the recognition coefficient.

[0019] Sorting the output results from the first learning model allows the verification process to be carried out quickly.

[0020] In the above disclosure, the verification processing unit can be configured as an information processing system that displays the extracted object identification information and its image information, and performs the verification processing by accepting the selection of the correct result of the image information input into the first learning model from a computer used by the verifier.

[0021] The accuracy of the verification process can be increased if a verifier checks the output results.

[0022] In the above disclosure, the verification processing unit can be configured as an information processing system that performs the verification processing by comparing image information corresponding to the extracted object identification information with image information stored in a specified memory area.

[0023] In the above disclosure, the verification processing unit can be configured as an information processing system that performs a verification process in which image information corresponding to the extracted object identification information is compared with sample information stored in a specified memory area, and the object identification information of the sample information determined to have the highest similarity is determined to be the object identification information of the image information input to the first learning model.

[0024] Since the verification process is burdensome for a verifier, it may be automated as in the present disclosure.

[0025] In the above disclosure, the first processing unit can be configured as an information processing system having a classification processing unit that classifies image information of multiple objects, a provisional identification information processing unit that associates the classified groups with provisional identification information, a first model generation processing unit that executes a machine learning learning process using learning data including image information of the objects included in the groups and the provisional identification information to generate the first learning model, a first recognition processing unit that inputs accepted sample information into the first learning model to output provisional identification information, and a first output processing unit that associates the output provisional identification information with object identification information corresponding to the sample information.

[0026] An information processing system of one embodiment of the present disclosure is an information processing system that performs processing related to a learning model used in machine learning, and includes: a classification processing unit that classifies image information of multiple objects; a tentative identification information processing unit that associates the classified groups with tentative identification information; a first model generation processing unit that performs a machine learning learning process using learning data including image information of the objects included in the groups and the tentative identification information to generate the first learning model; a first recognition processing unit that inputs accepted sample information into the first learning model, thereby outputting tentative identification information; and a first output processing unit that associates the output tentative identification information with object identification information corresponding to the sample information.

[0027] By configuring as disclosed above, it is possible to efficiently generate a learning model to be used in machine learning.

[0028] In the above disclosure, the information processing system can be configured as an information processing system that inputs image information to be identified into the learning model, and outputs object identification information corresponding to the provisional identification information output by the learning model as an output value.

[0029] By configuring as in the present disclosure, when image information of an identification target is input to a learning model, object identification information can be output.

[0030] In the above disclosure, the information processing system can be configured as an information processing system having a cleansing processing unit that performs a cleansing process to exclude object image information that may contain errors from image information of objects included in the classified group.

[0031] By performing a cleansing process as disclosed herein, it is possible to remove object image information contained in a group that may contain errors, thereby improving the accuracy of image classification.

[0032] In the above disclosure, the cleansing processing unit can be configured as an information processing system that determines image information of objects to be excluded using an information distance from a reference value calculated using index values ​​of image information of objects included in the group.

[0033] In the above disclosure, the cleansing processing unit can be configured as an information processing system that calculates a reference value using an index value of image information of the object included in the group, calculates an information distance using the reference value and the index value of the image information of the object, and if there is a deviation of the information distance that is equal to or greater than a predetermined threshold value or a predetermined ratio, determines that the image information of the object is image information of an object to be excluded.

[0034] In the above disclosure, the cleansing processing unit can be configured as an information processing system that determines image information of objects to be excluded using the similarity of image information of objects included in the group.

[0035] In the above disclosure, the cleansing processing unit can be configured as an information processing system that calculates the similarity between image information of an object included in the group and image information of other objects included in the group, and if a predetermined condition using the similarity is satisfied, determines the image information of that object as image information of an object to be excluded.

[0036] The cleansing process can be carried out as disclosed in these publications.

[0037] In the above disclosure, the objects are products displayed on a display shelf, and the classification processing unit can be configured as an information processing system that cuts out image information of the displayed products from image information of the display shelf and extracts it as object image information, compares the similarity of the object image information of adjacent products, and if the similarity satisfies certain conditions, classifies the object image information of the adjacent products into the same group.

[0038] Products of the same type are often displayed adjacent to each other on display shelves. Therefore, when objects are displayed on a display shelf, there is a possibility that the object image information of adjacent products can be classified as the same type, i.e., the same group. Therefore, it is preferable to compare the similarity of the object image information of adjacent products and, if the similarity satisfies the conditions, classify them into the same group.

[0039] In the above disclosure, the target object is a product displayed on a display shelf, and the first recognition processing unit can be configured as an information processing system that inputs sample information of products corresponding to the sales amount received as input into the learning model and outputs provisional identification information corresponding to the sample information received by the learning model.

[0040] In the case of a learning model that identifies products displayed on a shelf, the number of products can be in the thousands or even tens of thousands. In such cases, inputting sample information for all products is a heavy burden.

[0041] Generally, a small number of best-selling products account for the majority of sales. And it is highly likely that best-selling products are displayed on display shelves. Therefore, if best-selling products that are likely to be identified are input as sample information, even if it is not possible to identify all products displayed on display shelves, it will be possible to recognize many products based on sales figures, and a system that is sufficient for practical purposes can be constructed.

[0042] In the above disclosure, the target object is a product displayed on a display shelf, and the first recognition processing unit can be configured as an information processing system that inputs sample information of the product for each organization that receives input into the learning model and outputs provisional identification information corresponding to the sample information received by the learning model.

[0043] When the information processing system of the present disclosure is used for marketing, a specific organization, such as a specific company, may want to understand the display status of its own or its competitors' products. Therefore, by inputting sample product information for each organization, as in the present invention, even if it is not possible to identify all products displayed on the shelves, the organization can obtain important marketing information, and a system that is sufficient for practical purposes can be constructed.

[0044] In the above disclosure, the information processing system can be configured as an information processing system having a second processing unit that performs verification processing on the output result of object identification information corresponding to the image information input to the first learning model.

[0045] In the above disclosure, the information processing system can be configured as an information processing system having a second model generation processing unit that uses the learning data verified in the second processing unit to perform a machine learning learning process to generate a second learning model.

[0046] The above-disclosed aspect can be realized by loading and executing the program of the present disclosure into a computer. That is, the information processing program causes a computer to function as a first processing unit that identifies object identification information corresponding to tentative identification information using a first learning model generated using object image information and tentative identification information associated with the object image information, a second processing unit that performs a verification process on an output result of the object identification information corresponding to image information input to the first learning model, and a second model generation processing unit that performs a machine learning learning process using the learning data verified by the second processing unit to generate a second learning model.

[0047] The above-disclosed aspect can be realized by loading and executing the program of the present disclosure into a computer. That is, the information processing program causes a computer to function as a classification processing unit that classifies image information of a plurality of objects, a temporary identification information processing unit that associates the classified groups with temporary identification information, a first model generation processing unit that executes machine learning learning processing to generate the first learning model using learning data including image information of the objects included in the groups and the temporary identification information, a first recognition processing unit that inputs accepted sample information into the first learning model to output temporary identification information, and a first output processing unit that associates the output temporary identification information with object identification information corresponding to the sample information.

[0048] By using the information processing system of the present disclosure, it is possible to efficiently generate learning models used in machine learning.

[0049] 1 is a diagram illustrating an example of a conceptual diagram of processing of an information processing system of an embodiment of the present disclosure. FIG. 1 is a diagram illustrating an example of a conceptual diagram of processing of an information processing system of an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a conceptual diagram of processing of an information processing system of an embodiment of the present disclosure. FIG. 3 is a block diagram schematically illustrating an example of a configuration of an information processing system of an embodiment of the present disclosure. FIG. 4 is a block diagram schematically illustrating an example of a hardware configuration of a computer used in the information processing system of an embodiment of the present disclosure. FIG. 5 is a flowchart illustrating an example of a generation process of a first learning model in the information processing system of an embodiment of the present disclosure. FIG. 6 is a flowchart illustrating an example of a generation process of a second learning model in the information processing system of an embodiment of the present disclosure. FIG. 7 is a diagram illustrating an example of image information of an image of a display shelf. FIG. 8 is a diagram illustrating an example of a state in which object image information is extracted from image information of an image of a display shelf. FIG. 9 is a diagram illustrating an example of a state in which object image information is classified in a classification processing unit. FIG. 10 is a diagram illustrating an example of a state in which temporary identification information is associated in a temporary identification information processing unit. FIG. 11 is a diagram illustrating an example of a state in which sample information is input to a learning model and temporary identification information is output as an output value. FIG. 12 is a diagram illustrating an example of a state in which temporary identification information and object identification information are linked. FIG. 13 is a diagram schematically illustrating an example of a configuration of an information processing system in Example 2.

[0050] 1 to 4 show exemplary conceptual diagrams of processing in an information processing system 1 according to an embodiment of the present disclosure. Also, FIG. 5 shows an exemplary block diagram of processing functions of the information processing system 1 according to an embodiment of the present disclosure. The information processing system 1 is a system that generates a model that recognizes objects, such as products displayed on a display shelf, from image information using a machine learning model. While the object will be described using products displayed on a display shelf, the present disclosure is not limited to this.

[0051] The management terminal 2 in the information processing system 1 is realized using a computer. An example of the hardware configuration of a computer is shown in Figure 6. The computer has a CPU and the like that executes the arithmetic processing of a program, a storage device 71 such as RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or a mouse that can input information, and a communication device 74 that sends and receives the processing results of the arithmetic device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0052] If the computer is equipped with a touch panel display, the display device 72 may be integrally configured with the input device 73. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited to these.

[0053] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

[0054] The functions of the means in the present disclosure are only logically distinct, and may be physically or practically in the same area. The order of the processes in the means in the present disclosure may be changed as appropriate. In addition, some of the processes may be omitted.

[0055] The management terminal 2 has a first processing unit 20, a first learning model memory unit 21, an image information memory unit 22, a second processing unit 23, a learning information memory unit 24, a second model generation processing unit 25, and a second learning model memory unit 26.

[0056] The first processing unit 20 generates a learning model (first learning model) using image information (object image information) of an object captured in, for example, image information of a display shelf, and tentative identification information (described later) associated with the object image information, and identifies object identification information corresponding to the tentative identification information. The first processing unit 20 has a classification processing unit 200, a tentative identification information processing unit 201, a first model generation processing unit 202, a specimen information reception processing unit 203, a first recognition processing unit 204, and a first output processing unit 205.

[0057] The classification processing unit 200 classifies the object image information by clustering processing or the like from image information of objects (object image information) captured in, for example, image information of a display shelf. When the object is a commodity displayed on a display shelf, the object image information may be image information of the commodity, such as image information obtained by cutting out an area including the displayed commodity, which is the object, from image information of the display shelf. The cutout may be rectangular, or may be image information cut out in any shape corresponding to the outer shape of the commodity.

[0058] Identification information (object identification information) of an object shown in the object image information, for example, if the object is a product, the product name, JAN code, or other code may not be known.

[0059] The classification processing unit 200 performs clustering processing on a plurality of pieces of object image information using a known method, and classifies the image information into a plurality of groups.

[0060] Although the classification processing unit 200 has been described as classifying the object image information using clustering processing, the object image information may be classified using a method other than clustering processing.

[0061] The temporary identification information processing unit 201 associates temporary identification information (temporary identification information) with each group classified by the classification processing unit 200. The temporary identification information may be automatically generated, or input temporary identification information may be associated by a predetermined operation. The temporary identification information associated with the object image information classified into each group is then associated as a label to form a data set, which is used as learning data.

[0062] The first model generation processing unit 202 performs machine learning using the learning data by a known method to generate a learning model. That is, since temporary identification information is associated with each classified group of object image information, a learning model (first learning model) is generated using learning data that uses a data set in which the object image information and its label are used as temporary identification information.

[0063] The first model generation processing unit 202 stores the generated learning model (first learning model) in the first learning model storage unit 21.

[0064] The specimen information reception processing unit 203 receives input of image information to be input to the first model generation processing unit 202. The image information received here is object image information for which object identification information for identifying the object is known. For example, it is image information of a product that may be displayed on a display shelf, and is image information for which product identification information (object identification information) such as the product name, JAN code, or other code corresponding to the product is known. The image information input here will be referred to as specimen information.

[0065] The first recognition processing unit 204 inputs the image information (sample information) received by the sample information receiving processing unit 203 into the first learning model stored in the first learning model storage unit 21, and outputs provisional identification information. In this case, when sample information is input into a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, provisional identification information is output as an output value. When multiple output values ​​are output together with their recognition coefficients (reliability), the output value with the highest recognition coefficient (reliability) is adopted.

[0066] The first output processing unit 205 links the tentative identification information, which is the output value output by the first recognition processing unit 204, with the object identification information of the sample information used for its input. Then, the first output processing unit 205 outputs the object identification information linked to the tentative identification information output by the first recognition processing unit 204 as the recognition result of the first recognition processing unit 204.

[0067] In addition, instead of outputting object identification information linked to the provisional identification information output by the first recognition processing unit 204, the first output processing unit 205 may replace the object identification information with the provisional identification information associated with the group output by the classification processing unit 200, and cause the first model generation processing unit 202 to perform the machine learning learning process again.

[0068] Furthermore, if the recognition coefficient (reliability) of the tentative identification information output by the first recognition processing unit 204 is equal to or less than a predetermined threshold, the processing of the first output processing unit 205 does not need to be executed.

[0069] By performing the above-described processing in the first processing unit 20, a learning model (first learning model) to be used in machine learning can be efficiently generated.

[0070] The image information storage unit 22 stores image information and corresponding object identification information. For example, it stores object image information that appears in image information of a display shelf, and corresponding object identification information. Note that the image information storage unit 22 may store provisional identification information in addition to the object identification information. The image information storage unit 22 may also store specimen information received by the specimen information reception processing unit 203.

[0071] The second processing unit 23 inputs image information into the first learning model stored in the first learning model storage unit 21, verifies the output value for the input, and uses the input as learning data for generating a second learning model in the second model generation processing unit 25 (described later). The second processing unit 23 has an image information reception processing unit 230, a second recognition processing unit 231, a second output processing unit 232, and a verification processing unit 233.

[0072] The image information reception processing unit 230 receives input of image information to be input to the first learning model stored in the first learning model storage unit 21. The image information received as input here may be object image information that was not used in the classification process by the classification processing unit 200, or other image information. For example, it may be object image information that appears in image information of a display shelf. The image information stored in the image information storage unit 22 may be image information used in the processing of the second processing unit 23.

[0073] The second recognition processing unit 231 inputs the image information received by the image information receiving processing unit 230 to the first learning model stored in the first learning model storage unit 21. The first learning model outputs at least one piece of provisional identification information and its recognition coefficient (reliability) as output values ​​for the input image information. In this case, when image information is input to a learning model (first learning model) in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, provisional identification information and its recognition coefficient (reliability) are output as output values.

[0074] The second output processing unit 232 links the corresponding object identification information to the tentative identification information that is the output value output by the second recognition processing unit 231. Then, the second output processing unit 232 outputs the object identification information linked to the tentative identification information output by the second recognition processing unit 231 and a recognition coefficient (reliability) as the recognition result of the second recognition processing unit 231.

[0075] Furthermore, if the recognition coefficient (reliability) of the tentative identification information output by the second recognition processing unit 231 is equal to or less than a predetermined threshold, it is not necessary to execute the processing of the second output processing unit 232. Furthermore, the second recognition processing unit 231 or the second output processing unit 232 only needs to output tentative identification information or object identification information with a recognition coefficient (reliability) equal to or greater than a predetermined threshold.

[0076] Furthermore, the second output processing unit 232 sorts and outputs the results in descending order of the recognition coefficient (reliability) based on the recognition coefficient (reliability).

[0077] For example, when the second recognition processing unit 231 inputs the image information received by the image information reception processing unit 230 into the first learning model, and the output values ​​are provisional identification information "XXX," "XXY," and "XXZ," with recognition coefficients of "0.58423," "0.32014," and "0.03293," respectively, the second output processing unit 232 identifies the object identification information "product X" corresponding to the provisional identification information "XXX," the object identification information "product Y" corresponding to the provisional identification information "XXY," and the object identification information "product Z" corresponding to the provisional identification information "XXZ" (the correspondence between the provisional identification information and the object identification information is performed by the first processing unit 20), and sorts and outputs the provisional identification information and / or the object identification information based on their respective recognition coefficients.

[0078] The verification processing unit 233 identifies image information stored in the image information storage unit 22 based on the tentative identification information and / or object identification information output by the second output processing unit 232. Then, a list of the output results is displayed, and the verifier selects the object identification information corresponding to the input image information. For example, if the second output processing unit outputs the object identification information "Product X," "Product Y," and "Product Z" and sorts them by recognition coefficient (reliability), the verification processing unit 233 extracts image information corresponding to "Product X," "Product Y," and "Product Z" from the image information storage unit 22. Then, the extracted image information and its object identification information are displayed as a list on a specific computer used by the verifier.

[0079] The verification processing unit 233 accepts the selection of the correct output result for the input image information from the list. For example, if the verifier determines that "Product X" is the correct output result for the input image information out of "Product X," "Product Y," and "Product Z," the input image information and the object identification information "Product X" are associated with each other and stored in the learning information storage unit as a data set of learning data.

[0080] As described above, the second processing unit 23 inputs each image information into the first learning model stored in the first learning model storage unit 21, has a verifier verify the object identification information corresponding to the output value, and accepts the input, thereby storing a learning data set (learning data set) in which the image information and the object identification information are associated with each other in the learning information storage unit. Thus, learning data having a plurality of learning data sets can be generated.

[0081] Furthermore, the learning data stored in the learning information storage unit has been verified by a verifier, and since it is only necessary to input image information and verify it, highly accurate learning data can be generated with simple operations.

[0082] The learning information storage unit 24 stores learning data having a learning data set in which image information and object identification information are stored in association with each other. The image information is preferably image information of an object.

[0083] The second model generation processing unit 25 generates a learning model by performing machine learning using a known method, using learning data having a learning dataset stored in the learning information storage unit 24. That is, a learning model (second learning model) is generated using image information in the learning dataset and learning data whose labels are used as object identification information.

[0084] The second model generation processing unit 25 stores the generated learning model (second learning model) in the second learning model storage unit 26.

[0085] The second learning model stored in the second learning model memory unit is machine-learned using learning data in which image information and object identification information are associated, so that when image information is input, the corresponding object identification information is output.

[0086] Next, an example of processing by the information processing system 1 according to one embodiment of the present disclosure will be described with reference to the flowcharts of Fig. 7 and Fig. 8. Fig. 7 is a flowchart illustrating processing for generating a first learning model using the first processing unit 20 and identifying object identification information corresponding to tentative identification information. Fig. 8 is a flowchart illustrating processing for inputting image information into the first learning model, verifying the output result, and then generating learning data for generating a second learning model using the second processing unit.

[0087] In the following description, a case will be described in which products displayed on a display shelf are identified from image information obtained by photographing the display shelf. In this case, the target objects are the products displayed on the display shelf, and the target identification information is the product identification information.

[0088] First, image information captured of a display shelf displaying products in a store or the like is corrected so that the products are positioned in a frontal view. Various known methods can be used for the correction, such as trapezoidal correction. If the image information captured of the display shelf is captured from a frontal view, no correction is required. In this case, the image information may be of the entire display shelf, or may be of a portion of the display shelf, such as one or more shelf levels. Figure 9 shows an example of image information of the shelf level area of ​​a display shelf corrected to a frontal view.

[0089] Then, from the corrected image information or image information of a display shelf (image information to be processed), the displayed product area is extracted using a known method and is extracted as object image information. For example, the product area can be identified by identifying thin, narrow shadows between products, identifying repetitive patterns in the image, identifying steps on the top edge of the packaging, or identifying separator positions based on constraints such as the product width being the same, from the corrected image information or image information of a display shelf. Alternatively, the product area can be identified using machine learning such as deep learning. In this case, image information of the target area, such as a display shelf or shelf area, can be input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the product area can be identified based on the output value. Furthermore, the learning model can be one in which the product area is provided as correct answer data in image information of various target areas, such as a display shelf or shelf area.

[0090] Fig. 10 shows an example of image information obtained by extracting object image information from image information of the shelf area of ​​the display shelf in Fig. 9. At this point, the object image information has simply been extracted from image information showing the display shelf, so the product identification information of the products that are the objects shown in the object image information does not need to be known.

[0091] The classification processing unit 200 then performs a known clustering process or the like using the object image information to group similar image information together and classify the information into a plurality of groups (S100). This state is shown schematically in FIG. 11.

[0092] The temporary identification information processing unit 201 associates temporary identification information with each group classified by the classification processing unit 200 (S110). For example, arbitrary temporary identification information such as "a123," "b456," "c789," and "d012" is associated with each group. This state is schematically shown in FIG. 12. In this way, the temporary identification information is associated as a label with the object image information of each group. A data set using this object image information and the labels of the temporary identification information is then used as learning data.

[0093] After the classification processing unit 200 associates provisional identification information with each group classified, the first model generation processing unit 202 performs machine learning using a known method as learning data (S120) to generate a learning model (first learning model) (S130). The generated learning model (first learning model) is stored in the first learning model storage unit.

[0094] Next, the specimen information reception processing unit 203 receives input of specimen information, which is image information for which object identification information is known, and the first recognition processing unit 204 inputs the received input of specimen information into the learning model (first learning model) generated in S130 (S140). The first recognition processing unit 204 receives input of specimen information and outputs an output result based on machine learning using the learning model (first learning model). This output result includes provisional identification information (S150). This is schematically shown in FIG. 13.

[0095] The specimen information is image information for which object identification information, such as product identification information, is known in advance. Therefore, the tentative identification information output by the first recognition processing unit 204 is considered to correspond to the object identification information of the specimen information input to the learning model. Therefore, the first output processing unit 205 associates the output tentative identification information with the object identification information of the input specimen information (S160). For example, as shown in FIG. 14 , the tentative identification information and the object identification information are associated and stored in a predetermined storage area.

[0096] This linking allows the provisional identification information to be associated with the object identification information. By storing this association, the provisional identification information output by the first recognition processing unit 204 can be replaced with the corresponding object identification information and output. When image information to be subjected to identification processing is input into the learning model, the object identification information can be output as the output value of the learning model.

[0097] Next, the image information reception processing unit 230 of the second processing unit 23 receives input of the image information to be processed (S200), and the second recognition processing unit 231 inputs the received input image information into a learning model (first learning model) stored in the first learning model memory unit 21 (S210).

[0098] The second recognition processing unit 231 outputs the output result of machine learning using the learning model (first learning model), which includes one or more pieces of provisional identification information corresponding to the input image information and their recognition coefficients (reliability).

[0099] The second output processing unit 232 links the corresponding object identification information to the tentative identification information, which is the output value output by the second recognition processing unit 231. Then, the second output processing unit 232 outputs the object identification information linked to the tentative identification information output by the second recognition processing unit 231 and a recognition coefficient (reliability) as the recognition result of the second recognition processing unit 231 (S220).

[0100] Furthermore, the second output processing unit 232 sorts the recognition coefficients (reliabilities) in descending order based on the recognition coefficients (reliabilities) (S230).

[0101] The verification processing unit 233 identifies the image information stored in the image information storage unit 22 based on the object identification information output by the second output processing unit 232 (S240), displays a list of the output results, and executes the verification process by having the verifier select the object identification information corresponding to the input image information (S250). The verification processing unit 233 displays a list of the object identification information corresponding to the output result of the learning model (first learning model) and its image information on a specific computer used by the verifier, and accepts the selection of the object identification information that is correct as the object appearing in the input image information. The verification processing unit 233 may display the input image information together with the list.

[0102] The selected object identification information is then associated with the input image information and stored as a learning data set in the learning information storage unit 24 (S260).

[0103] By performing the above processing on multiple image information in the second processing unit 23, learning data having a learning data set that corresponds image information to object identification information can be stored in the learning information storage unit 24.

[0104] At a predetermined timing, the second model generation processing unit 25 performs machine learning using a known method using the learning data having the learning dataset stored in the learning information storage unit 24 (S270) to generate a learning model (second learning model) (S280). That is, the learning model (second learning model) is generated using the image information in the learning dataset and the learning data whose labels are used as object identification information. The learning model (second learning model) generated by the second model generation processing unit 25 is stored in the second learning model storage unit 26.

[0105] By the above-described processing, a learning model (second learning model) can be generated by machine learning using learning data in which image information and object identification information are associated with each other.

[0106] The classification process performed by the classification processing unit 200 may be less accurate than classification processes performed by humans. As a result, errors may be introduced. Therefore, the information processing system 1 may be provided with a cleansing processing unit 26 that performs a cleansing process on the image information (target image information) of each group classified by the classification processing unit 200, removes object image information that may contain errors from the image information classified into the groups, and reduces the amount of object image information to be used as learning data for the first model generation processing unit 202. An example of the configuration of the information processing system 1 in this case is shown in FIG. 15 .

[0107] That is, when the classification processing unit 200 classifies the object image information into groups, it calculates index values ​​such as feature amounts that indicate the characteristics of the image information in that group. Then, it calculates the information distance from a reference value such as the median or average value of the index values ​​for each object image information in that group (a value calculated using a predetermined formula as the deviation from the reference value to the index value for each image information), and object image information whose information distance deviates by a certain value or a certain ratio or more may be excluded from the group.

[0108] In addition to using information distance for the cleansing process, similarity of image information may also be used. In this case, the similarity between image information in a group is quantified, and if the number of images with a calculated similarity above a certain level is equal to or greater than a predetermined value, the image information is retained as object image information, and if the number is less than the predetermined value, the image information is excluded from the object image information.

[0109] For example, if a group contains five pieces of target image information (image information 1 to image information 5), the similarity between each of the five pieces of image information and the other pieces of image information is calculated. That is, the similarity between image information 1 and image information 2 to image information 5 is calculated, the similarity between image information 2 and image information 5 is calculated, the similarity between image information 3 and image information 5 is calculated, and the similarity between image information 4 and image information 5 is calculated. In this way, the similarity between each piece of image information can be calculated.

[0110] Assume that for image information 1, the similarity with image information 2 is 0.9, the similarity with image information 3 is 0.95, the similarity with image information 4 is 0.3, and the similarity with image information 5 is 0.8; for image information 2, the similarity with image information 3 is 0.8, the similarity with image information 4 is 0.2, and the similarity with image information 5 is 0.6; for image information 3, the similarity with image information 4 is 0.4 and the similarity with image information 5 is 0.9; and for image information 4, the similarity with image information 5 is 0.5.

[0111] Here, assuming that the similarity of the reference image information is 0.75 and the number of pieces of image information below the reference level is three, then for image information 1, only one piece of image information 4 falls below the reference similarity level; for image information 2, two pieces of image information 4 and image information 5 fall below the reference similarity level; for image information 3, only one piece of image information 4 falls below the reference similarity level; for image information 4, four pieces of image information 1, image information 2, image information 3, and image information 5 fall below the reference similarity level; and for image information 5, two pieces of image information 2 and image information 4 fall below the reference similarity level.

[0112] Therefore, the cleansing processing unit 26 excludes image information 4, which has three or more pieces of image information below the standard similarity, from the group, and executes processing by the provisional identification information processing unit 201 with four pieces of target image information for the group: image information 1, image information 2, image information 3, and image information 5.

[0113] By excluding the deviated image information from the grouped object image information as in this embodiment, it is possible to narrow down the grouped object image information, which leads to an improvement in accuracy.

[0114] When extracting object image information by cutting out products displayed on a display shelf, it is common for the same products to be placed adjacent to each other in either the top, bottom, left, or right direction.

[0115] Therefore, after extracting object image information from the image information of the display shelf, the classification processing unit 200 compares the similarity of the object image information of adjacent products, and if it satisfies certain conditions, for example, if the similarity is equal to or greater than a predetermined threshold, it may determine that the objects are of the same type and classify the object image information of the adjacent products into the same group.

[0116] When targeting products displayed on shelves, it is preferable to generate learning models of products that may be displayed in advance. However, in this case, the target products may number from hundreds to thousands, or even tens of thousands. In this case, if a learning model capable of recognizing thousands to tens of thousands of products is used, the sample information input by the sample information reception processing unit 203 will be in the thousands to tens of thousands. Although the workload is reduced compared to conventional learning models, the workload is still significant.

[0117] On the other hand, it is known in marketing that generally, about 10% of product types tend to account for about 90% of the sales of their classification. Therefore, products with high sales may be used as the specimen information to be input to the specimen information reception processing unit 203. In this case, information provided by a marketing company or the like may be used.

[0118] This makes it possible to recognize about 90% of products based on sales, which is sufficient for practical purposes as output of object identification information. Note that when a product with low sales is input to the learning model, the object identification information is not linked to the provisional identification information, and the first output processing unit 205 will output the provisional identification information as is. When the first output processing unit 205 outputs provisional identification information that does not correspond to the provisional identification information, it may display a predetermined message such as "no correspondence information."

[0119] Furthermore, if it is sufficient to recognize only products of a specific company or other organization, image information of the company's products may be input as sample information to the sample information reception processing unit 203. This allows object identification information to be output for the company's products. On the other hand, when a product of a company other than the specific company is input, as in the above case, the object identification information and the provisional identification information are not linked, so the first output processing unit 205 will output the provisional identification information as is. When the first output processing unit 205 outputs provisional identification information that does not have a correspondence between the provisional identification information and the object identification information, a predetermined display such as "no correspondence information" may be displayed.

[0120] For the purpose of improving accuracy, the learning model (first learning model) generated as in Examples 1 to 4 may be reconstructed again. In this case, new sample information may be input, and new object identification information may be linked to provisional identification information to which no object identification information has been linked. Alternatively, the classification processing unit 200 may perform classification processing using object image information of a new object so that the new object can be recognized.

[0121] As another example of Examples 1 to 5, the verification process in the verification processing unit 233 of the second processing unit 23 may be performed by a computer rather than by a verifier. For example, the verification processing unit 233 may compare image information identified from the image information storage unit 22 with image information stored in another storage area, such as image information stored as sample information, and determine the object identification information of the sample information determined to have the highest similarity as the object identification information of the image information input by the second recognition processing unit 231 to the first learning model. Furthermore, the comparison may be made with image information on the Internet instead of sample information.

[0122] When performing such processing, it is not necessary to provide output of recognition coefficients, sorting processing, etc.

[0123] In the above-described first to sixth embodiments, a learning model (first learning model) is generated for identifying products displayed on a display shelf from image information of the display shelf. However, the present disclosure can also be applied to other cases. In particular, the present disclosure is useful for automating the identification of various types of objects from image information.

[0124] As an example, the target may be an animal. For example, this method can be applied to cases where multiple types of animals, such as sea lions and fur seals, live in large numbers in places where people cannot approach, and the types and populations of these animals need to be efficiently identified. In this case, the habitat is photographed from the air using a drone or other device, and object image information for each individual is extracted. The object image information for each individual is then subjected to image classification processing by the classification processing unit 200 and grouped. Then, using the learning data generated by the provisional identification information processing unit 201, which associates provisional identification information with each group, the first model generation processing unit 202 performs machine learning to generate a learning model (first learning model). The specimen information reception processing unit 203 then accepts input of specimen information by type, and the first recognition processing unit 204 inputs the accepted specimen information as input values ​​into the learning model (first learning model), thereby outputting provisional identification information. This links the provisional identification information with the object identification information (such as the animal's scientific name or appellation) corresponding to the input specimen information, enabling the first output processing unit 205 to output the object identification information.

[0125] The second recognition processing unit 231 inputs the image information received by the image information reception processing unit 230 into a learning model (first learning model), and the verification processing unit 233 performs verification processing using the output result of the second output processing unit 232, thereby generating learning data having a learning data set.Then, the second model generation processing unit 25 can generate a new learning model (second learning model) using the learning data.

[0126] As in the above, the target may be a bird, plant, fish, insect, microorganism, virus, cell, etc. Even in the case of birds, plants, fish, insects, microorganism, virus, cell, etc., processing can be performed in the same way as for the above-mentioned animals, and processing can be performed by replacing "animal" with "bird," "plant," "fish," "insect," "microorganism," "virus," "cell," etc.

[0127] By using the information processing system 1 of the present disclosure, it is possible to efficiently generate a learning model used in machine learning.

[0128] 1: Information processing system 2: Management terminal 20: First processing unit 21: First learning model storage unit 22: Image information storage unit 23: Second processing unit 24: Learning information storage unit 25: Second model generation processing unit 26: Second learning model storage unit 70: Arithmetic unit 71: Storage unit 72: Display device 73: Input device 74: Communication device 200: Classification processing unit 201: Tentative identification information processing unit 202: First model generation processing unit 203: Specimen information reception processing unit 204: First recognition processing unit 205: First output processing unit 206: Cleansing processing unit 230: Image information reception processing unit 231: Second recognition processing unit 232: Second output processing unit 233: Verification processing unit

Claims

1. An information processing system that executes processing related to a learning model used in machine learning, comprising: a first processing unit that identifies object identification information corresponding to tentative identification information using a first learning model generated using object image information and tentative identification information associated with the object image information; a second processing unit that performs verification processing of the output result of object identification information corresponding to image information input to the first learning model; and a second model generation processing unit that executes machine learning learning processing using learning data verified by the second processing unit to generate a second learning model.

2. The information processing system of claim 1, wherein the second processing unit comprises: an image information reception processing unit that receives input of image information; a second recognition processing unit that inputs the received input image information into the first learning model and outputs at least one piece of tentative identification information corresponding to the image information; a second output processing unit that outputs object identification information corresponding to the output tentative identification information; and a verification processing unit that extracts image information of the output object identification information from a specified memory area, uses the extracted image information and the object identification information to perform verification processing of the object identification information of the input image information, and uses the verified object identification information and the image information as learning data.

3. The information processing system of claim 2, wherein the second recognition processing unit outputs provisional identification information and a recognition coefficient corresponding to the image information, and the second output processing unit outputs object identification information corresponding to the output provisional identification information and sorts the object identification information based on the recognition coefficient.

4. An information processing system as described in claim 2 or claim 3, wherein the verification processing unit performs the verification process by displaying the extracted object identification information and its image information, and accepting the selection of the correct result of the image information input into the first learning model from a computer used by the verifier.

5. An information processing system as described in claim 2 or claim 3, wherein the verification processing unit performs the verification processing by comparing image information corresponding to the extracted object identification information with image information stored in a specified memory area.

6. The information processing system of claim 5, wherein the verification processing unit performs a verification process in which it compares image information corresponding to the extracted object identification information with sample information stored in a specified memory area, and determines that the object identification information of the sample information determined to have the highest similarity is the object identification information of the image information input to the first learning model.

7. The information processing system of claim 1, wherein the first processing unit comprises: a classification processing unit that classifies image information of multiple objects; a provisional identification information processing unit that associates the classified groups with provisional identification information; a first model generation processing unit that executes a machine learning learning process using learning data including image information of objects included in the groups and the provisional identification information to generate the first learning model; a first recognition processing unit that inputs accepted sample information into the first learning model, thereby outputting provisional identification information; and a first output processing unit that associates the output provisional identification information with object identification information corresponding to the sample information.

8. An information processing system that executes processing related to a learning model used in machine learning, comprising: a classification processing unit that classifies image information of multiple objects; a tentative identification information processing unit that associates the classified groups with tentative identification information; a first model generation processing unit that executes machine learning learning processing using learning data including image information of objects included in the groups and the tentative identification information to generate the first learning model; a first recognition processing unit that inputs accepted sample information into the first learning model, thereby outputting tentative identification information; and a first output processing unit that associates the output tentative identification information with object identification information corresponding to the sample information.

9. The information processing system according to claim 7 or claim 8, wherein the image information to be subjected to the identification process is input to the first learning model, and object identification information corresponding to the provisional identification information output by the learning model is output as an output value.

10. The information processing system according to claim 7 or claim 8, further comprising: a cleansing processing unit that performs a cleansing process to remove object image information that may contain errors from the image information of objects included in the classified groups.

11. The information processing system according to claim 10, wherein the cleansing processing unit determines the image information of the object to be excluded using an information distance from a reference value calculated using an index value of the image information of the object included in the group.

12. The information processing system according to claim 11, wherein the cleansing processing unit calculates a reference value using an index value of the image information of the object included in the group, calculates an information distance using the reference value and the index value of the image information of the object, and if there is a deviation of the information distance by more than a predetermined threshold or a predetermined ratio, determines the image information of the object as image information of an object to be excluded.

13. The information processing system according to claim 10, wherein the cleansing processing unit determines image information of objects to be excluded using a similarity of image information of objects included in the group.

14. The information processing system of claim 13, wherein the cleansing processing unit calculates the similarity between image information of an object included in the group and image information of other objects included in the group, and if a predetermined condition using the similarity is satisfied, determines the image information of that object as image information of an object to be excluded.

15. An information processing system as described in claim 7 or claim 8, wherein the object is a product displayed on a display shelf, and the classification processing unit cuts out image information of the displayed product from image information of the display shelf and extracts it as object image information, compares the similarity of the object image information of adjacent products, and if the similarity satisfies certain conditions, classifies the object image information of the adjacent products into the same group.

16. An information processing system as described in claim 7 or claim 8, wherein the target object is a product displayed on a display shelf, and the first recognition processing unit inputs sample information of products corresponding to the sales amount received as input into the learning model, and outputs provisional identification information corresponding to the sample information received by the learning model.

17. An information processing system as described in claim 7 or claim 8, wherein the target object is a product displayed on a display shelf, and the first recognition processing unit inputs sample information of the product for each organizational entity that receives input into the learning model, and outputs provisional identification information corresponding to the sample information received by the learning model.

18. The information processing system according to claim 8, further comprising: a second processing unit that performs verification processing of the output result of object identification information corresponding to the image information input to the first learning model.

19. The information processing system according to claim 18, further comprising: a second model generation processing unit that uses the learning data verified by the second processing unit to perform a machine learning learning process to generate a second learning model.

20. An information processing program that causes a computer to function as: a first processing unit that identifies object identification information corresponding to provisional identification information using a first learning model generated using object image information and provisional identification information associated with the object image information; a second processing unit that performs a verification process on the output result of object identification information corresponding to image information input to the first learning model; and a second model generation processing unit that performs a machine learning learning process using learning data verified by the second processing unit to generate a second learning model.

21. An information processing program that causes a computer to function as: a classification processing unit that classifies image information of multiple objects; a provisional identification information processing unit that associates the classified groups with provisional identification information; a first model generation processing unit that executes a machine learning learning process using learning data including image information of objects included in the groups and the provisional identification information to generate the first learning model; a first recognition processing unit that inputs accepted sample information into the first learning model, thereby outputting provisional identification information; and a first output processing unit that associates the output provisional identification information with object identification information corresponding to the sample information.

Images