Information processing system

Abstract

An object of the present invention is to provide an information processing system related to generation of a learning model (network) in machine learning.SOLUTION: An information processing system that executes processing related to a learning model used in machine learning, the information processing system comprising: a first processing section that specifies object identification information corresponding to pseudonymous identification information using a first learning model generated using object image information and pseudonymous identification information associated with the object image information; a second processing section that performs verification processing of an output result of object identification information corresponding to image information input to the first learning model; and a second model generation processing section that executes learning processing of machine learning using learning data verified by the second processing section to generate a second learning model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system for generating a learning model (network) in machine learning. [Background technology]

[0002] In recent years, machine learning has become widely used as a computer-based image analysis process. In machine learning, a learning dataset (learning data) is created in advance, and then the computer is loaded with the data to perform a predetermined learning process.

[0003] However, preparing learning data in advance itself is a heavy workload. To address this issue, systems such as those shown in Patent Documents 1 and 2 below have been disclosed.

[0004] In the invention of Patent Document 1, product images are photographed and codes attached to the products are read, and product images and labels are then created to generate learning data.

[0005] Furthermore, in the invention of Patent Document 2, a classification of an object contained in image data is selected from an image database storing image data including objects that have been labeled with information relating to the classification in advance, and the object is extracted from the image data to generate a template image, thereby generating training data labeled with the template image. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2020-095537 [Patent Document 2] Japanese Patent Publication No. 2022-076296 Summary of the Invention [Problem to be solved by the invention]

[0007] In the case of the invention of Patent Document 1, products are prepared, photographed, and the codes attached to the products are read with a code reader to generate learning data, which places a heavy workload on the worker. If there are many products to be read as learning data, the workload becomes enormous.

[0008] In the case of the invention of Patent Document 2, the processing cannot be performed unless an image database is prepared in advance, in which image data including objects labeled with information relating to their classification is stored.

[0009] In retail stores and other retailers, recognizing products on shelves and understanding their display status is extremely important for a company's marketing strategy. Therefore, there is a demand for knowing how a company's products, or products of competitors, are displayed on shelves in retail stores.

[0010] Therefore, there are systems that take photos of retail store shelves and identify the products displayed on the shelves, and machine learning is used to identify the products. In this machine learning system, as in the above, a learning process is performed using training data, but a wide variety of products are displayed on the shelves. In this case, too, a learning model has conventionally been generated using training data in which the displayed products are associated with their identification information as labels.

[0011] However, even in this case, the burden of preparing the learning data is large, and the burden of generating the learning model is also large. [Means for solving the problem]

[0012] In view of the above problems, the present inventors have invented an information processing system that efficiently generates a learning model used in machine learning.

[0013] The first invention is an information processing system that performs processing related to a learning model used in machine learning, and includes: a first processing unit that identifies object identification information corresponding to tentative identification information using a first learning model generated using object image information and tentative identification information associated with the object image information; a second processing unit that performs verification processing of the output result of object identification information corresponding to image information input to the first learning model; and a second model generation processing unit that performs machine learning learning processing using learning data verified in the second processing unit to generate a second learning model.

[0014] By configuring as in the present invention, it is possible to efficiently generate a learning model used in machine learning.

[0015] In the above-mentioned invention, the second processing unit can be configured as an information processing system having an image information reception processing unit that receives input of image information, a second recognition processing unit that inputs the received image information into the first learning model and outputs at least one tentative identification information corresponding to the image information, a second output processing unit that outputs object identification information corresponding to the output tentative identification information, and a verification processing unit that extracts image information of the output object identification information from a predetermined memory area, uses the extracted image information and the object identification information to perform a verification process for the object identification information of the input image information, and uses the verified object identification information and the image information as learning data.

[0016] As in the present invention, the output results are verified not only in the first process but also in the second process, so that a more accurate learning model can be generated efficiently.

[0017] In the above-mentioned invention, the second recognition processing unit can be configured as an information processing system, which outputs tentative identification information and a recognition coefficient corresponding to the image information, and the second output processing unit outputs object identification information corresponding to the output tentative identification information and sorts the object identification information based on the recognition coefficient.

[0018] Sorting the output results from the first learning model can speed up the verification process.

[0019] In the above-mentioned invention, the verification processing unit can be configured as an information processing system that displays the extracted object identification information and its image information, and performs the verification processing by accepting the selection of the correct result of the image information input into the first learning model from a computer used by the verifier.

[0020] The accuracy of the verification process can be increased if a verifier checks the output results.

[0021] In the above-mentioned invention, the verification processing unit can be configured as an information processing system that performs the verification processing by comparing image information corresponding to the extracted object identification information with image information stored in a specified memory area.

[0022] In the above-mentioned invention, the verification processing unit can be configured as an information processing system that performs a verification process in which image information corresponding to the extracted object identification information is compared with sample information stored in a specified memory area, and the object identification information of the sample information determined to have the highest similarity is determined to be the object identification information of the image information input to the first learning model.

[0023] Since the verification process would be burdensome if it were performed by a verifier, it may be automated as in the present invention.

[0024] In the above-mentioned invention, the first processing unit can be configured as an information processing system having a classification processing unit that classifies image information of multiple objects, a tentative identification information processing unit that associates the classified groups with tentative identification information, a first model generation processing unit that performs a machine learning learning process using learning data including image information of the objects included in the groups and the tentative identification information to generate the first learning model, a first recognition processing unit that inputs accepted sample information into the first learning model to output tentative identification information, and a first output processing unit that associates the output tentative identification information with object identification information corresponding to the sample information.

[0025] By configuring as in the present invention, it is possible to efficiently generate a learning model used in machine learning.

[0026] In the above-mentioned invention, the information processing system can be configured as an information processing system that inputs image information to be identified into the learning model, and outputs object identification information corresponding to the provisional identification information output by the learning model as an output value.

[0027] With the configuration of the present invention, when image information of an identification target is input to the learning model, object identification information can be output.

[0028] In the above-described invention, the information processing system can be configured as an information processing system having a cleansing processing unit that performs a cleansing process to exclude object image information that may contain errors from the image information of objects included in the classified group.

[0029] By performing a cleansing process as in the present invention, it is possible to remove object image information contained in a group that may contain errors, thereby improving the accuracy of image classification.

[0030] In the above-described invention, the cleansing processing unit can be configured as an information processing system that determines image information of objects to be excluded using an information distance from a reference value calculated using index values ​​of image information of objects included in the group.

[0031] In the above-described invention, the cleansing processing unit can be configured as an information processing system that calculates a reference value using an index value of image information of the object included in the group, calculates an information distance using the reference value and the index value of the image information of the object, and if there is a deviation of the information distance that is equal to or greater than a predetermined threshold value or a predetermined ratio, determines that the image information of the object is image information of an object to be excluded.

[0032] In the above-described invention, the cleansing processing unit can be configured as an information processing system that determines image information of objects to be excluded using the similarity of image information of objects included in the group.

[0033] In the above-described invention, the cleansing processing unit can be configured as an information processing system that calculates the similarity between image information of an object included in the group and image information of other objects included in the group, and if a predetermined condition using the similarity is satisfied, determines the image information of that object as image information of an object to be excluded.

[0034] The cleansing process can be carried out as in these inventions.

[0035] In the above-mentioned invention, the object is a product displayed on a display shelf, and the classification processing unit can be configured as an information processing system that cuts out image information of the displayed product from image information of the display shelf and extracts it as object image information, compares the similarity of the object image information of adjacent products, and if the similarity satisfies certain conditions, classifies the object image information of the adjacent products into the same group.

[0036] Products of the same type are often displayed adjacent to each other on display shelves. Therefore, when objects are displayed on a display shelf, there is a possibility that the object image information of adjacent products can be classified as the same type, i.e., the same group. Therefore, it is preferable to compare the similarity of the object image information of adjacent products and, if the similarity satisfies the conditions, classify them into the same group.

[0037] In the above-mentioned invention, the target object is a product displayed on a display shelf, and the first recognition processing unit can be configured as an information processing system that inputs sample information of products corresponding to the sales amount received as input into the learning model and outputs provisional identification information corresponding to the sample information received by the learning model.

[0038] In the case of a learning model that identifies products displayed on a shelf, the number of products can be in the thousands or even tens of thousands. In such cases, inputting sample information for all products is a heavy burden.

[0039] Generally, a small number of best-selling products account for the majority of sales. And it is highly likely that best-selling products are displayed on display shelves. Therefore, if best-selling products that are likely to be identified are input as sample information, even if it is not possible to identify all products displayed on display shelves, it will be possible to recognize many products based on sales figures, and a system that is sufficient for practical purposes can be constructed.

[0040] In the above-mentioned invention, the target object is a commodity displayed on a display shelf, and the first recognition processing unit can be configured as an information processing system that inputs sample information of the commodity for each organization that receives input into the learning model and outputs provisional identification information corresponding to the sample information received by the learning model.

[0041] When using this invention for marketing, there are cases where a specific organization, such as a specific company, wants to understand the display status of its own or its competitors' products. Therefore, by inputting sample product information for each organization, as in the present invention, even if it is not possible to identify all the products displayed on the shelves, it is possible to obtain important marketing information for that organization, and a system that is sufficient for practical purposes can be constructed.

[0042] The first invention can be realized by loading the program of the present invention into a computer and executing it. That is, the information processing program causes the computer to function as a first processing unit that identifies object identification information corresponding to tentative identification information using a first learning model generated using object image information and tentative identification information associated with the object image information, a second processing unit that performs a verification process on the output result of the object identification information corresponding to the image information input to the first learning model, and a second model generation processing unit that performs a machine learning learning process using the learning data verified by the second processing unit to generate a second learning model. [Effects of the Invention]

[0043] By using the information processing system of the present invention, it is possible to efficiently generate a learning model used in machine learning. [Brief explanation of the drawings]

[0044] [Figure 1] FIG. 1 is a diagram showing an example of a conceptual diagram of processing in an information processing system according to the present invention. [Figure 2] FIG. 1 is a diagram showing an example of a conceptual diagram of processing in an information processing system according to the present invention. [Figure 3] FIG. 1 is a diagram showing an example of a conceptual diagram of processing in an information processing system according to the present invention. [Figure 4] FIG. 1 is a diagram showing an example of a conceptual diagram of processing in an information processing system according to the present invention. [Figure 5] 1 is a block diagram schematically illustrating an example of a configuration of an information processing system according to the present invention. [Figure 6]FIG. 2 is a block diagram schematically illustrating an example of a hardware configuration of a computer used in the information processing system of the present invention. [Figure 7] 10 is a flowchart showing an example of a process for generating a first learning model in the information processing system of the present invention. [Figure 8] 10 is a flowchart showing an example of a process for generating a second learning model in the information processing system of the present invention. [Figure 9] FIG. 10 is a diagram showing an example of image information obtained by photographing a display shelf. [Figure 10] FIG. 10 is a diagram showing an example of a state in which object image information is extracted from image information obtained by photographing a display shelf. [Figure 11] 10 is a diagram showing an example of a state in which object image information is classified in a classification processing unit. FIG. [Figure 12] 10 is a diagram showing an example of a state in which temporary identification information is associated in a temporary identification information processing unit. FIG. [Figure 13] FIG. 10 is a diagram showing an example of a state in which sample information is input to a learning model and tentative identification information is output as an output value. [Figure 14] FIG. 10 is a diagram illustrating an example of a state in which temporary identification information and object identification information are linked together. [Figure 15] FIG. 10 is a diagram illustrating an example of the configuration of an information processing system according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0045] 1 to 4 show examples of conceptual diagrams of processing by the information processing system 1 of the present invention. FIG. 5 shows an example of a processing function of the information processing system 1 of the present invention in a block diagram. The information processing system 1 is a system that generates a model that recognizes objects such as products displayed on a display shelf from image information using a machine learning model. The object will be described using products displayed on a display shelf, but is not limited to this.

[0046] The management terminal 2 in the information processing system 1 is realized using a computer. An example of the hardware configuration of a computer is shown in Figure 6. The computer has a CPU and the like that executes the arithmetic processing of a program, a storage device 71 such as RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or a mouse that can input information, and a communication device 74 that sends and receives the processing results of the arithmetic device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0047] If the computer is equipped with a touch panel display, the display device 72 may be integrated with the input device 73. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited to these.

[0048] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

[0049] The functions of the various means in the present invention are only logically distinct, and may be physically or practically the same area. The order of the processes in the various means of the present invention may be changed as appropriate. Also, some of the processes may be omitted.

[0050] The management terminal 2 has a first processing unit 20, a first learning model memory unit 21, an image information memory unit 22, a second processing unit 23, a learning information memory unit 24, a second model generation processing unit 25, and a second learning model memory unit 26.

[0051] The first processing unit 20 generates a learning model (first learning model) using image information (object image information) of an object captured in, for example, image information of a display shelf, and tentative identification information (described later) associated with the object image information, and identifies object identification information corresponding to the tentative identification information. The first processing unit 20 has a classification processing unit 200, a tentative identification information processing unit 201, a first model generation processing unit 202, a sample information reception processing unit 203, a first recognition processing unit 204, and a first output processing unit 205.

[0052] The classification processing unit 200 classifies the object image information by clustering processing or the like from image information of objects (object image information) that appear in, for example, image information of a display shelf. If the object is a product displayed on a display shelf, the object image information may be image information of the product, such as image information obtained by cutting out an area including the displayed product, which is the object, from image information of the display shelf. In this case, the cutout may be rectangular, or may be image information cut out in any shape corresponding to the outer shape of the product.

[0053] Identification information (object identification information) of an object shown in the object image information, for example, if the object is a product, the product name or a code such as a JAN code, does not need to be known.

[0054] The classification processing unit 200 performs clustering processing on a plurality of pieces of object image information using a known method, and classifies the image information into a plurality of groups.

[0055] Although the classification processing unit 200 has been described as classifying the object image information using clustering processing, the object image information may be classified using a method other than clustering processing.

[0056] The temporary identification information processing unit 201 associates temporary identification information (temporary identification information) with each group classified by the classification processing unit 200. The temporary identification information may be automatically generated, or input temporary identification information may be associated by a predetermined operation. Then, the temporary identification information associated with the object image information classified into each group is associated as a label to form a data set, which is used as learning data.

[0057] The first model generation processing unit 202 performs machine learning using the learning data by a known method to generate a learning model. That is, since temporary identification information is associated with each classified group of object image information, a learning model (first learning model) is generated using learning data that uses a data set in which the object image information and its label are used as temporary identification information.

[0058] The first model generation processing unit 202 stores the generated learning model (first learning model) in the first learning model storage unit 21.

[0059] The specimen information reception processing unit 203 receives input of image information to be input to the first model generation processing unit 202. The image information received as input here is object image information for which object identification information for identifying the object is known. For example, it is image information of a product that may be displayed on a display shelf, and for which product identification information (object identification information) such as the product name, JAN code, or other code of the corresponding product is known. The image information input here will be referred to as specimen information.

[0060] The first recognition processing unit 204 inputs the image information (sample information) received by the sample information receiving processing unit 203 into the first learning model stored in the first learning model storage unit 21, and outputs provisional identification information. In this case, when sample information is input into a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, provisional identification information is output as an output value. When multiple output values ​​are output together with their recognition coefficients (reliability), the output value with the highest recognition coefficient (reliability) is adopted.

[0061] The first output processing unit 205 links the tentative identification information, which is the output value output by the first recognition processing unit 204, with the object identification information of the sample information used for its input. Then, the first output processing unit 205 outputs the object identification information linked to the tentative identification information output by the first recognition processing unit 204 as the recognition result of the first recognition processing unit 204.

[0062] In addition, instead of outputting object identification information linked to the provisional identification information output by the first recognition processing unit 204, the first output processing unit 205 may replace the object identification information with the provisional identification information associated with the group output by the classification processing unit 200, and cause the first model generation processing unit 202 to perform the machine learning learning process again.

[0063] Furthermore, if the recognition coefficient (reliability) of the tentative identification information output by the first recognition processing unit 204 is equal to or less than a predetermined threshold, the processing of the first output processing unit 205 does not need to be executed.

[0064] By executing the above-described processing in the first processing unit 20, it is possible to efficiently generate a learning model (first learning model) used in machine learning.

[0065] The image information storage unit 22 stores image information and corresponding object identification information. For example, it stores object image information captured in image information of a display shelf and corresponding object identification information. Note that the image information storage unit 22 may store provisional identification information in addition to the object identification information. The image information storage unit 22 may also store specimen information received by the specimen information reception processing unit 203.

[0066] The second processing unit 23 inputs image information to the first learning model stored in the first learning model storage unit 21, verifies the output value for the input, and uses the input as learning data for generating a second learning model in the second model generation processing unit 25 described later. The second processing unit 23 has an image information reception processing unit 230, a second recognition processing unit 231, a second output processing unit 232, and a verification processing unit 233.

[0067] The image information reception processing unit 230 receives input of image information to be input to the first learning model stored in the first learning model storage unit 21. The image information received as input here may be object image information that was not used in the classification process by the classification processing unit 200, or other image information. For example, it may be object image information that appears in image information of a display shelf. The image information stored in the image information storage unit 22 may be image information used in the processing by the second processing unit 23.

[0068] The second recognition processing unit 231 inputs the image information received by the image information receiving processing unit 230 to the first learning model stored in the first learning model storage unit 21. The first learning model outputs at least one piece of tentative identification information and its recognition coefficient (reliability) as output values ​​for the input image information. In this case, when image information is input to a learning model (first learning model) in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, tentative identification information and its recognition coefficient (reliability) are output as output values.

[0069] The second output processing unit 232 links the corresponding object identification information to the tentative identification information that is the output value output by the second recognition processing unit 231. Then, the second output processing unit 232 outputs the object identification information linked to the tentative identification information output by the second recognition processing unit 231 and a recognition coefficient (reliability) as the recognition result of the second recognition processing unit 231.

[0070] Furthermore, if the recognition coefficient (reliability) of the tentative identification information output by the second recognition processing unit 231 is equal to or less than a predetermined threshold, it is not necessary to execute the processing of the second output processing unit 232. Furthermore, the second recognition processing unit 231 or the second output processing unit 232 only needs to output tentative identification information or object identification information with a recognition coefficient (reliability) equal to or greater than a predetermined threshold.

[0071] Furthermore, the second output processing unit 232 sorts and outputs the results in descending order of the recognition coefficients (reliabilities) based on the recognition coefficients (reliabilities).

[0072] For example, when the second recognition processing unit 231 inputs the image information received by the image information reception processing unit 230 into the first learning model, and the output values ​​are provisional identification information "XXX", "XXY", and "XXZ", with recognition coefficients of "0.58423", "0.32014", and "0.03293", respectively, the second output processing unit 232 identifies the object identification information "product X" corresponding to the provisional identification information "XXX", the object identification information "product Y" corresponding to the provisional identification information "XXY", and the object identification information "product Z" corresponding to the provisional identification information "XXZ" (the correspondence between the provisional identification information and the object identification information is performed by the first processing unit 20), and sorts and outputs the provisional identification information and / or the object identification information based on their respective recognition coefficients.

[0073] The verification processing unit 233 identifies image information stored in the image information storage unit 22 based on the tentative identification information and / or object identification information output by the second output processing unit 232. Then, a list of the output results is displayed, and the verifier selects the object identification information corresponding to the input image information. For example, if the second output processing unit outputs the object identification information "Product X," "Product Y," and "Product Z" and sorts them by recognition coefficient (reliability), the verification processing unit 233 extracts image information corresponding to "Product X," "Product Y," and "Product Z" from the image information storage unit 22. Then, the extracted image information and its object identification information are displayed as a list on a predetermined computer used by the verifier.

[0074] The verification processing unit 233 accepts the selection of the correct output result for the input image information from the list. For example, if the verifier determines that "Product X" is the correct output result for the input image information out of "Product X," "Product Y," and "Product Z," the input image information and the object identification information "Product X" are associated with each other and stored as a data set of learning data in the learning information storage unit.

[0075] As described above, the second processing unit 23 inputs each image information into the first learning model stored in the first learning model storage unit 21, has a verifier verify the object identification information corresponding to the output value, and accepts the input, thereby storing a learning data set (learning data set) in which the image information and the object identification information are associated in the learning information storage unit. Then, learning data having a plurality of learning data sets can be generated.

[0076] Furthermore, the learning data stored in the learning information storage unit has been verified by a verifier, and since it is only necessary to input image information and verify it, highly accurate learning data can be generated with simple operations.

[0077] The learning information storage unit 24 stores learning data having a learning data set in which image information and object identification information are stored in association with each other. The image information is preferably image information of an object.

[0078] The second model generation processing unit 25 generates a learning model by performing machine learning using a known method, using learning data having a learning dataset stored in the learning information storage unit 24. That is, a learning model (second learning model) is generated using image information in the learning dataset and learning data whose labels are used as object identification information.

[0079] The second model generation processing unit 25 stores the generated learning model (second learning model) in the second learning model storage unit .

[0080] The second learning model stored in the second learning model memory unit is machine-learned using learning data in which image information and object identification information are associated, so that when image information is input, the corresponding object identification information is output. [Example]

[0081] Next, an example of processing by the information processing system 1 of the present invention will be described with reference to the flowcharts of Figures 7 and 8. Figure 7 is a flowchart showing processing for generating a first learning model using the first processing unit 20 and identifying object identification information corresponding to tentative identification information. Figure 8 is a flowchart showing processing for inputting image information into the first learning model using the second processing unit, verifying the output result, and then generating learning data for generating a second learning model.

[0082] In the following description, a case will be described in which products displayed on a display shelf are identified from image information obtained by photographing the display shelf. In this case, the target objects are the products displayed on the display shelf, and the target identification information is the product identification information.

[0083] First, image information captured of a display shelf displaying products in a store or the like is corrected so that the products are positioned in a frontal view. Various known methods can be used for this correction, such as trapezoidal correction. If the image information captured of the display shelf is captured from a frontal view, no correction process is required. The image information may be of the entire display shelf, or may be of a portion of the display shelf, such as one or more shelf levels. Figure 9 shows an example of image information of the shelf level area of ​​a display shelf corrected to a frontal view.

[0084] Then, from the corrected image information or image information of a display shelf (image information to be processed), the area of ​​the displayed products is cut out using a known method and extracted as object image information. For example, the product area can be identified by identifying thin, narrow shadows between products, identifying repetitive patterns in the image, identifying steps on the top edge of the packaging, or identifying separator positions based on constraints such as the product width being the same, from the corrected image information or image information of a display shelf. Alternatively, the product area can be identified using machine learning such as deep learning. In this case, image information of the area to be processed, such as a display shelf or shelf area, can be input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the product area can be identified based on the output value. Alternatively, the learning model can be one in which the product area is provided as ground truth data in image information of various areas to be processed, such as a display shelf or shelf area.

[0085] Fig. 10 shows an example of image information obtained by extracting object image information from image information of the shelf area of ​​the display shelf in Fig. 9. At this point, the object image information has only been extracted from image information showing the display shelf, so the product identification information of the products that are the objects shown in the object image information does not need to be known.

[0086] Then, the classification processing unit 200 performs a known clustering process or the like using the object image information to group similar image information together and classify the information into a plurality of groups (S100). This state is shown schematically in FIG.

[0087] The temporary identification information processing unit 201 associates temporary identification information with each group classified by the classification processing unit 200 (S110). For example, arbitrary temporary identification information such as "a123," "b456," "c789," and "d012" is associated with each group. This state is schematically shown in FIG. 12. In this way, the temporary identification information is associated as a label with the object image information of each group. A data set using this object image information and the labels of the temporary identification information is then used as learning data.

[0088] When the classification processing unit 200 associates provisional identification information with each group classified, the first model generation processing unit 202 performs machine learning using a known method as learning data (S120) to generate a learning model (first learning model) (S130). The generated learning model (first learning model) is stored in the first learning model storage unit.

[0089] Next, the specimen information reception processing unit 203 receives input of specimen information, which is image information for which object identification information is known, and the first recognition processing unit 204 inputs the received input of specimen information to the learning model (first learning model) generated in S130 (S140). The first recognition processing unit 204 receives input of specimen information and outputs an output result based on machine learning using the learning model (first learning model). As this output result, tentative identification information is output (S150). This is schematically shown in FIG. 13.

[0090] The specimen information is image information for which object identification information, such as product identification information, is known in advance. Therefore, the tentative identification information output by the first recognition processing unit 204 is considered to correspond to the object identification information of the specimen information input to the learning model. Therefore, the first output processing unit 205 associates the output tentative identification information with the object identification information of the input specimen information (S160). For example, as shown in FIG. 14, the tentative identification information and the object identification information are associated and stored in a predetermined storage area.

[0091] This linking allows the provisional identification information to be associated with the object identification information. By storing this association, the provisional identification information output by the first recognition processing unit 204 can be replaced with the corresponding object identification information and output. When image information to be subjected to identification processing is input into the learning model, the object identification information can be output as the output value of the learning model.

[0092] Next, the image information reception processing unit 230 of the second processing unit 23 receives input of image information to be processed (S200), and the second recognition processing unit 231 inputs the received input image information into a learning model (first learning model) stored in the first learning model memory unit 21 (S210).

[0093] The second recognition processing unit 231 outputs an output result based on machine learning using a learning model (first learning model), which includes one or more pieces of provisional identification information corresponding to the input image information and their recognition coefficients (reliability).

[0094] The second output processing unit 232 links the corresponding object identification information to the tentative identification information that is the output value output by the second recognition processing unit 231. Then, the second output processing unit 232 outputs the object identification information linked to the tentative identification information output by the second recognition processing unit 231 and the recognition coefficient (reliability) as the recognition result of the second recognition processing unit 231 (S220).

[0095] Furthermore, the second output processing unit 232 sorts the recognition coefficients (reliabilities) in descending order based on the recognition coefficients (reliabilities) (S230).

[0096] The verification processing unit 233 identifies the image information stored in the image information storage unit 22 based on the object identification information output by the second output processing unit 232 (S240), displays a list of the output results, and executes the verification process by having the verifier select the object identification information corresponding to the input image information (S250). The verification processing unit 233 displays a list of the object identification information corresponding to the output result of the learning model (first learning model) and its image information on a specific computer used by the verifier, and accepts the selection of the object identification information that is correct as the object appearing in the input image information. The verification processing unit 233 may display the input image information together with the list.

[0097] The object identification information whose selection has been accepted is then associated with the input image information, and stored as a learning data set in the learning information storage unit 24 (S260).

[0098] By performing the above processing on multiple pieces of image information in the second processing unit 23, learning data having a learning data set that associates image information with object identification information can be stored in the learning information storage unit 24.

[0099] At a predetermined timing, the second model generation processing unit 25 performs machine learning using a known method using learning data having a learning dataset stored in the learning information storage unit 24 (S270), and generates a learning model (second learning model) (S280). That is, the learning model (second learning model) is generated using image information in the learning dataset and learning data whose labels are used as object identification information. The learning model (second learning model) generated by the second model generation processing unit 25 is stored in the second learning model storage unit 26.

[0100] Through the above-described processing, a learning model (second learning model) can be generated by machine learning using learning data in which image information and object identification information are associated with each other. [Example]

[0101] The classification process performed by the classification processing unit 200 may be less accurate than classification process performed by a human being. As a result, errors may be mixed in. Therefore, a cleansing processing unit 26 may be provided that performs a cleansing process on the image information (target image information) of each group classified by the classification processing unit 200, and removes target image information that may contain errors from the image information classified into the groups, thereby reducing the amount of target image information to be used as learning data for the first model generation processing unit 202. An example of the configuration of the information processing system 1 in this case is shown in FIG. 15.

[0102] That is, when the classification processing unit 200 classifies the object image information into groups, it calculates index values ​​such as feature amounts that indicate the characteristics of the image information in that group. Then, it calculates the information distance from a reference value such as the median or average value of the index values ​​for each object image information in that group (a value calculated using a predetermined formula to calculate the deviation from the reference value to the index value for each image information), and object image information whose information distance deviates by a certain value or a certain ratio or more may be excluded from the group.

[0103] In addition to using information distance for the cleansing process, similarity of image information may also be used. In this case, the similarity between image information in a group is quantified, and if the number of images with a calculated similarity above a certain level is equal to or greater than a predetermined value, the image information is retained as object image information, and if the number is less than the predetermined value, the image information is excluded from the object image information.

[0104] For example, if a group contains five pieces of target image information (image information 1 to image information 5), the similarity between each of the five pieces of image information and the other pieces of image information is calculated. That is, the similarity between image information 1 and image information 5 is calculated, between image information 2 and image information 5, between image information 2 and image information 5, between image information 3 and image information 5, between image information 4 and image information 5, and between image information 4 and image information 5. In this way, the similarity between each piece of image information can be calculated.

[0105] Assume that for image information 1, the similarity with image information 2 is 0.9, the similarity with image information 3 is 0.95, the similarity with image information 4 is 0.3, and the similarity with image information 5 is 0.8; for image information 2, the similarity with image information 3 is 0.8, the similarity with image information 4 is 0.2, and the similarity with image information 5 is 0.6; for image information 3, the similarity with image information 4 is 0.4 and the similarity with image information 5 is 0.9; and for image information 4, the similarity with image information 5 is 0.5.

[0106] Here, if the similarity of the reference image information is 0.75 and the number of pieces of image information below the reference level is three, then for image information 1, only one piece of image information 4 falls below the reference similarity level; for image information 2, two pieces of image information 4 and image information 5 fall below the reference similarity level; for image information 3, only one piece of image information 4 falls below the reference similarity level; for image information 4, four pieces of image information 1, 2, 3, and 5 fall below the reference similarity level; and for image information 5, two pieces of image information 2 and 4 fall below the reference similarity level.

[0107] Therefore, the cleansing processing unit 26 excludes image information 4, which has three or more pieces of image information below the standard similarity, from the group, and executes processing by the provisional identification information processing unit 201 with four pieces of object image information for the group: image information 1, image information 2, image information 3, and image information 5.

[0108] By excluding the deviated image information from the grouped object image information as in this embodiment, it is possible to narrow down the grouped object image information, which leads to an improvement in accuracy. [Example]

[0109] When extracting object image information by cutting out products displayed on a display shelf, it is common for the same products to be placed adjacent to each other in either the top, bottom, left, or right direction.

[0110] Therefore, after extracting object image information from image information photographed of the display shelf, the classification processing unit 200 compares the similarity of the object image information of adjacent products, and if it satisfies certain conditions, for example, if the similarity is equal to or greater than a predetermined threshold, it may determine that the objects are of the same type and classify the object image information of the adjacent products into the same group. [Example]

[0111] When targeting products displayed on shelves, it is preferable to generate learning models of products that may be displayed in advance. However, in this case, the target products may number from hundreds to thousands, or even tens of thousands. In this case, if a learning model capable of recognizing thousands to tens of thousands of products is used, the sample information input by the sample information reception processing unit 203 will be in the thousands to tens of thousands. Although the workload is reduced compared to conventional learning models, the workload is still significant.

[0112] On the other hand, it is known in marketing that generally about 10% of product types tend to account for about 90% of the sales of their classification. Therefore, products with high sales may be used as the specimen information to be input to the specimen information reception processing unit 203. In this case, information provided by a marketing company or the like may be used.

[0113] This makes it possible to recognize about 90% of products based on sales, which is sufficient for practical purposes as output of object identification information. Note that when a product with low sales is input to the learning model, the object identification information is not linked to the provisional identification information, and the first output processing unit 205 will output the provisional identification information as is. When the first output processing unit 205 outputs provisional identification information that does not associate the provisional identification information with the object identification information, it may display a predetermined message such as "no correspondence information."

[0114] Furthermore, if it is sufficient to recognize only products of a specific company or other organization, image information of the company's products may be input as sample information to the sample information reception processing unit 203. This allows object identification information to be output for the company's products. On the other hand, when a product of a company other than the specific company is input, as in the above case, the object identification information and the provisional identification information are not linked, so the first output processing unit 205 will output the provisional identification information as is. When the first output processing unit 205 outputs provisional identification information that does not have a correspondence between the provisional identification information and the object identification information, it may display a predetermined message such as "no correspondence information." [Example]

[0115] For the purpose of improving accuracy, the learning model (first learning model) generated as in Examples 1 to 4 may be reconstructed again. In this case, new sample information may be input, and new object identification information may be linked to tentative identification information to which no object identification information has been linked, or the classification processing unit 200 may perform classification processing using object image information of a new object so that the new object can be recognized. [Example]

[0116] As another example of Examples 1 to 5, the verification process in the verification processing unit 233 in the second processing unit 23 may be performed by a computer rather than by a verifier. For example, the verification processing unit 233 may compare image information identified from the image information storage unit 22 with image information stored in another storage area, for example, image information stored as sample information, and determine the object identification information of the sample information determined to have the highest similarity as the object identification information of the image information input by the second recognition processing unit 231 to the first learning model. Furthermore, the comparison may be made with image information on the Internet instead of the sample information.

[0117] When performing such processing, it is not necessary to provide output of recognition coefficients, sorting processing, etc. [Example]

[0118] In the above-described first to sixth embodiments, a learning model (first learning model) is generated for identifying products displayed on a display shelf from image information of the display shelf. However, the present invention can also be applied to other cases. In particular, the present invention is useful for automating the identification of various types of objects from image information.

[0119] As an example, the target may be an animal. For example, this method can be applied to cases where multiple types of animals, such as sea lions and fur seals, live in large numbers in places where people cannot approach, and the types and populations of these animals need to be efficiently identified. In this case, the habitat is photographed from the air using a drone or other device, and object image information for each individual is extracted. The object image information for each individual is then subjected to image classification processing by the classification processing unit 200 and grouped. Then, using learning data generated by the temporary identification information processing unit 201 by associating temporary identification information with each group, the first model generation processing unit 202 performs machine learning to generate a learning model (first learning model). The specimen information reception processing unit 203 then accepts input of specimen information by type, and the first recognition processing unit 204 inputs the accepted specimen information as input values ​​into the learning model (first learning model), causing it to output temporary identification information. This links the temporary identification information to the object identification information (such as the animal's scientific name or appellation) corresponding to the input specimen information, enabling the first output processing unit 205 to output the object identification information.

[0120] Then, the second recognition processing unit 231 inputs the image information received by the image information reception processing unit 230 into a learning model (first learning model), and the verification processing unit 233 performs verification processing using the output result of the second output processing unit 232, thereby generating learning data having a data set for learning. Then, the second model generation processing unit 25 can generate a new learning model (second learning model) using the learning data.

[0121] As in the above, the target object may be a bird or a plant. In the case of a bird or a plant, processing can be performed in the same way as for the above-mentioned animal, and processing can be performed by replacing "animal" with "bird" or "plant." [Industrial Applicability]

[0122] By using the information processing system 1 of the present invention, it is possible to efficiently generate a learning model used in machine learning. [Explanation of symbols]

[0123] 1: Information processing system 2: Management terminal 20: First processing section 21: First learning model memory unit 22: Image information storage unit 23: Second processing section 24: Learning information storage unit 25: Second model generation processing unit 26: Second learning model memory unit 70: Arithmetic device 71:Storage device 72:Display device 73: Input device 74:Communication equipment 200: Classification processing unit 201: Temporary identification information processing unit 202: First model generation processing unit 203: Specimen information reception processing unit 204: First recognition processing unit 205: First output processing unit 206: Cleansing processing section 230: Image information reception processing unit 231: Second recognition processing unit 232: Second output processing unit 233: Verification processing unit

Claims

1. An information processing system that performs processing related to a learning model used in machine learning, a first processing unit that identifies object identification information corresponding to the tentative identification information using a first learning model generated using object image information and tentative identification information associated with the object image information; a second processing unit that performs a verification process on an output result of object identification information corresponding to the image information input to the first learning model; a second model generation processing unit that executes a machine learning learning process using the learning data verified by the second processing unit to generate a second learning model; An information processing system comprising:

2. The second processing unit is an image information reception processing unit that receives input of image information; a second recognition processing unit that inputs the received image information into the first learning model and outputs at least one tentative identification information corresponding to the image information; a second output processing unit that outputs object identification information corresponding to the output tentative identification information; a verification processing unit that extracts image information of the output object identification information from a predetermined storage area, performs verification processing of the object identification information of the input image information using the extracted image information and the object identification information, and uses the verified object identification information and the image information as learning data; 2. The information processing system according to claim 1, further comprising:

3. The second recognition processing unit outputting provisional identification information and a recognition coefficient corresponding to the image information; The second output processing unit outputting object identification information corresponding to the output tentative identification information, and sorting the object identification information based on the recognition coefficient; 3. The information processing system according to claim 2.

4. The verification processing unit The extracted object identification information and its image information are displayed, and a selection of a correct result for the image information input to the first learning model is received from a computer used by a verifier, thereby performing the verification process.

4. The information processing system according to claim 2 or 3.

5. The verification processing unit performing the verification process by comparing image information corresponding to the extracted object identification information with image information stored in a predetermined storage area; 4. The information processing system according to claim 2 or 3.

6. The verification processing unit performing a verification process of comparing image information corresponding to the extracted object identification information with sample information stored in a predetermined storage area, and determining that the object identification information of the sample information determined to have the highest similarity is the object identification information of the image information input to the first learning model; 6. The information processing system according to claim 5.

7. The first processing unit a classification processing unit that classifies image information of a plurality of objects; a temporary identification information processing unit that associates the classified groups with temporary identification information; a first model generation processing unit that executes a machine learning learning process using learning data including image information of the objects included in the group and the provisional identification information to generate the first learning model; a first recognition processing unit that inputs the received sample information into the first learning model to output provisional identification information; a first output processing unit that associates the output tentative identification information with object identification information corresponding to the specimen information; 2. The information processing system according to claim 1, further comprising:

8. The information processing system includes: inputting image information to be subjected to the identification process into the first learning model, thereby outputting object identification information corresponding to the provisional identification information output by the learning model as an output value; 8. The information processing system according to claim 7,

9. The information processing system includes: a cleansing processing unit that executes a cleansing process to remove object image information that may contain errors from the image information of the objects included in the classified groups; 8. The information processing system according to claim 7, further comprising:

10. The cleansing processing unit includes: determining image information of an object to be excluded using an information distance from a reference value calculated using index values ​​of image information of the objects included in the group; 10. The information processing system according to claim 9.

11. The cleansing processing unit includes: calculating a reference value using index values ​​of image information of the objects included in the group; calculating an information distance using the reference value and an index value of the image information of the object; If the information distance is greater than or equal to a predetermined threshold value or a deviation greater than or equal to a predetermined ratio, the image information of the object is determined to be image information of an object to be excluded.

11. The information processing system according to claim 10.

12. The cleansing processing unit includes: determining image information of an object to be excluded using the similarity of image information of the object included in the group; 10. The information processing system according to claim 9.

13. The cleansing processing unit includes: Calculating a similarity between image information of the object included in the group and image information of other objects included in the group; If a predetermined condition using the similarity is satisfied, the image information of the object is determined to be image information of an object to be excluded.

13. The information processing system according to claim 12.

14. The object is a commodity displayed on a display shelf, The classification processing unit Image information of displayed products is extracted from the image information of the display shelves as object image information, Compare the similarity of the object image information of adjacent products, If the similarity satisfies a certain condition, the object image information of adjacent products is classified into the same group.

8. The information processing system according to claim 7,

15. The object is a commodity displayed on a display shelf, The first recognition processing unit inputting sample information of products corresponding to the received input sales amount into the learning model, and outputting provisional identification information corresponding to the sample information received by the learning model; 8. The information processing system according to claim 7,

16. The object is a commodity displayed on a display shelf, The first recognition processing unit inputting the sample information of the products for each organization entity that has received the input into the learning model, and outputting provisional identification information corresponding to the sample information received by the learning model; 8. The information processing system according to claim 7,

17. Computer, a first processing unit that identifies object identification information corresponding to the tentative identification information using a first learning model generated using object image information and tentative identification information associated with the object image information; a second processing unit that performs a verification process on the output result of the object identification information corresponding to the image information input to the first learning model; a second model generation processing unit that executes a machine learning learning process using the learning data verified by the second processing unit to generate a second learning model; An information processing program characterized by causing the program to function as:

Images