Information processing system

Abstract

To provide an information processing system related to generation of a learning model in machine learning.SOLUTION: An information processing system which executes processing related to a learning model for use in machine learning includes: a detection processing unit which detects a predetermined target object from image information; and a model processing unit which generates a learning model for machine learning, using the detected target object. The model processing unit includes: a classification processing unit which classifies image information of the detected target object; a temporary identification information processing unit which associates classified groups with temporary identification information; and a model generation unit which executes learning processing for machine learning, using learning data including image information of the target object included in the classified groups and the temporary identification information, to generate a learning model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system used to generate a learning model (network) in machine learning. [Background technology]

[0002] In retail stores and other stores, POP for product sales promotion is sometimes installed on the display shelves where products are displayed. POP is attached, installed, or placed on the products displayed on the shelves, product tags (price cards), shelves, or above or to the sides of the shelves.

[0003] There are two types of POP: manufacturer POP, which is created by the manufacturer of the product and requested to be installed by the retailer, and retail POP (distribution POP), which is created and installed by the retailer. Since the manufacturer requests the retailer to install the manufacturer POP, the manufacturer would like to check whether it is installed as requested.

[0004] In the past, in such cases, a manufacturer's staff would physically visit each store to check, but this was a heavy burden. Therefore, it is thought that an automatic process would be possible.

[0005] An example of automatic processing is machine learning, a method of image analysis processing using a computer. Machine learning requires that a learning data set (learning data) is created in advance and that the computer is then loaded with the data to execute a specific learning process.

[0006] However, preparing learning data in advance itself is a heavy workload. To address this issue, systems such as those shown in Patent Documents 1 and 2 below have been disclosed.

[0007] In the invention of Patent Document 1, a product image is photographed, and a code attached to the product is read, whereby the product image is created and labeled, thereby generating learning data.

[0008] Furthermore, in the invention of Patent Document 2, a classification of an object contained in image data is selected from an image database in which image data including objects that have been labeled with information relating to the classification in advance is stored, and the object is extracted from the image data to generate a template image, thereby generating learning data that is labeled with the template image. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] JP 2020-095537 A [Patent Document 2] Patent Publication No. 2022-076296 Summary of the Invention [Problem to be solved by the invention]

[0010] In the case of the invention of Patent Document 1, the product is prepared, photographed, and the code attached to the product is read with a code reader to generate learning data, which places a heavy burden on the worker. If there are many products to be read as learning data, the amount of work becomes enormous. In addition, while codes are attached to products, codes are not attached to POPs, so the invention of Patent Document 1 cannot be used to generate learning data for POPs.

[0011] In the case of the invention of Patent Document 2, the processing cannot be performed unless an image database is prepared in which image data including objects labeled with classification information is stored in advance. POPs are diverse, and it is not realistic to prepare an image database with image data labeled for each POP. [Means for solving the problem]

[0012] In view of the above problems, the present inventor has invented an information processing system that efficiently generates learning models for use in machine learning. In particular, the information processing system of the present invention is highly effective when used to generate learning models for POPs that are installed on store shelves.

[0013] A first invention is an information processing system that executes processing related to a learning model used in machine learning, and includes a detection processing unit that detects a specified object from image information, and a model processing unit that generates a learning model for machine learning using the detected object, and the model processing unit includes a classification processing unit that classifies image information of the detected object, a provisional identification information processing unit that associates the classified groups with provisional identification information, and a model generation processing unit that executes a machine learning learning process using learning data including image information of objects included in the classified groups and the provisional identification information to generate a learning model.

[0014] By using the present invention, it is possible to efficiently generate a learning model for use in machine learning. In particular, image information used as learning data and its label (type) can be detected from the image information and used as a data set, thereby reducing the workload.

[0015] In the above-mentioned invention, the model processing unit can be configured as an information processing system having a recognition processing unit that inputs the received sample information into the learning model to output provisional identification information, and an output processing unit that matches the output provisional identification information with object identification information of the input sample information.

[0016] A learning model for machine learning may be generated by executing the processes of these inventions.

[0017] In the above-mentioned invention, the model processing unit can be configured as an information processing system that inputs image information to be subjected to identification processing into the learning model, and outputs object identification information corresponding to the provisional identification information output by the learning model as an output value.

[0018] With the configuration of the present invention, when image information of an identification target is input to the learning model, object identification information can be output.

[0019] In the above-mentioned invention, the model processing unit can be configured as an information processing system having a cleansing processing unit that executes a cleansing process to remove object image information that may contain errors from image information of objects included in the classified groups.

[0020] By performing a cleansing process as in the present invention, it is possible to remove object image information that may contain errors from the object image information included in a group, thereby improving the accuracy of classification.

[0021] In the above-mentioned invention, the cleansing processing unit can be configured as an information processing system that determines image information of objects to be excluded by using an information distance from a reference value calculated using an index value of image information of the objects included in the classified group.

[0022] In the above-mentioned invention, the cleansing processing unit can be configured as an information processing system that calculates a reference value using an index value of image information of an object included in the classified group, calculates an information distance using the reference value and the index value of the image information of the object, and if there is a deviation of the information distance by a predetermined threshold value or more or a predetermined ratio or more, determines the image information of the object to be image information of an object to be excluded.

[0023] In the above-mentioned invention, the cleansing processing unit can be configured as an information processing system that determines image information of objects to be excluded using a similarity of image information of objects included in the classified groups.

[0024] In the above-mentioned invention, the cleansing processing unit can be configured as an information processing system that calculates a similarity between image information of an object included in the classified group and image information of other objects included in the group, and if a predetermined condition using the similarity is satisfied, determines the image information of the object as image information of an object to be excluded.

[0025] The cleansing process can be carried out as in these inventions.

[0026] In the above-mentioned invention, the detection processing unit can be configured as an information processing system having an object detection processing unit that detects objects appearing in the image information, and an individual discrimination processing unit that discriminates the type of the detected object.

[0027] With the configuration of the present invention, it is possible to determine whether the type of object captured in the image information is a target object or a candidate object.

[0028] In the above-mentioned invention, the information processing system can be configured as an information processing system having an area detection processing unit that detects an area of ​​the image information, and the individual discrimination processing unit that discriminates the type of the detected object using the detected area and the type of the detected object.

[0029] The type of an object can also be determined based on the area the object is located in. Therefore, by configuring as in the present invention, the type of an object may be determined.

[0030] An eleventh invention is an information processing system that performs processing related to a learning model used in machine learning, the information processing system having an object detection processing unit that detects an object appearing in image information, an individual discrimination processing unit that discriminates the type or candidate type of the detected object, a classification processing unit that classifies image information of the object types, among the object types, into groups for which the learning model is to be generated, a provisional identification information processing unit that associates the classified groups with provisional identification information, and a model generation processing unit that performs a machine learning learning process using learning data including image information of objects included in the groups and the provisional identification information to generate the learning model.

[0031] In the above-mentioned invention, the information processing system can be configured as an information processing system having a recognition processing unit that inputs the received sample information into the learning model to output provisional identification information, and an output processing unit that matches the output provisional identification information with object identification information of the input sample information.

[0032] By using these inventions, it is possible to efficiently generate learning models for use in machine learning. In particular, image information used as learning data and its label (type) can be detected from the image information and used as a dataset, thereby reducing the workload.

[0033] A thirteenth invention is an information processing system that executes processing related to a learning model used in machine learning, the information processing system having a detection processing unit that detects POPs from image information, and a model processing unit that generates a learning model for machine learning using the detected POPs.

[0034] In the above-mentioned invention, the model processing unit can be configured as an information processing system having a classification processing unit that classifies image information of the detected POPs, a provisional identification information processing unit that matches the classified groups with provisional identification information, and a model generation processing unit that uses learning data including image information of POPs included in the group and the provisional identification information to perform a machine learning learning process to generate a learning model.

[0035] By using the present invention, it is possible to efficiently generate a learning model for use in machine learning. In particular, image information used as learning data and its label (type) can be detected from the image information and used as a data set, thereby reducing the workload.

[0036] The inventors of this application have noticed that a typical display shelf is equipped with merchandise such as displayed merchandise, product tags such as product tags, shelf components, and POPs. That is, POPs themselves are not easy to distinguish because the shapes and designs of POPs themselves are diverse, but merchandise, product tags, and shelf components are easy to distinguish. Therefore, if an object detected from image data is not a type of merchandise, product tag, or shelf component, it is determined to be a POP, making it possible to distinguish the POP in the image data.

[0037] In the above-mentioned invention, the model processing unit can be configured as an information processing system having a cleansing processing unit that performs a cleansing process to exclude image information of POPs that may contain errors from the image information of POPs included in the classified group.

[0038] By performing a cleansing process as in the present invention, it is possible to remove object image information that may contain errors from the object image information included in a group, thereby improving the accuracy of classification.

[0039] The 16th invention is an information processing system that performs processing related to a learning model used in machine learning, the information processing system having an object detection processing unit that detects POPs appearing in image information, an individual discrimination processing unit that discriminates the type or candidate type of the detected POP, a classification processing unit that classifies image information of the POP types for which the learning model is generated into groups, a provisional identification information processing unit that matches the classified groups with provisional identification information, and a model generation processing unit that performs a machine learning learning process using learning data including image information of POPs included in the classified groups and the provisional identification information to generate the learning model.

[0040] By using the present invention, it is possible to efficiently generate a learning model for use in machine learning. In particular, image information used as learning data and its label (type) can be detected from the image information and used as a data set, thereby reducing the workload.

[0041] The inventors of this application have noticed that a typical display shelf is equipped with merchandise such as displayed merchandise, product tags such as product tags, shelf components, and POPs. That is, POPs themselves are not easy to distinguish because the shapes and designs of POPs themselves are diverse, but merchandise, product tags, and shelf components are easy to distinguish. Therefore, if an object detected from image data is not a type of merchandise, product tag, or shelf component, it is determined to be a POP, making it possible to distinguish the POP in the image data.

[0042] The first invention can be realized by loading and executing the information processing program of the present invention into a computer. That is, the information processing program causes a computer to function as a detection processing unit that detects a predetermined object from image information and a model processing unit that generates a learning model for machine learning using the detected object, the model processing unit having a classification processing unit that classifies image information of the detected object, a provisional identification information processing unit that associates the classified group with provisional identification information, and a model generation processing unit that executes a machine learning learning process using learning data including image information of objects included in the group and the provisional identification information to generate a learning model. The eleventh invention can be realized by loading the information processing program of the present invention into a computer and executing it. That is, the information processing program causes the computer to function as an object detection processing unit that detects an object captured in image information, an individual discrimination processing unit that discriminates the type or a candidate type of the detected object, a classification processing unit that classifies, by group, image information of a type of object for which a learning model for machine learning is to be generated among the object types, a provisional identification information processing unit that associates the classified groups with provisional identification information, and a model generation processing unit that executes a machine learning learning process using learning data including image information of objects included in the groups and the provisional identification information to generate the learning model.

[0043] The thirteenth aspect of the present invention can be realized by loading and executing the information processing program of the present invention into a computer, that is, the information processing program causes a computer to function as a detection processing unit that detects POPs from image information and as a model processing unit that generates a learning model for machine learning using the detected POPs. The sixteenth aspect of the present invention can be realized by loading and executing the information processing program of the present invention into a computer. That is, the information processing program causes the computer to function as an object detection processing unit that detects POPs captured in image information, an individual discrimination processing unit that discriminates the type or candidate types of the detected POPs, a classification processing unit that classifies, by group, image information of a POP type for which a learning model for machine learning is to be generated among the POP types, a provisional identification information processing unit that associates the classified groups with provisional identification information, and a model generation processing unit that executes a machine learning learning process using learning data including image information of POPs included in the group and the provisional identification information to generate the learning model. Effect of the Invention

[0044] By using the information processing system of the present invention, a learning model for use in machine learning can be efficiently generated. [Brief description of the drawings]

[0045] [Figure 1] FIG. 2 is a block diagram illustrating an example of a processing function of the information processing system of the present invention. [Diagram 2] 4 is a block diagram illustrating an example of a processing function of a detection processing unit of the information processing system of the present invention. FIG. [Diagram 3] FIG. 2 is a block diagram illustrating an example of a processing function of a model processing unit of the information processing system of the present invention. [Figure 4] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer used in the information processing system of the present invention. [Diagram 5] FIG. 2 is a diagram showing an example of a conceptual diagram of a detection processing unit of the information processing system of the present invention. [Figure 6] FIG. 2 is a diagram showing an example of a conceptual diagram of processing in the information processing system of the present invention. [Figure 7] FIG. 2 is a diagram showing an example of a conceptual diagram of processing in the information processing system of the present invention. [Figure 8] FIG. 4 is a diagram showing an example of captured image information. [Figure 9] 4 is a flowchart showing an example of an entire processing process in the information processing system of the present invention. [Figure 10] 10 is a flowchart showing an example of a processing process of a detection processor in the information processing system of the present invention. [Figure 11] 5 is a flowchart showing an example of a processing process of a model processing unit in the information processing system of the present invention. [Figure 12] FIG. 11 is a diagram showing an example of a result of executing an object detection process from image information. [Figure 13] 11A and 11B are diagrams illustrating an example of a process for determining whether a region of an object detected from image information satisfies a predetermined condition to determine the region of the object. [Figure 14] 13 is a diagram showing an example of each area after areas that satisfy a predetermined condition are determined as areas of a target object in relation to the object detection result in FIG. 12. FIG. [Figure 15] FIG. 13 is a diagram showing an example of a state in which object image information is extracted from image information. [Figure 16] 11 is a diagram showing an example of a state in which object image information is classified in a classification processing unit; FIG. [Figure 17] 11 is a diagram showing an example of a state in which temporary identification information is associated in a temporary identification information processing unit; FIG. [Figure 18] FIG. 13 is a diagram illustrating an example of a state in which a learning model is generated using a learning dataset. [Figure 19] 11 is a diagram showing an example of a state in which sample information is input to a learning model and tentative identification information is output as an output value. FIG. [Figure 20] 11 is a diagram showing an example of a state in which temporary identification information and object identification information are linked to each other; FIG. [Figure 21] FIG. 11 is a block diagram illustrating an example of a configuration of a detection processing unit in a second embodiment. [Figure 22] 13 is a flowchart showing an example of a processing process of a detection processing unit in the second embodiment. [Diagram 23] FIG. 11 is a diagram illustrating an example of an outline of a process according to a second embodiment. [Figure 24]FIG. 11 is a diagram illustrating an example of an individual discrimination process in the second embodiment. [Diagram 25] 13 is a diagram showing a schematic diagram of a state in which input of designation of a product display area, a product tag area, and a shelf top area has been received for image information. FIG. [Figure 26] 13 is a flowchart showing an example of a processing process of a detection processing unit in the third embodiment. [Figure 27] FIG. 13 is a block diagram illustrating an example of a configuration of a model processing unit in a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0046] An example of the processing function of the information processing system 1 of the present invention is shown in block diagrams in Figures 1 to 3. Figure 1 shows an example of the overall processing function of the information processing system 1 in a block diagram, Figure 2 shows an example of the processing function of the detection processing unit 20 in a block diagram, and Figure 3 shows an example of the processing function of the model processing unit 21 in a block diagram.

[0047] The information processing system 1 is a system that detects objects captured in predetermined image information, and generates a learning model for use in machine learning using the image information of the various detected objects.

[0048] For example, if the target object is a product displayed on a display shelf, a product tag (price card), a shelf, or a POP that is attached, installed, etc. above or to the side of a shelf, the system detects the POP that appears in the image information of the display shelf that includes the POP, and uses the POP to generate a learning model to be used in machine learning. Note that the target object will be described using the case of a product displayed on a display shelf, a product tag (price card), a shelf, or a POP that is attached, installed, etc. above or to the side of a shelf, but is not limited to this.

[0049] The management terminal 2 and the image information input terminal 3 in the information processing system 1 are realized by using a computer. An example of the hardware configuration of a computer is shown in Fig. 4. The computer has a calculation device 70 such as a CPU that executes the calculation processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or a mouse that can input information, and a communication device 74 that transmits and receives the processing results of the calculation device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0050] In the case where the computer is equipped with a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited to these.

[0051] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display by a predetermined input device (such as a touch panel pen) or a finger.

[0052] The image information input terminal 3 may include a photographing device such as a camera in addition to the above devices. A portable communication terminal such as a mobile phone, a smartphone, or a tablet computer may also be used as the image information input terminal 3. The image information input terminal 3 photographs image information (photographed image information) of a specific object and inputs it to the management terminal 2. For example, a display shelf in a store is photographed and the image information is input to the management terminal 2. In this case, products are displayed on the display shelf, and POPs are attached to the display shelf or the products. Therefore, the photographed image information includes displayed products such as products displayed on the display shelf, product tags, shelf components, POPs, etc. It is preferable that unnecessary objects such as people are not photographed.

[0053] The functions of the various means in the present invention are only logically distinct, and may be physically or practically the same area. The order of the processes in the various means of the present invention may be changed as appropriate. In addition, some of the processes may be omitted.

[0054] The management terminal 2 includes a detection processing unit 20 and a model processing unit 21.

[0055] The detection processing unit 20 detects image information of an area in which an object is captured from the captured image information (object image information). For example, image information of an area in which a POP is captured as object image information from image information of a display shelf is detected (POP image information). FIG. 5 is an example of a conceptual diagram of processing in the detection processing unit 20. FIG. 5(a) is an example of captured image information in which an object is captured, and FIG. 5(b) is an example of a state in which an object has been detected from the captured image information. The captured image information in FIG. 5(a) is a schematic diagram of an image of a store display shelf. FIG. 5(b) is a schematic diagram of an image in which a POP is detected as an object. The shaded portion in FIG. 5(b) is image information of the area of ​​the detected POP.

[0056] The model processing unit 21 generates a learning model to be used in machine learning for the object image information detected by the detection processing unit 20. Examples of conceptual diagrams of the processing in the model processing unit 21 are shown in Fig. 6 and Fig. 7. Fig. 6 is a diagram that illustrates a process of generating a learning model using the object image information detected by the detection processing unit 20. Fig. 7 is a diagram that illustrates a process of linking tentative identification information associated with a group for each classified object image information with identification information of an object (object identification information).

[0057] The detection processing unit 20 includes an image information input reception processing unit 200 , an image information storage unit 201 , an image information alignment processing unit 202 , an object detection processing unit 203 , and an individual discrimination processing unit 204 .

[0058] The image information input reception processing unit 200 receives input of photographed image information in which an object is photographed by the image information input terminal 3, and stores it in the image information storage unit 201 described later. For example, input of image information in which a display shelf in a store is photographed is received, and stores it in the image information storage unit 201 described later. In addition to photographed image information, the image information input terminal 3 may also receive input of accompanying information such as store identification information such as the photographing date and time and the store name, display shelf identification information for identifying the display shelf, and image information identification information for identifying the image information. FIG. 8 shows an example of image information in which a display shelf is photographed as photographed image information. Note that FIG. 8 shows a case in which one display shelf is photographed in the photographed image information, but multiple display shelves may be photographed, or a part of a display shelf may be photographed. In addition, although the processing is not particularly specified in the present invention, display shelves and shelves may be long in the horizontal direction. Therefore, in the processing, the processing may be performed by dividing the image into a certain width and writing within the divided range.

[0059] The image information storage unit 201 stores the captured image information received from the image information input terminal 3 in association with accompanying information such as the capture date and time, store identification information, display shelf identification information, and image information identification information. The captured image information may be any image information that is the subject of processing in the present invention. It also includes image information obtained by combining multiple images of a single display shelf into a single piece of image information when multiple images are captured. The captured image information also includes the corrected image information that has been processed by the image information correcting unit 202 described below, and the image information after distortion correction processing has been performed.

[0060] The image information orthogonalization processing unit 202 executes a correction process (orthogonalization process) for the captured image information stored in the image information storage unit 201 to correct the subject to face upright, and generates image information (orthogonalized image information) in which the captured image information has been orthogonalized. Orthogonalization process is a process for transforming image information so that the optical axis of the lens of the image capture device is aligned along the perpendicular direction of the plane of the subject to be captured, so that the image information becomes the same as when the image is captured from a sufficiently distant location, and includes, for example, keystone correction process. Keystone correction process is a correction process performed so that objects captured in the captured image information, such as shelves of a display shelf, are horizontal, and products displayed there and product tags (for example, price tags) for those products are vertical.

[0061] The image information alignment processing unit 202 receives input of four vertices in the captured image information, and executes keystone correction processing using each of the vertices. The four vertices to be specified may be any four points in the captured image information, for example, if the subject is a display shelf, they may be the four vertices of a shelf level of the display shelf, or the four vertices of the shelf position of the display shelf. They may also be the four vertices of a group of two or three shelf levels. Any four points can be specified as the four vertices. The image information alignment processing unit 202 can use various known processing methods.

[0062] The object detection processing unit 203 executes a process for detecting an object appearing in the captured image information (including orthogonal captured image information; the same applies hereinafter in this specification) stored in the image information storage unit 201. As the object detection process, it is preferable to perform detection using so-called deep learning. In this case, the captured image information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the area of ​​the object appearing in the input captured image information (or its image information) and the type (class) of the object are output as output values.

[0063] As the learning model, a predetermined object that may appear in the captured image information is used as annotation data, and the above-mentioned learning model is learned by executing a predetermined learning process using the annotation data. For example, an object that may appear in the captured image information of a display shelf, such as a product, a product tag (price tag), or a shelf component, is used as annotation data, and the above-mentioned learning model is learned by executing a predetermined learning process using the annotation data. For example, data that associates image information of a product with the fact that it is a product or the product name or product code (JAN code, etc.) is used as annotation data, and a learning process for the learning model is executed. Image information of a product is image information in the annotation data, and is a class indicating that it is a product, or that it is a product and the product identification information such as the product name or product code is "products". Similarly, image information of a product tag is image information in the annotation data, and is a class indicating that it is a product tag, or that it is a product tag and the product name or product identification information is "product tags". Furthermore, image information of a shelf component is image information in the annotation data, and is a class indicating that it is a shelf component or the part name of the shelf component is "shelf component". The types (classes) of objects used as annotation data by the object detection processing unit 203 include things that are generally displayed on display shelves, such as merchandise, product tags, and shelf components, but are not limited to these, and the object detection processing unit 203 may be trained to detect any object.

[0064] The individual discrimination processing unit 204 discriminates that the object type detected as a type other than a predetermined class among the object types (classes) detected from the captured image information by the object detection processing unit 203 is a predetermined object or a candidate for a predetermined object. For example, the object type detected as a type other than merchandise, merchandise tags, and shelf components is a POP or a candidate for a POP. Furthermore, when an object detected as a type (class) of merchandise, merchandise tags, or shelf components satisfies a predetermined condition, the object type (class) is discriminated as a POP or a candidate for a POP.

[0065] For example, for an object whose type is detected as merchandise, it is determined whether or not there is a product-specific advertisement (POP) attached to that area, and if it is determined that there is a POP, the type of object in that area is determined to be a POP or a POP candidate. In this case, there is a method of detecting POP or POP candidates attached to displayed products from the area of ​​the object whose type is detected as merchandise, and a method of detecting POP or POP candidates by comparing image information of the area of ​​the object whose type is detected as merchandise with image information of products without POP.

[0066] In the former method, a large amount of image information including POP parts attached to displayed products is collected in advance, and the POP parts are annotated for each image information including the POP parts. A learning model that has been previously trained using image information of normal displayed products without POPs as annotation data is then additionally trained using the image information annotated with the POP parts to generate a new learning model. Image information of the area of ​​an object detected as a type of product can be input to this new learning model, and in addition to outputting product identification information during product recognition processing, it can also additionally output whether or not a POP is attached to the displayed product in the image information.

[0067] In the latter method, after performing product recognition processing on image information of the area of ​​an object detected as a commodity, standard image information (e.g., specimen information) of the recognized commodity without a POP is compared with the image information of the area of ​​the object, and the similarity is compared. If the similarity is equal to or greater than a predetermined value, it is determined that the object detected as a commodity does not have a POP, and if the similarity is less than the predetermined value, it is determined that the object detected as a commodity has a POP or a candidate for a POP.

[0068] In the above-mentioned method, by comparing image information of an object area detected as a commodity with standard image information of a commodity without a POP recognized by the commodity recognition process, it is possible to determine whether or not it is a POP based on the location of the different area, which can result in more accurate determination. For example, if it is determined that the different area between the two pieces of image information is localized in a specific position, for example, near the top, shoulder, or tail of the commodity, it can be determined that it is a POP or a candidate for a POP.

[0069] As the commodity recognition process, various known processing methods can be used. For example, a process of identifying commodity identification information can be executed for image information of the object region using deep learning. In this case, the image information of the object region can be input to a learning model in which the weighting coefficient between neurons in each layer of a neural network consisting of many intermediate layers is optimized, and the commodity identification information can be identified based on the output value. As the learning model, a learning model in which commodity identification information is given as correct answer data to various image information in which the commodity is shown can be used. In addition, the commodity sample information (image information of the commodity shown from one or more directions or information such as a feature based thereon) can be compared with the image information of the object region, and the commodity identification information can be identified using the similarity. The commodity recognition process is not limited to these.

[0070] Furthermore, for objects detected as being of a product tag type, it is determined whether the image information of the area of ​​the object satisfies the requirements of a product tag type. For example, OCR recognition processing is performed on the area of ​​an object detected as being of a product tag type, and if it contains characters that are typically included in product tags, such as a price or product name, it is determined to be a product tag, and if not, it is determined to be a POP or a POP candidate. Alternatively, if the result of the OCR recognition processing contains a predetermined word, such as a word typically used in POP, such as "member discount," "tester," "sample," or "sample," it may be determined to be a POP or a POP candidate. Also, if handwritten characters are included, it may be determined to be a POP or a POP candidate.

[0071] For objects detected as being classified as shelf components, the system determines whether the image information of the object's area satisfies the requirements for a shelf component. For example, the system performs OCR recognition processing on the area of ​​an object detected as being classified as a shelf component, and if it contains a specific word, such as a member discount or other word that is typically used in POPs, it is determined to be a POP or a POP candidate, and if not, it is determined to be a shelf component.

[0072] When the individual discrimination processing unit 204 discriminates that the type (class) of the image information of the object region is a candidate for a predetermined object, the image information of the object region may be displayed on the display device 72 of the management terminal 2 or the like, and an input as to whether or not it is an object may be accepted, and if the input is an object, the type (class) of the image information of the object region may be set to the object. For example, when the individual discrimination processing unit 204 discriminates that the type (class) of the image information of the object region is a candidate for a POP, the image information of the object region may be displayed on the display device 72 of the management terminal 2 or the like, and an input as to whether or not it is a POP may be accepted, and if the input is a POP, the type (class) of the image information of the object region may be set to the POP.

[0073] The model processing unit 21 has a classification processing unit 210, a tentative identification information processing unit 211, a model generation processing unit 212, a specimen information reception processing unit 213, a recognition processing unit 214, and an output processing unit 215. An example of a conceptual diagram of the processing of the model processing unit 21 is shown in Fig. 7 and Fig. 8.

[0074] The classification processing unit 210 classifies the object image information from the object image information of the object identified by the detection processing unit 20 by clustering processing or the like. For example, image information (POP image information) of the area of ​​an object determined as a POP that is captured in image information of a display shelf is classified by clustering processing or the like. When the object is a product displayed on a display shelf, a product tag (price card), a shelf, or a POP that is attached, mounted, or installed above or to the side of a shelf, the object image information is the POP image information. In this case, the image information may be cut out in a rectangle including the POP area, or may be cut out in an arbitrary shape according to the outer shape of the POP. Cutting out the image information may be done physically, or the coordinate information of the area may be specified to specify the range.

[0075] Identification information (object identification information) of an object shown in the object image information, for example, if the object is a POP, a code such as information identifying the POP (POP identification information) does not need to be known.

[0076] The classification processing unit 210 performs clustering processing on the plurality of pieces of object image information by a known method, and classifies the image information into a plurality of groups. For example, if the object is a POP, the classification processing unit 210 performs clustering processing on the plurality of pieces of POP image information by a known method, and classifies the POP image information into a plurality of groups.

[0077] Although the classification processing unit 210 classifies the object image information using clustering processing in the above description, the object image information may be classified using a method other than clustering processing.

[0078] The temporary identification information processing unit 211 associates temporary identification information (temporary identification information) with each group of the object image information classified by the classification processing unit 210. When the object is a POP, temporary identification information (temporary identification information) is associated with each group of the POP image information classified by the classification processing unit 210. The temporary identification information may be automatically generated temporary identification information, or input temporary identification information may be associated by a predetermined operation. The temporary identification information associated with the object image information classified into each group is then associated as a label to form a data set, which is used as learning data.

[0079] The model generation processing unit 212 performs machine learning using a known method using the learning data to generate a learning model. That is, since temporary identification information is associated with each classified group of the object image information, a learning model is generated using learning data that uses a data set in which the object image information and its label are used as temporary identification information. In the case where the object is a POP, temporary identification information is associated with each group of the classified POP image information, so a learning model is generated using learning data that uses a data set in which the POP image information and its label are used as temporary identification information.

[0080] The specimen information reception processing unit 213 receives input of image information to be input to the model generation processing unit 212. The image information received as input here is object image information for which object identification information for identifying the object is known. For example, it is POP image information that may be attached to a display shelf, and the corresponding POP identification information (POP identification information) is known. The image information input here is referred to as specimen information.

[0081] The recognition processing unit 214 inputs the image information (sample information) received by the sample information receiving processing unit 213 to the learning model generated by the model generation processing unit 212, and outputs tentative identification information. In this case, when the sample information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, tentative identification information is output as an output value. When multiple output values ​​are output together with their probabilities, the output value with the highest probabilities is adopted.

[0082] The output processing unit 215 links the tentative identification information, which is the output value output by the recognition processing unit 214, with the object identification information of the sample information used for its input. Then, the output processing unit 215 outputs the object identification information linked to the tentative identification information output by the recognition processing unit 214 as the recognition result of the recognition processing unit 214.

[0083] In addition, instead of outputting object identification information linked to the provisional identification information output by the recognition processing unit 214, the output processing unit 215 may replace the provisional identification information associated with the group output by the classification processing unit 210 with the object identification information and cause the model generation processing unit 212 to execute the machine learning learning process again.

[0084] Furthermore, if the accuracy of the provisional identification information output by the recognition processing unit 214 is equal to or lower than a predetermined threshold, the processing of the output processing unit 215 does not need to be executed.

[0085] By performing the above-described processing, a learning model for use in machine learning can be efficiently generated. EXAMPLES

[0086] Next, an example of a processing process using the information processing system 1 of the present invention will be described with reference to the flowcharts of Figures 9 to 11. In the following explanation, a case will be described in which captured image information of a store's display shelves is input from the image information input terminal 3, a POP area is detected as an object from the captured image information, and a machine learning learning model for identifying POPs is generated using image information of the detected POP area.

[0087] First, the detection process in the detection processing unit 20 will be described (S100).

[0088] Captured image information of store display shelves is input from the image information input terminal 3, and the input is accepted by the image information input acceptance processing unit 200 of the management terminal 2 (S110). An example of the input captured image information is shown in FIG. 5(a). Also accepted are input of shooting date and time, store identification information, display shelf identification information, and image information identification information of the captured image information. Then, the image information input acceptance processing unit 200 stores the accepted input captured image information, shooting date and time, store identification information, store identification information, and image information identification information of the captured image information in the image information storage unit 201 in association with each other.

[0089] When a specified operation input is received in the management terminal 2, the image information alignment processing unit 202 extracts the captured image information to be stored in the image information storage unit 201, receives input of four points for performing alignment processing such as keystone correction processing, for example the four corners of a display shelf, and executes the alignment processing (S120).

[0090] Then, by receiving a predetermined operation input in the management terminal 2, the object detection processing unit 203 executes an object detection process on the upright image information that has been uprighted (S130), and classifies the type of object appearing in the upright image information (S140).

[0091] FIG. 12 shows a schematic example of a result of executing object detection processing from upright image information. In FIG. 12, "a" indicates merchandise, "b" indicates merchandise tags, "c" indicates shelf components, and "d" indicates the type of object detected other than "a" to "c". The type of object detected from the image information only needs to be associated with each region, and does not need to be written on each image. The region of the detected object may be cut out from the upright image information, but does not need to be actually cut out as long as the image information of the region is in a processable state. Cutting out image information of a region includes not only actually cutting out image information of the region from the upright image information, but also specifying the region by coordinate information or the like as a processable state without cutting out the image information of the region.

[0092] The individual discrimination processing unit 204 discriminates and cuts out the areas of objects detected by the object detection processing unit 203 that are of a type other than products, product tags, and shelf components (areas of objects "d") as POP areas (S150, S160).

[0093] In addition, the individual discrimination processing unit 204 determines whether an area detected as a type of commodity among the areas of objects detected by the object detection processing unit 203 (S150) satisfies a predetermined condition, and determines whether the area is a POP area (S170).

[0094] For example, for the area of ​​an object detected as a type of commodity, it is determined whether or not the area has a product-specific advertisement (POP) attached, and if it is determined that a POP is present, the type of the area is determined as a POP. For the area of ​​an object detected as a type of commodity, a process is executed to detect POPs attached to displayed products, or a POP is detected by comparing image information of the area of ​​an object detected as a type of commodity with image information of products without POPs (product sample image information).

[0095] Similarly, for an object region detected as a product tag, the system determines whether the image information of the object region satisfies the requirements for a product tag, and for an object detected as a shelf member, the system determines whether the image information of the object region satisfies the requirements for a shelf member.

[0096] As described above, for the area of ​​an object detected as a type of merchandise, product tags, or shelf components (S150), it is determined whether the area satisfies a predetermined condition for each object (S170), and if the condition is satisfied, all or part of the area is determined to be a POP (S180). An example of this process is shown in Fig. 13. Also, Fig. 14 shows an example of each area after areas of an object detected as a type of merchandise, product tags, or shelf components that satisfy a predetermined condition are determined to be POP areas based on the object detection results in Fig. 12.

[0097] POP areas in image information can be detected by executing the above-mentioned processing in the detection processing unit 20. This makes it possible to detect image information of various types of POP areas from captured image information of various display shelves, and to obtain image information used to generate a learning model in the model processing unit 21, which will be described later.

[0098] Next, a process of generating a learning model in the model processing unit 21 using image information of the POP detected by the detection processing unit 20 will be described (S200).

[0099] 15 shows an example of POP image information detected by the detection processing unit 20. At this point, since only the POP image information, which is image information of the object, has been detected from the captured image information, the POP identification information (object identification information) for identifying the POP, which is the object, does not need to be known.

[0100] Then, the classification processing unit 210 performs a known clustering process or the like using the POP image information, which is the object image information, to group similar image information together and classify the information into a plurality of groups (S210). This state is shown diagrammatically in FIG.

[0101] The temporary identification information processing unit 211 associates temporary identification information with each group classified by the classification processing unit 210 (S220). For example, any temporary identification information such as "a123", "b456", "c789", "d012", and "e345" is associated with each group. This state is shown in FIG. 17. In this way, the temporary identification information is associated as a label with the object image information of each group. Then, a data set using the POP image information that is the object image information and the labels of the temporary identification information is used as learning data.

[0102] When the provisional identification information is associated with each group classified by the classification processing unit 210, the model generation processing unit 212 performs machine learning using a known method as learning data (S230) to generate a learning model (S240). This is shown diagrammatically in Fig. 18. Note that Fig. 18 shows a case where one POP image information and provisional identification information for each group are used as the learning data set, but in reality, it is preferable to perform machine learning by associating each POP image information and provisional identification information in each group as the learning data set. This is because, even within the same group, there is not only image information that matches perfectly, but also image information that differs to some extent, and by associating these with the provisional identification information for that group and performing machine learning, there is a possibility that accuracy can be improved.

[0103] Next, the specimen information reception processing unit 213 receives input of specimen information, which is image information for which POP identification information serving as object identification information has been identified, and inputs the received input of the specimen information to the learning model generated in S240 (S250). The recognition processing unit 214 receives input of the specimen information, and outputs an output result based on machine learning using the learning model. As the output result, tentative identification information is output (S260). This is shown diagrammatically in FIG. 19.

[0104] The object identification information of the sample information, such as POP identification information, is known in advance. Therefore, the tentative identification information output by the recognition processing unit 214 is considered to correspond to the object identification information of the sample information input to the learning model. Therefore, the output processing unit 215 links the output tentative identification information with the object identification information of the input sample information (S270). This is shown diagrammatically in FIG. 20.

[0105] This linking makes it possible to associate the provisional identification information with the POP identification information, which is the object identification information. Therefore, by storing this association, the provisional identification information output by the recognition processing unit 214 can be replaced with the corresponding object identification information and output. In this way, when image information to be subjected to identification processing is input into the learning model, the object identification information can be output as the output value of the learning model.

[0106] By carrying out the above model processing, a learning model for machine learning of the target object can be generated. For example, POPs can be detected from image information obtained by photographing a display shelf, and the detected POPs can be used to generate a learning model for machine learning to detect POPs from input image information. EXAMPLES

[0107] Another embodiment of the detection processing in the detection processing unit 20 will be described. The detection processing unit 20 of this embodiment further includes an area detection processing unit 205. An example of the configuration of the detection processing unit 20 in this embodiment is shown in Fig. 21, and an example of the processing process of the detection processing unit 20 is shown in the flowchart of Fig. 22. Also, an example of the outline of the processing is shown in Fig. 23, and an example of the individual discrimination processing is shown in Fig. 24.

[0108] The area detection processing unit 205 detects the product display area where the products are displayed, the product tag area where the product tags are placed, and the upper shelf area which is the area above the shelves, from the image information (including the normalized image information) stored in the image information storage unit 201. The product display area, product tag area, and upper shelf area may be detected by the operator of the management terminal 2 manually specifying the product display area, product tag area, and upper shelf area, which are then accepted by the area detection processing unit 205, or the product display area, product tag area, and upper shelf area may be automatically detected from the second time onwards, based on the information on the product display area, product tag area, and upper shelf area that was manually accepted the first time. The area detection processing unit 205 may also detect areas other than these and use them as processing targets.

[0109] FIG. 25 shows a schematic diagram of the state in which input of designation of a product display area, a product tag area, and a shelf top area has been received for image information that has been normalized from image information captured of a display shelf on which products are displayed.

[0110] When detecting the product display area, product tag area, and shelf top area, the area detection processing unit 205 may use deep learning to identify the product display area, product tag area, and shelf top area. In this case, the above-mentioned orthogonalized image information may be input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the product display area, product tag area, and shelf top area may be detected based on the output value. In addition, as the learning model, various image information may be given the product display area, product tag area, and shelf top area as correct answer data.

[0111] The area detection processing unit 205 may detect areas using various known methods in addition to using deep learning as described above.

[0112] The area detection processing unit 205 may actually extract the detected area as image information, or may virtually extract the area by specifying the range of the area without actually extracting the area as image information. In this case, the range of the area can be specified by coordinates.

[0113] The object detection processing unit 203 in this embodiment may perform object detection processing from the range of the upright image information, like the detection processing unit 20 in the first embodiment, or may perform object detection processing for each area detected by the area detection processing unit 205.

[0114] The individual discrimination processing unit 204 in this embodiment discriminates the type of each object detected by the object detection processing unit 203 using the type (class) of the object and the area in which the object is located. The discrimination is performed according to predetermined individual discrimination conditions, such as the discrimination tables shown in Figs. 23 and 24.

[0115] "a1" to "a4", "b1" to "b10", and "d1" to "d5" in FIG. 23 correspond to "a1" to "a4", "b1" to "b10", and "d1" to "d5" in FIG.

[0116] Next, an example of the processing process of the detection processing unit 20 in this embodiment will be described with reference to the flowchart of FIG.

[0117] The captured image information of a store's display shelves is input from the image information input terminal 3, and the input is accepted by the image information input acceptance processing unit 200 of the management terminal 2 (S300). The image information input acceptance processing unit 200 also accepts input of the image capture date and time, store identification information, display shelf identification information, and image information identification information of the captured image information. The image information input acceptance processing unit 200 then stores the accepted input image information, image capture date and time, store identification information, display shelf identification information, and image information identification information of the captured image information in the image information storage unit 201 in association with each other.

[0118] When a specified operation input is received in the management terminal 2, the image information alignment processing unit 202 extracts the captured image information to be stored in the image information storage unit 201, and receives input of any four vertices that are used for alignment processing such as keystone correction processing, for example the four corners of a display shelf, and executes the alignment processing (S310).

[0119] Then, by receiving a specified operational input in the management terminal 2, the area detection processing unit 205 performs area detection processing on the aligned image information to detect the product display area, product tag area, and shelf top area (S320), and determines the range of each area.

[0120] Furthermore, by receiving a predetermined operation input in the management terminal 2, the object detection processing unit 203 executes an object detection process on the aligned image information (S330) and classifies the type of the object appearing in the image information (S340). An example of the result of executing the object detection process on the image information is classified as shown in FIG. 12, similar to the first embodiment.

[0121] Then, for each object in the image information detected by the object detection processing unit 203, the individual discrimination processing unit 204 classifies the detected object in accordance with predetermined discrimination conditions using the type of the object and the area in which the object is located (the area determined in S320).

[0122] Furthermore, individual discrimination processing section 204 uses the type of object in the image information detected by object detection processing section 203 and the area in which the object is located to determine whether a predetermined condition is satisfied (S350), and determines whether the type of object is a POP or something other than POP (products, product tags, shelf components, etc.) (S360, S370). Note that which area each object is located in may be determined by any appropriate method, for example, by determining whether the center of gravity of the object's area is included in the area of ​​products, product tags, or the top of the shelf.

[0123] For example, when an object is in a product display area and the type of the object is a commodity, the individual discrimination processing unit 204 judges whether or not there is a commodity-adjacent POP in the area of ​​the object. For example, as in the first embodiment, discrimination is performed by a method of detecting a POP attached to a displayed commodity from the area of ​​the object detected as a commodity, and a method of detecting a POP by comparing image information of the area of ​​the object detected as a commodity with image information of a commodity without a POP. If a commodity-adjacent POP is detected, the area is discriminated as a POP (a2), and other areas are discriminated as a commodity (a1). If a commodity-adjacent POP is not detected, an OCR recognition process is performed on the area to discriminate whether a specific word used in POP such as "tester", "sample", or "sample" is included, and if not, the object in the area is discriminated as a commodity (a1). If it is included, the object in the area is discriminated as a POP such as a tester or sample (a3).

[0124] Furthermore, when an object in the product display area is classified as a product tag, the individual discrimination processing unit 204 determines that the object in that area is a POP (d1).

[0125] Also, when an object in a product display area is classified as a shelf member, the individual discrimination processing unit 204 discriminates that the object in that area is a shelf member (c).

[0126] Moreover, if an object is in the product display area and the type of the object is other than products, product tags, or shelf members, it is determined to be a POP (d2).

[0127] When the object is in the product tag area and the type of the object is a product, the individual discrimination processing unit 204 discriminates that the object is a POP such as a tester or a sample (d4).

[0128] If the object is in the product tag area and the type of the object is a product tag, OCR recognition processing is performed on the object area, and if it is determined that there is handwriting in the area, it is determined to be a POP (retail POP) created by a store (retailer) (b4). If it is determined that there is no handwriting in the area as a result of OCR recognition processing on the object area, and if it is determined that the area contains a price and product name display, and if it contains a word indicating a condition such as "member", the type of the object is determined to be a product tag indicating a set product display (b2). On the other hand, if it is determined that the price and product name display is included, but does not contain a word indicating a condition such as "member", the type of the object is determined to be a product tag (b1). If it is determined that the price and product name display are not included, and if there is a point display, the type of the object is determined to be a POP with a "point display" (b3). Furthermore, if there is no point display in the above, it is determined to be a POP created by the product manufacturer (manufacturer POP) (b9). Note that here, a case has been shown in which manufacturer POP and retail POP are distinguished from each other and are determined to be the same, but it may simply be determined as a POP.

[0129] Furthermore, when an object is in the product tag area and the type of the object is a shelf member, the individual discrimination processing unit 204 discriminates that the object in that area is a shelf member (c).

[0130] Furthermore, if an object is in the product tag area and the type of the object is other than products, product tags, or shelf components, OCR recognition processing is performed on that area. Then, it is determined whether or not the object contains specific words used in POP such as "tester," "sample," or "sample," and if so, it is determined to be a POP such as a tester or sample (d4). On the other hand, if the object does not contain specific words used in POP such as "tester," "sample," or "sample," it is determined to be a POP such as a promotional item (d3).

[0131] When the object is in the upper shelf area and the type of the object is a commodity, the individual discrimination processing unit 204 discriminates that the object is a stock commodity (a4).

[0132] If an object is located in the upper shelf area and the type of the object is a product tag, OCR recognition processing is performed on the area of ​​the object, and if it is determined that there is handwriting in the area, it is determined to be a retail POP (b8). If the result of OCR recognition processing on the area of ​​the object is that it is determined that there is no handwriting in the area, and if it is determined that the area contains a display of the price and product name, and if it contains a word indicating a condition such as "member", the type of the object is determined to be a product tag indicating a set product display (b6). On the other hand, if it is determined that the area contains a display of the price and product name, but does not contain a word indicating a condition such as "member", the type of the object is determined to be a product tag (b5). If it is determined that the above-mentioned price and product name display is not included, and if there is a display of points, the type of the object is determined to be a POP with a "point display" (b7). Furthermore, if there is no display of points in the above, it is determined to be a manufacturer POP (b10).

[0133] Furthermore, when an object is in the shelf upper region and the type of the object is a shelf member, the individual discrimination processing unit 204 discriminates that the object in that region is a shelf member (c).

[0134] Furthermore, if an object is in the upper shelf area and the type of the object is other than merchandise, merchandise tags, or shelf components, it is determined to be a POP such as a large promotional item (d5).

[0135] As described above, the individual discrimination processing unit 204 can use the area detected by the area detection processing unit 205 and the type of object detected by the object detection processing unit 203 to determine whether the object is a POP or not, and detect the area of ​​the POP in the image information. EXAMPLES

[0136] As a modified example of the processing of the first and second embodiments, the processing shown in the flowchart of FIG. 26 may be performed. In this case, if the type of the object detected by the object detection processing unit 203 is other than merchandise, merchandise tags, or shelf members, the individual discrimination processing unit 204 determines that the object is a POP (S460). Then, for other objects, the same processing as the processing of S350 to S370 of FIG. 22 may be performed. That is, using the type of the object in the image information detected by the object detection processing unit 203 and the area in which the object is located, it is determined whether a predetermined condition is satisfied (S470), and it is determined whether the object is a POP or something other than a POP (such as merchandise, merchandise tags, or shelf members) (S480, S490). Note that the processing from S400 to S440 may be the same as the processing from S300 to S340. EXAMPLES

[0137] In addition to the processing of the detection processing unit 20 in the first to third embodiments, modified examples of the model processing unit 21 will be described. The classification processing performed by the classification processing unit 210 may be less accurate than classification processing performed by a human being. Therefore, errors may be mixed in. Therefore, a cleansing processing unit 216 may be provided that performs a cleansing processing on the image information (target image information) of each group classified by the classification processing unit 210, removes target image information that may contain errors from the image information classified into the groups, and reduces the target image information used as learning data for the model generation processing unit 212. An example of the configuration of the model processing unit 21 in this case is shown in FIG. 27.

[0138] That is, when the classification processing unit 210 classifies the object image information into groups, index values ​​such as feature amounts showing the characteristics of the image information in the group are calculated. Then, in the group, the information distance from a reference value such as the median or average value of the index values ​​for each object image information is calculated (a value calculated using a predetermined formula as the deviation from the reference value to the index value for each image information), and object image information whose information distance deviates by a certain value or a certain ratio or more may be excluded from the group.

[0139] In addition to using the information distance as the cleansing process, the similarity of the image information may be used. In this case, the similarity between the image information in the group is quantified, and if the number of the image information with a certain degree of similarity is equal to or greater than a predetermined value, the image information is retained as the object image information, and if the number is less than the predetermined value, the image information is excluded from the object image information.

[0140] For example, if a group contains five pieces of target image information (image information 1 to image information 5), the similarity between each of the five pieces of image information and the other pieces of image information is calculated. That is, the similarity between image information 1 and image information 5 is calculated, between image information 2 and image information 5, between image information 2 and image information 5, between image information 3 and image information 5, between image information 3 and image information 5, and between image information 4 and image information 5. This allows the similarity between each piece of image information to be calculated.

[0141] For image information 1, the similarity with image information 2 is 0.9, the similarity with image information 3 is 0.95, the similarity with image information 4 is 0.3, and the similarity with image information 5 is 0.8; for image information 2, the similarity with image information 3 is 0.8, the similarity with image information 4 is 0.2, and the similarity with image information 5 is 0.6; for image information 3, the similarity with image information 4 is 0.4 and the similarity with image information 5 is 0.9; and for image information 4, the similarity with image information 5 is 0.5.

[0142] Here, if the similarity of the reference image information is 0.75 and the number of pieces of image information below the reference level is three, then for image information 1, one piece of image information 4 is below the reference similarity, for image information 2, two pieces of image information 4 and 5 are below the reference similarity, for image information 3, one piece of image information 4 is below the reference similarity, for image information 4, four pieces of image information 1, 2, 3, and 5 are below the reference similarity, and for image information 5, two pieces of image information 2 and 4 are below the reference similarity.

[0143] Therefore, the cleansing processing unit 216 excludes image information 4, which had three or more pieces of image information below the standard similarity, from the group, and executes processing by the provisional identification information processing unit 211 with four pieces of object image information for the group: image information 1, image information 2, image information 3, and image information 5.

[0144] By excluding the deviated image information from the grouped object image information as in this embodiment, the grouped object image information can be narrowed down, leading to improved accuracy. EXAMPLES

[0145] When extracting POP image information as object image information by cutting out a POP installed on a display shelf, etc., multiple similar POPs may be installed adjacent to each other in either the up, down, left, or right direction in order to attract customer attention.

[0146] Therefore, after extracting POP image information as object image information from the image information photographed of the display shelves, the classification processing unit 210 compares the similarity of the POP image information as adjacent object image information, and if it satisfies certain conditions, for example if the similarity is equal to or greater than a predetermined threshold, it may determine that they are the same type of object, and classify the POP image information as object image information of the adjacent POPs into the same group. EXAMPLES

[0147] When the target object is the POP of a product displayed on a shelf, it is preferable to generate a learning model of the POP that may be displayed in advance. However, in that case, the target POP may be hundreds to thousands, or in some cases tens of thousands. In this case, if a learning model capable of recognizing a learning model of thousands to tens of thousands is used, the sample information input by the sample information reception processing unit 213 will be in the thousands to tens of thousands, and although the amount of work is reduced compared to conventional learning models, the workload is still large.

[0148] On the other hand, it is known in marketing that generally about 10% of product types tend to account for about 90% of the sales of their classification. Therefore, POP of products with high sales may be used as the specimen information to be input to the specimen information reception processing unit 213. In this case, POP of products provided by a marketing company or the like may be used.

[0149] This makes it possible to recognize the POPs of about 90% of products based on sales, which is sufficient for practical purposes as output of object identification information. When the POP of a product with low sales is input to the learning model, the object identification information is not linked to the provisional identification information, so the output processing unit 215 outputs the provisional identification information as is. When the output processing unit 215 outputs provisional identification information that does not have a correspondence between the provisional identification information and the object identification information, a predetermined display such as "no correspondence information" may be displayed.

[0150] Furthermore, if it is sufficient to recognize only the POP of a product of a particular company or other organization, image information of the product of that company may be input as sample information to the sample information reception processing unit 213. This allows object identification information to be output for the POP of a product of a certain company. On the other hand, when a POP of a product of a company other than the certain company is input, as in the above case, the object identification information is not linked to the provisional identification information, so the output processing unit 215 outputs the provisional identification information as is. When the output processing unit 215 outputs provisional identification information that does not correspond to the provisional identification information and the object identification information, a predetermined display such as "no correspondence information" may be displayed. EXAMPLES

[0151] For the learning model generated as in the first to sixth embodiments, the learning model may be reconstructed again for the purpose of improving accuracy. In this case, new sample information may be inputted, and object identification information may be newly linked to tentative identification information to which no object identification information is linked, or the classification process may be performed in the classification processing unit 210 using object image information of a new object so that the new object can be recognized. EXAMPLES

[0152] In the above-mentioned first to seventh embodiments, a learning model is generated for identifying POPs displayed on a display shelf from image information obtained by photographing the display shelf, but the present invention can also be applied to other cases. In particular, the present invention is useful for automating the identification of many types of objects from image information.

[0153] As an example, the target may be an animal. For example, the method may be applied to a case where a large number of animals such as sea lions and fur seals live in a place where people cannot approach, and the type and number of animals living there are to be efficiently grasped. In this case, the habitat is photographed from the sky using a drone or the like, object image information is cut out for each individual, and the object image information for each individual is subjected to image classification processing by the classification processing unit 210 and grouped. Then, the model generation processing unit 212 performs machine learning using learning data generated by the provisional identification information processing unit 211 by associating provisional identification information with each group, and generates a learning model. Then, the specimen information reception processing unit 213 receives input of specimen information by type, and the recognition processing unit 214 inputs the received specimen information as an input value into the learning model, and outputs provisional identification information. This allows the output processing unit 215 to output the object identification information by linking the provisional identification information with the object identification information (such as the scientific name of the animal) corresponding to the input specimen information.

[0154] As in the above, the target object may be a bird or a plant. Even in the case of a bird or a plant, the processing can be performed in the same manner as for the above-mentioned animal, and the processing can be performed by replacing "animal" with "bird" or "plant". [Industrial Applicability]

[0155] By using the information processing system 1 of the present invention, it is possible to efficiently generate a learning model used in machine learning. [Explanation of symbols]

[0156] 1: Information processing system 2: Management terminal 3: Image information input terminal 20: Detection processing unit 21: Model processing section 70: Arithmetic device 71:Storage device 72:Display device 73: Input device 74: Communication equipment 200: Image information input reception processing unit 201: Image information storage unit 202: Image information alignment processing unit 203: Object detection processing unit 204: Individual discrimination processing unit 205: Area detection processing unit 210: Classification processing unit 211: Provisional identification information processing unit 212: Model generation processing unit 213: Specimen information reception processing unit 214: Recognition processing unit 215: Output processing unit 216: Cleansing processing section

Claims

1. An information processing system for performing processing related to a learning model used in machine learning, A detection processing unit that detects a predetermined object from image information; A model processing unit that generates a learning model for machine learning using the detected object, The model processing unit: A classification processing unit that classifies image information of the detected object; a temporary identification information processing unit that associates the classified groups with the temporary identification information; a model generation processing unit that executes a machine learning learning process using learning data including image information of the objects included in the classified groups and the provisional identification information to generate a learning model; An information processing system comprising:

2. The model processing unit: a recognition processing unit that outputs provisional identification information by inputting the received sample information to the learning model; an output processing unit that associates the output tentative identification information with object identification information of the input specimen information; 2. The information processing system according to claim 1, further comprising:

3. The model processing unit: inputting image information to be subjected to the identification process into the learning model, thereby outputting, as an output value, object identification information corresponding to provisional identification information output by the learning model; 3. The information processing system according to claim 2.

4. The model processing unit: a cleansing processing unit that executes a cleansing process to remove object image information that may contain errors from the image information of the objects included in the classified groups; 2. The information processing system according to claim 1, further comprising:

5. The cleansing processing unit includes: determining image information of an object to be excluded using an information distance from a reference value calculated using an index value of image information of the object included in the classified group; 5. The information processing system according to claim 4.

6. The cleansing processing unit includes: Calculating a reference value using index values ​​of image information of objects included in the classified groups; Calculating an information distance using the reference value and an index value of the image information of the object; If the information distance is equal to or greater than a predetermined threshold value or equal to or greater than a predetermined ratio, the image information of the object is determined to be image information of an object to be excluded.

6. The information processing system according to claim 5.

7. The cleansing processing unit includes: determining image information of an object to be excluded using a similarity of image information of the object included in the classified group; 5. The information processing system according to claim 4.

8. The cleansing processing unit includes: Calculating a similarity between image information of an object included in the classified group and image information of other objects included in the group; When a predetermined condition using the similarity is satisfied, the image information of the object is determined as image information of an object to be excluded.

8. The information processing system according to claim 7.

9. The detection processing unit is an object detection processing unit that detects an object appearing in the image information; an individual discrimination processing unit that discriminates a type of the detected object; 2. The information processing system according to claim 1, further comprising:

10. The information processing system includes: a region detection processing unit for detecting a region of the image information, The individual discrimination processing unit is determining a type of the detected object using the detected area and the type of the detected object; 10. The information processing system according to claim 9.

11. An information processing system for performing processing related to a learning model used in machine learning, an object detection processing unit that detects an object appearing in the image information; an individual discrimination processing unit that discriminates a type or a candidate type of the detected object; A classification processing unit that classifies image information of the object type for which the learning model is to be generated into groups, among the object types; a temporary identification information processing unit that associates the classified groups with temporary identification information; a model generation processing unit that executes a machine learning learning process using learning data including image information of the objects included in the group and the provisional identification information to generate the learning model; An information processing system comprising:

12. The information processing system includes: a recognition processing unit that outputs provisional identification information by inputting the received sample information to the learning model; an output processing unit that associates the output tentative identification information with object identification information of the input specimen information; 12. The information processing system according to claim 11, further comprising:

13. An information processing system for performing processing related to a learning model used in machine learning, A detection processing unit that detects a POP from image information; A model processing unit that generates a learning model for machine learning using the detected POPs; An information processing system comprising:

14. The model processing unit: a classification processing unit that classifies image information of the detected POP; a temporary identification information processing unit that associates the classified groups with temporary identification information; a model generation processing unit that executes a machine learning learning process using learning data including image information of the POPs included in the group and the provisional identification information to generate a learning model; 14. The information processing system according to claim 13, further comprising:

15. The model processing unit: a cleansing processing unit that executes a cleansing process to remove image information of POPs that may contain errors from the image information of POPs included in the classified groups; 15. The information processing system according to claim 14, further comprising:

16. An information processing system for performing processing related to a learning model used in machine learning, an object detection processing unit that detects POPs appearing in the image information; an individual discrimination processing unit that discriminates a type or a candidate type of the detected POP; A classification processing unit that classifies image information of a POP type for which the learning model is generated, among the POP types, into groups; a temporary identification information processing unit that associates the classified groups with the temporary identification information; a model generation processing unit that executes a machine learning learning process using learning data including image information of the POPs included in the classified groups and the provisional identification information to generate the learning model; An information processing system comprising:

17. Computer, a detection processing unit that detects a predetermined object from the image information; An information processing program that functions as a model processing unit that generates a learning model for machine learning using the detected object, The model processing unit: A classification processing unit that classifies image information of the detected object; a temporary identification information processing unit that associates the classified groups with temporary identification information; a model generation processing unit that executes a machine learning learning process using learning data including image information of the objects included in the group and the provisional identification information to generate a learning model; An information processing program comprising:

18. Computer, an object detection processing unit that detects an object appearing in the image information; an individual discrimination processing unit that discriminates a type or a candidate type of the detected object; a classification processing unit that classifies, into groups, image information of a target object type for generating a learning model for machine learning among the object types; a temporary identification information processing unit that associates the classified groups with temporary identification information; a model generation processing unit that executes a machine learning learning process using learning data including image information of the objects included in the group and the provisional identification information to generate the learning model; An information processing program that functions as a

19. Computer, a detection processing unit that detects a POP from image information; a model processing unit that generates a learning model for machine learning using the detected POPs; An information processing program that functions as a

20. Computer, an object detection processing unit that detects POPs appearing in the image information; an individual discrimination processing unit that discriminates a type or a candidate type of the detected POP; a classification processing unit that classifies image information of a POP type for generating a learning model for machine learning among the POP types into groups; a temporary identification information processing unit that associates the classified groups with temporary identification information; a model generation processing unit that executes a machine learning learning process using learning data including image information of the POPs included in the group and the provisional identification information to generate the learning model; An information processing program that functions as a

Images