Information processing system

Abstract

To provide an information processing system related to the generation of a learning model in machine learning.SOLUTION: An information processing system for performing processing related to a learning model used in machine learning has: a detection processing unit which detects a POP from image information; and a model processing unit which generates a learning model for machine learning by using the detected POP. The model processing unit has: a classification processing unit which classifies image information of the detected POP; a temporary identification information processing unit which associates a classified group with temporary identification information; and a model generation processing unit which performs learning processing in machine learning to generate a learning model by using data for learning containing the image information of the POP included in the group and the temporary identification information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system used for generating a learning model (network) in machine learning.

Background Art

[0002] In stores such as retail stores, POP for promoting the sale of products may be installed on the display shelves for displaying products. The POP is attached, mounted, installed, etc. on the products displayed on the display shelves, product tags (price cards), shelf tiers, above or on the side of the shelf tiers.

[0003] There are manufacturer POPs created by manufacturers that produce products and request the installation in retail stores, and retail POPs (distribution POPs) created and installed by retail stores themselves. Since the manufacturer requests the installation of the manufacturer POP in the retail store, the manufacturer wants to confirm whether it is installed as requested.

[0004] In such a case, conventionally, the person in charge of the manufacturer actually visited each store for confirmation, but the burden was large. Therefore, automatic processing can be considered.

[0005] As automatic processing, for example, there is machine learning as a method of image analysis processing using a computer. In machine learning, it is necessary to create a learning dataset (learning data) in advance and let the computer read it and execute a predetermined learning process.

[0006] However, preparing the learning data in advance itself has a large workload. Therefore, systems as disclosed in Patent Document 1 and Patent Document 2 below are disclosed.

[0007] In the invention of Patent Document 1, a product image is taken, the code attached to the product is read, and the product image and its labeling are performed to generate learning data.

[0008] Furthermore, in the invention described in Patent Document 2, the classification of an object included in the image data is selected from an image database that stores image data containing objects that have been pre-labeled with classification information, and the object is extracted from the image data to generate a template image, thereby generating training data consisting of the template image and labels. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Japanese Patent Publication No. 2020-095537 [Patent Document 2] Japanese Patent Publication No. 2022-076296 [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] In the invention described in Patent Document 1, training data is generated by preparing a product, photographing it, and reading the code attached to the product with a code reader, which places a heavy burden on the worker. If there are many products to be read for training data, the amount of work becomes enormous. Furthermore, while products have codes attached, POP displays do not, so the invention described in Patent Document 1 cannot be used to generate training data for POP displays.

[0011] In the case of the invention described in Patent Document 2, the processing cannot be performed unless an image database is prepared containing image data of objects that have been pre-labeled with classification information. POPs are diverse, and it is not practical to prepare an image database with labeled image data for each POP. [Means for solving the problem]

[0012] In view of the above problems, the inventors have invented an information processing system for efficiently generating learning models used in machine learning. In particular, the information processing system of the present invention is highly effective when used to generate learning models for point-of-purchase (POP) displays installed on store shelves.

[0013] The first invention is an information processing system that performs processing related to a learning model used in machine learning, and comprises a detection processing unit that detects POPs from image information, and a model processing unit that generates a learning model for machine learning using the detected POPs, wherein the model processing unit comprises a classification processing unit that classifies the image information of the detected POPs, a provisional identification information processing unit that associates the classified groups with provisional identification information, and a model generation processing unit that performs machine learning processing to generate a learning model using training data that includes the image information of the POPs included in the groups and the provisional identification information.

[0014] By using this invention, it is possible to efficiently generate learning models used in machine learning. In particular, since image information used as training data and its label (type) can be detected from the image information and used as a dataset, the workload can be reduced.

[0015] The inventors of this invention focused on the fact that typical display shelves are equipped with product categories such as the products to be displayed, product tags such as product tags, shelf components, and POP displays. In other words, while the shape and design of the POP displays themselves vary widely and are not easy to distinguish, product categories, product tags, and shelf components are easy to distinguish. Therefore, if an object detected from image data is not one of the categories of product category, product tag, or shelf component, it is determined to be a POP display, thereby enabling the identification of POP displays in image data.

[0016] In the above-described invention, the model processing unit can be configured as an information processing system having a cleansing processing unit that performs a cleansing process to exclude image information of POPs that may contain errors from the image information of POPs included in the classified group.

[0017] By executing the cleansing process as in the present invention, it is possible to exclude object image information that may contain errors from the object image information included in the group, and improve the accuracy of classification.

[0018] The first invention can be realized by causing a computer to read and execute the information processing program of the present invention. That is, an information processing program that causes a computer to function as a detection processing unit that detects POPs from image information, and a model processing unit that generates a learning model for machine learning using the detected POPs, wherein the model processing unit includes a classification processing unit that classifies the image information of the detected POPs, a pseudo-identification information processing unit that associates the classified group with pseudo-identification information, and a model generation processing unit that executes learning processing of machine learning using learning data including the image information of the POPs included in the group and the pseudo-identification information to generate a learning model.

Effects of the Invention

[0019] By using the information processing system of the present invention, a learning model used in machine learning can be efficiently generated.

Brief Description of Drawings

[0020] [Figure 1] It is a block diagram schematically showing an example of the processing functions of the information processing system of the present invention. [Figure 2] It is a block diagram schematically showing an example of the processing functions of the detection processing unit of the information processing system of the present invention. [Figure 3] It is a block diagram schematically showing an example of the processing functions of the model processing unit of the information processing system of the present invention. [Figure 4] It is a block diagram schematically showing an example of the hardware configuration of a computer used in the information processing system of the present invention. [Figure 5] It is a diagram showing an example of a conceptual diagram of the detection processing unit of the information processing system of the present invention. [Figure 6] This is a diagram showing an example of a conceptual diagram of the processing of the information processing system of the present invention. [Figure 7] This is a diagram showing an example of a conceptual diagram of the processing of the information processing system of the present invention. [Figure 8] This is a diagram showing an example of photographed image information. [Figure 9] This is a flowchart showing an example of the overall processing process in the information processing system of the present invention. [Figure 10] This is a flowchart showing an example of the processing process of the detection processing unit in the information processing system of the present invention. [Figure 11] This is a flowchart showing an example of the processing process of the model processing unit in the information processing system of the present invention. [Figure 12] This is a diagram showing an example of the result of performing object detection processing from image information. [Figure 13] This is a diagram showing an example of a process of determining whether a predetermined condition is satisfied for the region of an object detected from image information and determining the region of a target object. [Figure 14] This is a diagram showing an example of each region after determining, as the region of a target object, the region that satisfies a predetermined condition with respect to the result of object detection in FIG. 12. [Figure 15] This is a diagram showing an example of a state in which target object image information is extracted from image information. [Figure 16] This is a diagram showing an example of a state in which target object image information is classified in a classification processing unit. [Figure 17] This is a diagram showing an example of a state in which provisional identification information is associated in a provisional identification information processing unit. [Figure 18] This is a diagram showing an example of a state in which a learning model is generated using a learning dataset. [Figure 19] This is a diagram showing an example of a state in which sample information is input to a learning model and provisional identification information is output as an output value. [Figure 20] This is a diagram showing an example of a state in which provisional identification information and target object identification information are associated. [Figure 21]This is a schematic block diagram showing an example of the configuration of the detection processing unit in Example 2. [Figure 22] This flowchart shows an example of the processing process of the detection unit in Example 2. [Figure 23] This figure shows an example of the process outline in Example 2. [Figure 24] This figure shows an example of the individual discrimination process in Example 2. [Figure 25] This diagram schematically shows the state in which input for specifying the product display area, product tag area, and shelf top area has been received for the image information. [Figure 26] This flowchart shows an example of the processing process of the detection unit in Example 3. [Figure 27] This is a schematic block diagram showing an example of the configuration of the model processing unit in Example 4. [Modes for carrying out the invention]

[0021] An example of the processing functions of the information processing system 1 of the present invention is shown in block diagrams from Figures 1 to 3. Figure 1 is a block diagram showing an example of the overall processing functions of the information processing system 1, Figure 2 is a block diagram showing an example of the processing functions of the detection processing unit 20, and Figure 3 is a block diagram showing an example of the processing functions of the model processing unit 21.

[0022] Information processing system 1 is a system that detects objects depicted in predetermined image information and generates a learning model for use in machine learning using the image information of the various detected objects.

[0023] For example, if the target object is a product displayed on a shelf, a product tag (price card), a shelf, or a POP (point-of-purchase) display attached, mounted, or installed on the top or side of a shelf, this system detects the POP from image information of the display shelf including the POP, and uses that POP to generate a learning model for machine learning. While the explanation uses products displayed on a shelf, product tags (price cards), shelves, or POP attached, mounted, or installed on the top or side of a shelf as examples of target objects, the system is not limited to these.

[0024] The management terminal 2 and image information input terminal 3 in the information processing system 1 are implemented using a computer. Figure 4 schematically shows an example of the computer's hardware configuration. The computer has an arithmetic unit 70 such as a CPU that executes program calculations, a storage device 71 such as RAM or a hard disk that stores information, a display device 72 such as a display that shows information, an input device 73 such as a keyboard or mouse that can input information, and a communication device 74 that sends and receives the processing results of the arithmetic unit 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0025] If the computer is equipped with a touch panel display, the display device 72 and the input device 73 may be configured as an integrated unit. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited to these.

[0026] A touch panel display is a device in which the functions of a display device 72 and an input device 73 are integrated, as input can be performed directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

[0027] The image information input terminal 3 may be equipped with a camera or other photographic device in addition to the devices described above. A portable communication terminal such as a mobile phone, smartphone, or tablet computer can also be used as the image information input terminal 3. The image information input terminal 3 takes a picture of a predetermined object (captured image information) and inputs it to the management terminal 2. For example, it takes a picture of a store display shelf and inputs the image information to the management terminal 2. In this case, products are displayed on the display shelf, and POP (point of purchase) displays are attached to the display shelf or products. Therefore, the captured image information includes the displayed products such as the products displayed on the display shelf, product tags, shelf components, POP displays, etc. Preferably, it is preferable that no unwanted objects such as people are captured in the image.

[0028] Each of the means in this invention may be physically or in fact the same domain, even though their functions are logically distinct. The processing order in each of the means in this invention can be changed as appropriate. Furthermore, some of the processing may be omitted.

[0029] The management terminal 2 has a detection processing unit 20 and a model processing unit 21.

[0030] The detection processing unit 20 detects image information of the area containing the object (object image information) from the captured image information. For example, from image information of a display shelf, it detects image information of the area containing the POP (point-of-purchase) display (POP image information) as the object image information. Figure 5 is an example of a conceptual diagram of the processing in the detection processing unit 20. Figure 5(a) is an example of captured image information containing an object, and Figure 5(b) is an example of the state in which the object has been detected from the captured image information. The captured image information in Figure 5(a) schematically shows the case when a display shelf in a store is photographed. Figure 5(b) is a schematic diagram showing the case when a POP is detected as the object. The shaded area in Figure 5(b) is the image information of the detected POP area.

[0031] The model processing unit 21 generates a learning model for machine learning using the object image information detected by the detection processing unit 20. An example of a conceptual diagram of the processing in the model processing unit 21 is shown in Figures 6 and 7. Figure 6 is a schematic diagram showing the process of generating a learning model using the object image information detected by the detection processing unit 20. Figure 7 is a schematic diagram showing the process of linking provisional identification information, which is associated with each group of classified object image information, with the identification information of the object (object identification information).

[0032] The detection processing unit 20 includes an image information input receiving processing unit 200, an image information storage unit 201, an image information orientation processing unit 202, an object detection processing unit 203, and an individual discrimination processing unit 204.

[0033] The image information input receiving processing unit 200 receives input of captured image information containing an object, taken by the image information input terminal 3, and stores it in the image information storage unit 201, which will be described later. For example, it receives input of image information of a store's display shelves and stores it in the image information storage unit 201. In addition to the captured image information, the image information input terminal 3 may also receive input of supplementary information such as the date and time of shooting, store identification information such as the store name, display shelf identification information to identify the display shelves, and image information identification information to identify the image information. Figure 8 shows an example of image information of a display shelf as captured image information. Note that Figure 8 shows a case where one display shelf is captured in the captured image information, but multiple display shelves may be captured, or only a part of a display shelf may be captured. Furthermore, although the processing is not specifically described in this invention, display shelves and shelves may be long in the horizontal direction. For this reason, in the processing, they may be divided into sections of a certain width, and the processing target may be divided into sections.

[0034] The image information storage unit 201 stores the captured image information received from the image information input terminal 3 in association with associated information such as the date and time of shooting, store identification information, display shelf identification information, and image information identification information. The captured image information can be any image information that is subject to processing in the present invention. This also includes image information that is synthesized from multiple images taken of a single display shelf. Image information that has been processed in the image information orientation processing unit 202 described later, and image information after distortion correction processing have been performed are also included in the captured image information.

[0035] The image information orientation processing unit 202 performs an orientation process on the captured image information stored in the image information storage unit 201 to correct the subject to be facing directly towards the camera, thereby generating oriented image information (orientation image information). The orientation process involves transforming the image information so that it is the same as if the optical axis of the lens of the camera device were aligned perpendicular to the plane of the subject being photographed, and the image was taken from a sufficiently far distance. An example of this is trapezoidal correction. Trapezoidal correction is a correction process performed so that objects in the captured image information, such as the shelves of a display shelf, are horizontal, and the products displayed there and their product tags (such as price tags) are vertical.

[0036] The image information orientation processing unit 202 accepts input specifying four vertices in the captured image information and performs trapezoidal correction processing using each of those vertices. The four vertices that can be specified can be any four points in the captured image information; for example, if the subject is a display shelf, they could be the four vertices of the shelf levels, or the four vertices of the shelf positions. Alternatively, they could be the four vertices of a group of two or three shelves. Any four points can be specified as the four vertices. The image information orientation processing unit 202 can use various known processing methods.

[0037] The object detection processing unit 203 performs a process to detect objects in the captured image information (including orthogonalized captured image information; the same applies hereinafter in this specification) stored in the image information storage unit 201. It is preferable to use so-called deep learning for object detection. In this case, the captured image information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many hidden layers are optimized, and the region of the object (or its image information) and the type (class) of that object are output as output values.

[0038] The learning model is trained by using annotation data to record specific objects that may appear in the captured image information, and then performing a predetermined learning process using this annotation data. For example, objects that may appear in an image of a display shelf, such as products, product tags (price tags), and shelf components, are used as annotation data, and the learning model is trained by performing a predetermined learning process using this annotation data. For example, data that associates product image information with the fact that it is a product, or the product name or product code (such as a JAN code), is used as annotation data, and the learning process for the learning model is performed. The product image information is the image information in the annotation data, and it becomes a class that indicates that it is a product, or that it is a product and the product identification information such as the product name or product code is "product category". Similarly, the product tag image information is the image information in the annotation data, and it becomes a class that indicates that it is a product tag, or that it is a product tag and its product name or product identification information is "product tag category". Furthermore, the shelf component image information is the image information in the annotation data, and it becomes a class that indicates that it is a shelf component, or that the part name of the shelf component is "shelf component". The object detection processing unit 203 can be trained to detect any object, although it is not limited to items commonly displayed on shelves, such as merchandise, product tags, and shelf components.

[0039] The individual discrimination processing unit 204 determines that any object type (class) detected by the object detection processing unit 203 from the captured image information, other than a predetermined class, is either a predetermined object or a candidate for a predetermined object. For example, an object type detected as anything other than merchandise, merchandise tags, or shelf components is determined to be a POP (Point of Purchase) display or a candidate for a POP. Furthermore, for objects detected as merchandise, merchandise tags, or shelf components, if they satisfy predetermined conditions, the individual discrimination processing unit 204 determines that the object type (class) is either a POP or a candidate for a POP.

[0040] For example, for an object detected as a product, the system determines whether a point-of-purchase (POP) display is attached to that area. If a POP is found to be present, the object in that area is classified as either a POP or a candidate for a POP. In this case, there are two methods for detecting a POP or a candidate for a POP attached to a displayed product from the area of ​​the object detected as a product, and a method for detecting a POP or a candidate for a POP by comparing the image information of the area of ​​the object detected as a product with the image information of a product without a POP.

[0041] In the former method, a large number of image data including the POP (point-of-purchase) display portion attached to the displayed products are collected in advance, and the POP portion is annotated for each image data containing the POP portion. Then, a learning model that has been previously trained using image data of normal displayed products without POP as annotation data is further trained using the above annotated image data to generate a new learning model. Image data of the region of an object detected as a product is input to this newly generated learning model, and in addition to outputting product identification information during product recognition processing, it is also possible to output whether or not a POP is attached to the displayed product in the image data.

[0042] In the latter method, after performing product recognition processing on the image information of the region of the object detected as a product, the image information of the object region is compared with standard image information of the recognized product without POP (e.g., sample information), and the similarity is compared. If the similarity is above a predetermined value, it is determined that the object detected as a product does not have POP; if the similarity is below the predetermined value, it is determined that the object detected as a product has POP or a candidate for POP.

[0043] In the method described above, comparing the image information of the region of an object detected as a product with the standard image information of the product without a POP (Point of Purchase) label recognized by the product recognition process, determining whether or not it is a POP based on the location of the differing region can lead to more accurate determination. For example, if the differing region in the two image pieces is determined to be localized to a specific location, such as the top, shoulder, or tail of the product, it can be determined to be a POP or a candidate for a POP.

[0044] Various known processing methods can be used for product recognition. For example, deep learning can be used to identify product identification information from image information of the object's region. In this case, the image information of the object's region may be input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many hidden layers are optimized, and product identification information may be identified based on its output value. As a learning model, a model can be used in which product identification information is given as ground truth data to various image information of the product. In addition to this, product identification information can be identified by comparing sample information of the product (image information of the product taken from one or more directions or information such as feature quantities based thereon) with image information of the object's region and using the similarity. Product recognition processing is not limited to these methods.

[0045] Furthermore, for objects detected as product tags, the system determines whether the image information of the object's region meets the requirements for product tags. For example, OCR recognition processing is performed on the region of an object detected as a product tag. If it contains characters typically included in product tags, such as price and product name, it is identified as a product tag; otherwise, it is identified as a POP (Point of Purchase) display or a candidate for a POP display. Alternatively, if the OCR recognition processing results in the inclusion of predetermined words commonly used in POP displays, such as "member discount," "tester," "sample," or "sample," it may be identified as a POP display or a candidate for a POP display. It may also be identified as a POP display or a candidate for a POP display if it contains handwritten text.

[0046] Furthermore, for objects detected as shelf components, the system determines whether the image information of the object's region meets the requirements for a shelf component. For example, OCR recognition processing is performed on the region of an object detected as a shelf component. If it contains a predetermined word, such as "member discount" or other words commonly used in POP displays, it is identified as a POP display or a candidate for a POP display; otherwise, it is identified as a shelf component.

[0047] If the individual discrimination processing unit 204 determines that the image information of an object's region is a candidate for a predetermined object, it may display the image information of the object's region on the display device 72 of the management terminal 2, accept input indicating whether or not it is an object, and if the input indicates it is an object, it may set the image information of the object's region to an object. For example, if the individual discrimination processing unit 204 determines that the image information of an object's region is a candidate for POP, it may display the image information of the object's region on the display device 72 of the management terminal 2, accept input indicating whether or not it is a POP, and if the input indicates it is a POP, it may set the image information of the object's region to POP.

[0048] The model processing unit 21 includes a classification processing unit 210, a provisional identification information processing unit 211, a model generation processing unit 212, a sample information receiving processing unit 213, a recognition processing unit 214, and an output processing unit 215. An example of a conceptual diagram of the processing of the model processing unit 21 is shown in Figures 7 and 8.

[0049] The classification processing unit 210 classifies the object image information of the object identified by the detection processing unit 20 using clustering or other methods. For example, it classifies the image information (POP image information) of the area of ​​an object identified as a POP in an image of a display shelf using clustering or other methods. When the object is a product displayed on a display shelf, a product tag (price card), a shelf, or a POP attached, mounted, or installed above or to the side of a shelf, the object image information will be the POP image information. In this case, the image information may be cropped using a rectangle that includes the area of ​​the POP, or it may be image information cropped in an arbitrary shape according to the outline of the POP. Furthermore, cropping the image information may be done physically, or the coordinate information of the area may be identified and its range determined.

[0050] The object identification information (object identification information) of the object captured in the object image information, such as the code for identifying the POP if the object is a POP (point-of-purchase) display, does not need to be known.

[0051] The classification processing unit 210 performs clustering processing on multiple object image information using a known method and classifies the image information into multiple groups. For example, if the object is a POP, it performs clustering processing on multiple POP image information using a known method and classifies the POP image information into multiple groups.

[0052] Although the classification processing unit 210 describes a case where object image information is classified using clustering, object image information may also be classified using methods other than clustering.

[0053] The provisional identification information processing unit 211 associates provisional identification information (provisional identification information) with each group of object image information classified by the classification processing unit 210. If the object is a POP, provisional identification information (provisional identification information) is associated with each group of POP image information classified by the classification processing unit 210. The provisional identification information may be automatically generated provisional identification information, or provisional identification information received as input may be associated by predetermined operations. Then, the provisional identification information associated with the object image information classified into each group is associated as labels to form a dataset, which is used as training data.

[0054] The model generation processing unit 212 generates a trained model by performing machine learning using known methods with the training data. Specifically, since provisional identification information is associated with each classified group of object image information, the trained model is generated using training data consisting of a dataset in which object image information and its label are used as provisional identification information. In the case of a POP (Point of Purchase) object, provisional identification information is associated with each group of classified POP image information, so the trained model is generated using training data consisting of a dataset in which POP image information and its label are used as provisional identification information.

[0055] The sample information receiving processing unit 213 receives image information to be input to the model generation processing unit 212. The image information received here is image information of an object for which object identification information for identifying the object is known. For example, it is image information of a POP (point of purchase) that may be attached to a display shelf, for which the identification information of the corresponding POP (POP identification information) is known. The image information input here will be called sample information.

[0056] The recognition processing unit 214 inputs the image information (sample information) received by the sample information receiving processing unit 213 into the learning model generated by the model generation processing unit 212, and outputs provisional identification information. In this case, when sample information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many hidden layers are optimized, provisional identification information is output as an output value. If multiple output values ​​are output along with their accuracy, the output value with the highest accuracy is adopted.

[0057] The output processing unit 215 links the provisional identification information, which is the output value output by the recognition processing unit 214, with the object identification information of the sample information used as input. Then, the output processing unit 215 outputs the object identification information linked to the provisional identification information output by the recognition processing unit 214 as the recognition result of the recognition processing unit 214.

[0058] Furthermore, instead of outputting object identification information linked to the provisional identification information output by the recognition processing unit 214, the output processing unit 215 may replace the object identification information with the provisional identification information associated with the group output by the classification processing unit 210, and have the model generation processing unit 212 execute the machine learning training process again.

[0059] Furthermore, if the accuracy of the provisional identification information output by the recognition processing unit 214 is below a predetermined threshold, the processing of the output processing unit 215 does not need to be executed.

[0060] By performing the above processes, it is possible to efficiently generate learning models used in machine learning. [Examples]

[0061] Next, an example of a processing process using the information processing system 1 of the present invention will be explained using the flowcharts in Figures 9 to 11. In the following explanation, we will describe a case in which captured image information of a store's display shelf is input from the image information input terminal 3, the area of ​​a POP is detected as the target object from the captured image information, and a machine learning model for identifying the POP is generated using the image information of the detected POP area.

[0062] First, the detection process in the detection processing unit 20 will be explained (S100).

[0063] Image information of store display shelves is input from the image information input terminal 3 and received by the image information input receiving processing unit 200 of the management terminal 2 (S110). An example of the input image information is shown in Figure 5(a). The system also accepts input of the date and time of shooting, store identification information, display shelf identification information, and image information identification information of the image information. The image information input receiving processing unit 200 then associates the received image information, date and time of shooting, store identification information, and image information identification information of the image information and stores them in the image information storage unit 201.

[0064] When the management terminal 2 receives a predetermined operation input, the image information orientation processing unit 202 extracts the captured image information stored in the image information storage unit 201, receives input for four points, such as the four corners of a display shelf, for performing orientation processing such as trapezoidal correction, and executes the orientation processing (S120).

[0065] Then, upon receiving a predetermined operation input at the management terminal 2, the object detection processing unit 203 performs object detection processing on the orthogonalized image information (S130) and classifies the types of objects depicted in the orthogonalized image information (S140).

[0066] Figure 12 schematically shows an example of the results of performing object detection processing from oriented image information. In Figure 12, "a" represents the types of products, "b" represents product tags, "c" represents shelf components, and "d" represents the types of objects detected other than "a" through "c". Note that the types of objects detected from the image information only need to be associated with their respective regions and do not need to be written on the images themselves. Furthermore, the regions of the detected objects may be cut out from the oriented image information, but they do not need to be actually cut out as long as the image information of those regions is in a processable state. Note that cutting out the image information of a region includes not only actually cutting out the image information of that region from the oriented image information, but also identifying the region using coordinate information or other means so that the image information of that region is in a processable state without cutting out the image information of that region.

[0067] The individual discrimination processing unit 204 identifies and cuts out the areas of objects other than products, product tags, and shelf components (areas of objects "d") from the area of ​​objects detected by the object detection processing unit 203 as POP areas (S150, S160).

[0068] Furthermore, the individual discrimination processing unit 204 determines whether the region of the object detected by the object detection processing unit 203 (S150) that has been detected as a product satisfies predetermined conditions and determines whether it is a POP region (S170).

[0069] For example, for an area of ​​an object detected as a product, the system determines whether a point-of-purchase (POP) display is attached to that area. If a POP is found to be present, the area is classified as a POP display. Alternatively, for an area of ​​an object detected as a product, the system can detect POP displays attached to the displayed product, or it can detect POP displays by comparing the image information of the area of ​​the object detected as a product with the image information of a product without a POP display (a sample image of the product).

[0070] Similarly, for areas of objects detected as product tags, it is determined whether the image information of that area meets the requirements for product tags. Furthermore, for objects detected as shelf components, it is determined whether the image information of that area meets the requirements for shelf components.

[0071] As described above, for the regions of objects detected as either products, product tags, or shelf components (S150), it is determined whether the predetermined conditions for each object are met (S170), and if the conditions are met, all or part of the region is identified as a POP (Point of Purchase) display (S180). An example of this process is shown in Figure 13. Furthermore, Figure 14 shows examples of each region after the region of an object detected as either a product, product tag, or shelf component that satisfies the predetermined conditions has been identified as a POP region, based on the object detection results in Figure 12.

[0072] By performing the above processing in the detection processing unit 20, the POP region in the image information can be detected. This makes it possible to detect image information of various types of POP regions from captured image information of various display shelves, and to obtain image information to be used in generating the learning model in the model processing unit 21 described later.

[0073] Next, we will explain the process by which the model processing unit 21 generates a learning model using the image information of the POP detected by the detection processing unit 20 (S200).

[0074] Figure 15 shows an example of POP image information detected by the detection processing unit 20. At this point, only the POP image information, which is the image information of the object, has been detected from the captured image information, so it is not necessary to know the POP identification information (object identification information) for identifying the object POP.

[0075] Then, the classification processing unit 210 uses the POP image information, which is the image information of the target object, to perform known clustering processes, etc., to group similar image information together and classify it into multiple groups (S210). Figure 16 schematically shows this state.

[0076] The provisional identification information processing unit 211 associates provisional identification information with each group classified by the classification processing unit 210 (S220). For example, it associates arbitrary provisional identification information such as "a123", "b456", "c789", "d012", and "e345" with each group. Figure 17 schematically shows this state. In this way, the provisional identification information is associated as a label with the object image information of each group. Then, a dataset using the POP image information which is the object image information and the labels of the provisional identification information is used as training data.

[0077] When the classification processing unit 210 associates provisional identification information with each group it has classified, the model generation processing unit 212 performs machine learning using known methods as training data (S230) and generates a training model (S240). This is schematically shown in Figure 18. Although Figure 18 shows a case where one POP image information and provisional identification information are used as the training dataset for each group, in practice it is preferable to associate each POP image information and provisional identification information in each group and perform machine learning using that as the training dataset. This is because even within the same group, there are not only perfectly matching image information but also some differences, and associating these with the provisional identification information in that group and performing machine learning on them may lead to improved accuracy.

[0078] Next, the sample information receiving processing unit 213 receives sample information, which is image information in which POP identification information, which is object identification information, is known, and inputs the received sample information into the learning model generated in S240 (S250). The recognition processing unit 214 receives the sample information input and outputs the output result of machine learning using the learning model. As this output result, provisional identification information is output (S260). This is schematically shown in Figure 19.

[0079] The sample information already includes object identification information such as POP identification information. Therefore, the provisional identification information output by the recognition processing unit 214 is thought to correspond to the object identification information of the sample information input into the learning model. Accordingly, the output processing unit 215 links the output provisional identification information with the object identification information of the input sample information (S270). This is schematically shown in Figure 20.

[0080] This linking allows for the correspondence between provisional identification information and POP identification information, which is object identification information. By storing this correspondence, the recognition processing unit 214 can replace the provisional identification information output by the recognition processing unit 214 with the corresponding object identification information and output it. When image information to be processed for identification is input to the learning model, the learning model can output object identification information as its output value.

[0081] Through the model processing described above, it is possible to generate a machine learning model for the target object. For example, it is possible to detect POP (point-of-purchase) displays from image information of a display shelf, and then use the detected POP to generate a machine learning model for detecting POP from input image information. [Examples]

[0082] Another embodiment of the detection process in the detection processing unit 20 will be described. In this embodiment, the detection processing unit 20 further includes a region detection processing unit 205. An example of the configuration of the detection processing unit 20 in this embodiment is shown in Figure 21, and an example of the processing process of the detection processing unit 20 is shown in the flowchart of Figure 22. Furthermore, an example of the processing overview is shown in Figure 23, and an example of the individual discrimination process is shown in Figure 24.

[0083] The area detection processing unit 205 detects the product display area, the product tag area, and the shelf top area, which is the area above the shelf, from the image information (including orthogonalized image information) stored in the image information storage unit 201. For the detection of the product display area, product tag area, and shelf top area, the operator of the management terminal 2 may manually specify the product display area, product tag area, and shelf top area, and the area detection processing unit 205 may accept this. Alternatively, based on the information of the product display area, product tag area, and shelf top area that was manually entered the first time, the area detection processing unit 205 may automatically detect the product display area, product tag area, and shelf top area from the second time onward. The area detection processing unit 205 may also detect and process areas other than those listed above.

[0084] Figure 25 schematically shows the state in which input for specifying the product display area, product tag area, and shelf top area has been received for image information that has been orthogonalized from image information of a display shelf on which products are displayed.

[0085] Furthermore, when detecting the product display area, product tag area, and shelf top area, the region detection processing unit 205 may use deep learning to identify the product display area, product tag area, and shelf top area. In this case, the orthogonalized image information may be input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized, and the product display area, product tag area, and shelf top area may be detected based on the output value. As a learning model, a model in which the product display area, product tag area, and shelf top area are given as ground truth data for various image information can be used.

[0086] In addition to using deep learning as described above, the region detection processing unit 205 may also detect regions using various known methods.

[0087] The region detection processing unit 205 may either actually extract the detected region as image information, or it may virtually extract it by specifying the range of the region without actually extracting it as image information. In this case, the range of the region can be specified by coordinates.

[0088] In this embodiment, the object detection processing unit 203 may perform object detection processing from the range of orienting image information, as in the detection processing unit 20 of Embodiment 1, or it may perform it for each region detected by the region detection processing unit 205.

[0089] In this embodiment, the individual discrimination processing unit 204 determines the type of each object detected by the object detection processing unit 203 using the object's type (class) and the region in which the object is located. For example, discrimination is performed according to predetermined individual discrimination conditions, such as the discrimination table shown in Figures 23 and 24.

[0090] In Figure 23, "a1" through "a4", "b1" through "b10", and "d1" through "d5" correspond to "a1" through "a4", "b1" through "b10", and "d1" through "d5" in Figure 24.

[0091] Next, an example of the processing process of the detection processing unit 20 in this embodiment will be explained using the flowchart in Figure 22.

[0092] Image information of store display shelves is input from the image information input terminal 3 and received by the image information input receiving processing unit 200 of the management terminal 2 (S300). The image information input receiving processing unit 200 also receives input of the shooting date and time, store identification information, display shelf identification information, and image information identification information of the captured image. The image information input receiving processing unit 200 then associates the received image information, shooting date and time, store identification information, display shelf identification information, and image information identification information of the captured image and stores it in the image information storage unit 201.

[0093] When the management terminal 2 receives a predetermined operation input, the image information orientation processing unit 202 extracts the captured image information stored in the image information storage unit 201, accepts input for four arbitrary points that are vertices for orientation processing such as trapezoidal correction processing, for example, the four corners of a display shelf, and executes the orientation processing (S310).

[0094] Then, upon receiving a predetermined operation input at the management terminal 2, the region detection processing unit 205 performs region detection processing on the orthogonalized image information to detect the product display area, product tag area, and shelf top area (S320), and determines the extent of each area.

[0095] Furthermore, upon receiving a predetermined operation input at the management terminal 2, the object detection processing unit 203 performs object detection processing on the orthogonalized image information (S330) and classifies the types of objects depicted in the image information (S340). An example of the result of performing object detection processing on the image information is classified as shown in Figure 12, similar to Example 1.

[0096] Then, the individual discrimination processing unit 204 classifies each object in the image information detected by the object detection processing unit 203 according to predetermined judgment conditions, using the type of object and the region in which the object is located (the region determined in S320).

[0097] Furthermore, the individual discrimination processing unit 204 uses the type of object in the image information detected by the object detection processing unit 203 and the region in which the object is located to determine whether predetermined conditions are met (S350), and to determine whether the type of object is a POP or something other than a POP (products, product tags, shelf components, etc.) (S360, S370). Note that the region in which each object is located can be determined by an appropriate method, for example, whether the centroid of the region in which the object's area is located is included in the product, product tag, or upper shelf region.

[0098] For example, if an object is in a product display area and its type is a product, the individual discrimination processing unit 204 determines whether there is a product-attached POP in the area of ​​the object. For example, similar to Embodiment 1, discrimination is performed by detecting a POP attached to a displayed product from the area of ​​an object detected as a product, or by detecting a POP by comparing the image information of the area of ​​an object detected as a product with the image information of a product without a POP. If a product-attached POP is detected, that area is identified as a POP (a2), and the other areas are identified as products (a1). If a product-attached POP is not detected, OCR recognition processing is performed on that area to determine whether it contains predetermined words used in POPs, such as "tester," "sample," or "sample." If it does not contain these words, the object in that area is identified as a product (a1). If it does contain these words, the object in that area is identified as a POP such as a tester or sample (a3).

[0099] Furthermore, if an object is located in the product display area and its type is a product tag, the individual identification processing unit 204 determines that the object in that area is a POP (d1).

[0100] Furthermore, if an object is located in the product display area and its type is a shelf component, the individual discrimination processing unit 204 determines that the object in that area is a shelf component (c).

[0101] Furthermore, if an object is located in the product display area and its type is other than products, product tags, or shelf components, it is determined to be a POP (point of purchase) display (d2).

[0102] If an object is located in the product tag area and its type is a product, the individual identification processing unit 204 determines that it is a POP such as a tester or sample (d4).

[0103] Furthermore, if an object is located in the product tag area and its type is a product tag, OCR recognition processing is performed on the object's area. If it is determined that there is handwriting in that area, it is determined to be a POP (retail POP) created by the store (retailer) (b4). If the OCR recognition processing on the object's area determines that there is no handwriting, and it is determined that the price and product name are displayed, and that words indicating conditions such as "member" are included, then the type of object is determined to be a product tag indicating a set product (b2). On the other hand, if it is determined that the price and product name are displayed, but words indicating conditions such as "member" are not included, then the type of object is determined to be a product tag (b1). If it is determined that the price and product name are not displayed, and there is a point display, then the type of object is determined to be a "point display" POP (b3). Furthermore, if there is no point display in the above case, it is determined to be a POP (manufacturer POP) created by the product manufacturer (b9). Note that here we have shown a case where manufacturer POPs and retail POPs are distinguished and identified, but it is also acceptable to simply identify them as POPs.

[0104] Furthermore, if an object is located in the product tag area and its type is a shelf component, the individual discrimination processing unit 204 determines that the object in that area is a shelf component (c).

[0105] Furthermore, if an object is located in the product tag area and its type is other than a product, product tag, or shelf component, OCR recognition processing is performed on that area. The system then determines whether it contains predetermined words used in POP displays, such as "tester," "sample," or "demonstration," and if it does, it is determined to be a POP display such as a tester or sample (d4). On the other hand, if it does not contain predetermined words used in POP displays such as "tester," "sample," or "demonstration," it is determined to be a POP display such as promotional goods (d3).

[0106] If an object is located in the upper area of ​​the shelf and its type is merchandise, the individual discrimination processing unit 204 determines that it is stock merchandise (a4).

[0107] Furthermore, if an object is located in the upper area of ​​the shelf and its type is a product tag, OCR recognition processing is performed on the area of ​​the object, and if it is determined that there is handwriting in that area, it is determined to be a retail POP (b8). If, as a result of the OCR recognition processing on the area of ​​the object, it is determined that there is no handwriting in that area, and it is determined that the price and product name are displayed, and that words indicating conditions such as "member" are also included, then the type of object is determined to be a product tag indicating a set product (b6). On the other hand, if it is determined that the price and product name are displayed, but words indicating conditions such as "member" are not included, then the type of object is determined to be a product tag (b5). If it is determined that the price and product name are not displayed, and there is a point display, then the type of object is determined to be a "point display" POP (b7). Furthermore, if there is no point display in the above case, then it is determined to be a manufacturer's POP (b10).

[0108] Furthermore, if an object is located in the upper area of ​​the shelf and its type is a shelf component, the individual discrimination processing unit 204 determines that the object in that area is a shelf component (c).

[0109] Furthermore, if an object is located in the upper area of ​​the shelf and its type is other than merchandise, product tags, or shelf components, it is identified as a large promotional item or other POP display (d5).

[0110] As described above, the individual discrimination processing unit 204 can determine whether an object is a POP or not by using the region detected by the region detection processing unit 205 and the type of object detected by the object detection processing unit 203, and can detect the region of a POP in the image information. [Examples]

[0111] As a variation of the processing in Examples 1 and 2, the processing shown in the flowchart of Figure 26 may be performed. In this case, the individual discrimination processing unit 204 determines that an object detected by the object detection processing unit 203 is a POP (point of purchase) if the type of object is other than merchandise, merchandise tags, or shelf components (S460). For other objects, the same processing as in Figure 22's S350 to S370 should be performed. That is, the object detection processing unit 203 uses the type of object in the image information detected by the object detection processing unit 203 and a region of that object to determine whether a predetermined condition is met (S470), and then determines whether the object is a POP or something other than a POP (merchandise, merchandise tags, shelf components, etc.) (S480, S490). Note that the processing from S400 to S440 may be the same as the processing from S300 to S340. [Examples]

[0112] In addition to the processing of the detection processing unit 20 in Examples 1 to 3, a modified model processing unit 21 will be described. The classification processing performed by the classification processing unit 210 may be less accurate than classification processing performed by a human. Therefore, errors may be introduced. To address this, a cleansing processing unit 216 may be provided that performs a cleansing process on the image information (target image information) of each group classified by the classification processing unit 210, and removes object image information that may contain errors from the image information classified into the group, thereby reducing the amount of object image information used as training data for the model generation processing unit 212. An example of the configuration of the model processing unit 21 in this case is shown in Figure 27.

[0113] In other words, when the classification processing unit 210 classifies the object image information into groups, it calculates index values ​​such as feature quantities that indicate the characteristics of the image information in each group. Then, for each group, it calculates the information distance from a reference value such as the median or mean of the index values ​​for each object image information (a value calculated using a predetermined calculation formula that shows the deviation from the reference value to the index value for each image information), and object image information whose information distance deviates by a certain value or a certain ratio or more may be excluded from that group.

[0114] In addition to using information distance as a cleansing process, the similarity of image information may also be used. In this case, the similarity between image information within a group is quantified, and if the number of image information for which a certain level of similarity is calculated is equal to or greater than a predetermined value, it is retained as the target image information; if it is less than the predetermined value, it is excluded from the target image information.

[0115] For example, if a group has five target image information files (image information 1 to image information 5), the similarity of each of these five image information files with the other image information files is calculated. That is, the similarity of image information 1 with image information 2 to image information 5 is calculated, image information 2 with image information 3 to image information 5, image information 3 with image information 4 to image information 5, and image information 4 with image information 5. In this way, the similarity between each image information file can be calculated.

[0116] Assume that for image information 1, the similarity to image information 2 is 0.9, the similarity to image information 3 is 0.95, the similarity to image information 4 is 0.3, and the similarity to image information 5 is 0.8. For image information 2, the similarity to image information 3 is 0.8, the similarity to image information 4 is 0.2, and the similarity to image information 5 is 0.6. For image information 3, the similarity to image information 4 is 0.4, the similarity to image information 5 is 0.9, and for image information 4, the similarity to image information 5 is 0.5.

[0117] Here, assuming a reference image similarity of 0.75 and 3 images falling below this reference, for image information 1, only one image (image information 4) falls below the reference similarity; for image information 2, two images (image information 4 and image information 5) fall below the reference similarity; for image information 3, only one image (image information 4) falls below the reference similarity; for image information 4, four images (image information 1, image information 2, image information 3, and image information 5) fall below the reference similarity; and for image information 5, two images (image information 2 and image information 4) fall below the reference similarity.

[0118] Therefore, the cleansing processing unit 216 excludes image information 4, which has three or more image information that falls below the standard similarity, from the group, and proceeds to execute the processing of the provisional identification information processing unit 211 with the four image information of the target object in that group: image information 1, image information 2, image information 3, and image information 5.

[0119] As in this embodiment, by excluding divergent image information from the grouped object image information, the grouped object image information can be narrowed down, leading to improved accuracy. [Examples]

[0120] When extracting POP (Point of Purchase) image information from displays on shelves, etc., multiple similar POPs are sometimes placed adjacent to each other in any direction (up, down, left, or right) to attract customer attention.

[0121] Therefore, after extracting POP image information as object image information from the image information of the display shelf, the classification processing unit 210 may compare the similarity of adjacent POP image information as object image information, and if certain conditions are met, for example, if the similarity is above a predetermined threshold, it may determine that they are objects of the same type and classify the adjacent POP image information as object image information into the same group. [Examples]

[0122] When the target object is a point-of-purchase (POP) display for products displayed on shelves, it is preferable to generate a learning model of the POPs that are likely to be displayed in advance. However, in that case, the number of target POPs could range from several hundred to several thousand, or even tens of thousands. In this case, if the learning model is capable of recognizing several thousand to tens of thousands of learning models, the sample information to be input into the sample information receiving processing unit 213 will also be several thousand to tens of thousands. Although the workload is reduced compared to conventional learning models, the workload is still significant.

[0123] On the other hand, it is generally known in marketing that about 10% of product categories tend to account for about 90% of sales within that category. Therefore, the sample information to be input into the sample information receiving processing unit 213 may be POP (point of purchase) displays for products with high sales. In this case, POP displays provided by marketing companies or the like may be used.

[0124] This allows for the recognition of POP displays for approximately 90% of products based on sales figures, making it practically sufficient for outputting object identification information. However, if POP displays for products with low sales are input into the learning model, the object identification information and provisional identification information are not linked, so the output processing unit 215 will output the provisional identification information as is. When the output processing unit 215 outputs provisional identification information for which there is no correspondence between the provisional identification information and the object identification information, it may display a predetermined message such as "No correspondence information."

[0125] Furthermore, if it is sufficient to recognize only the POP displays of products belonging to a specific company or organization, the image information of that company's products may be input to the sample information receiving processing unit 213 as sample information. This allows the system to output object identification information for the POP displays of products belonging to that company. On the other hand, if POP displays of products belonging to a company other than the specified company are input, the output processing unit 215 will output the provisional identification information as is, since there is no association between the object identification information and the provisional identification information, similar to the case described above. When the output processing unit 215 outputs provisional identification information for which there is no association between the provisional identification information and the object identification information, it may display a predetermined message such as "No association information." [Examples]

[0126] For the learning models generated as in Examples 1 to 6, the learning models may be reconstructed again for the purpose of improving accuracy. In this case, new sample information may be input and configured to associate object identification information with provisional identification information that is not currently associated with object identification information, or the classification processing unit 210 may be made to execute the classification process using the object image information of the new object so that the new object can be recognized. [Examples]

[0127] In the above-described Examples 1 to 7, we explained the generation of a learning model for identifying POP displays on a display shelf from image information of the shelf, but the present invention can be applied to other cases as well. In particular, the present invention is useful for automating the identification of various types of objects from image information.

[0128] One example is when the target object is an animal. For instance, if a large number of multiple animal species, such as sea lions or fur seals, inhabit an area inaccessible to humans, this system can be applied to efficiently identify the species and population size. In this case, the habitat is photographed from above using a drone, object image information is extracted for each individual, and the object image information for each individual is subjected to image classification processing in the classification processing unit 210 and grouped. Then, the provisional identification information processing unit 211 associates the provisional identification information with each group and generates training data, which is then used in the model generation processing unit 212 to perform machine learning and generate a training model. The specimen information receiving processing unit 213 receives input of specimen information by species, and the recognition processing unit 214 inputs the received specimen information as input values ​​into the training model and outputs provisional identification information. This links the provisional identification information with the corresponding object identification information (such as the scientific name of the animal) of the input specimen information, allowing the output processing unit 215 to output object identification information.

[0129] Similarly, birds and plants may be used as the target objects. Even in the case of birds and plants, the processing can be carried out in the same way as for animals as described above, and processing can be performed by substituting "animals" for "birds" and "plants." [Industrial applicability]

[0130] By using the information processing system 1 of the present invention, it is possible to efficiently generate learning models used in machine learning. [Explanation of Symbols]

[0131] 1: Information Processing System 2: Management terminal 3: Image information input terminal 20: Detection Processing Unit 21: Model Processing Section 70: Arithmetic device 71:Storage device 72:Display device 73: Input device 74: Communication equipment 200: Image Information Input Reception Processing Unit 201: Image Information Storage Unit 202: Image Information Orientation Processing Unit 203: Object Detection Processing Unit 204: Individual Discrimination Processing Unit 205: Region detection processing unit 210: Classification Processing Unit 211: Provisional Identification Information Processing Unit 212: Model generation processing unit 213: Specimen Information Reception Processing Unit 214: Recognition Processing Unit 215: Output Processing Unit 216: Cleansing Treatment Section

Claims

1. An information processing system that performs processing related to learning models used in machine learning, A detection processing unit that detects POP from image information, The system includes a model processing unit that generates a machine learning model using the detected POP, The aforementioned model processing unit, A classification processing unit that classifies the image information of the detected POP, A provisional identification information processing unit that associates the classified group with provisional identification information, A model generation processing unit generates a learning model by performing a machine learning learning process using learning data that includes image information of POPs included in the group and the provisional identification information. An information processing system characterized by having the following features.

2. The aforementioned model processing unit, A cleansing processing unit performs a cleansing process to remove image information of POPs that may contain errors from the image information of POPs included in the aforementioned classified group. The information processing system according to claim 1, characterized by having the following features.

3. Computers, A detection processing unit that detects POP from image information. A model processing unit that generates a machine learning model using the detected POPs. An information processing program that functions as such, The aforementioned model processing unit, A classification processing unit that classifies the image information of the detected POP, A provisional identification information processing unit that associates the classified group with provisional identification information, A model generation processing unit generates a learning model by performing a machine learning learning process using learning data that includes image information of POPs included in the group and the provisional identification information. An information processing program characterized by having the following features.

Images