Information Processing System

Abstract

To provide an information processing system configured to improve recognition accuracy in identifying an object from image information.SOLUTION: An information processing system for identifying an object included in image information comprises: a first learning processing unit which generates a first learning model by machine learning using first annotation data in which attributes are associated with data obtained by masking the inside of a closed region of an outline of the object; and a recognition processing unit which identifies object identification information of the object included in image information. The recognition processing unit specifies, using the image information and the first learning model, the outline of the object included in the image information, as an outline region, identifies, when an attribute corresponding to the specified outline region does not satisfy a predetermined condition, one or more pieces of object identification information of the object included in the specified outline region, or identifies, when the attribute corresponding to the specified outline region satisfies the predetermined condition, the object included in the specified outline region as a predetermined object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system that does not reduce recognition accuracy when identifying an object from image information. [Background technology]

[0002] In various stores such as convenience stores and supermarkets, it is common to sell products and other objects by placing them on display shelves. In some cases, multiple objects are displayed horizontally to make them more visible to customers, and in other cases, objects are displayed vertically so that even if one object is purchased, another person can purchase the same object. In terms of sales strategies for objects, it is important to manage where and how many objects are displayed on the display shelves.

[0003] Therefore, in order to grasp the actual display situation of objects in a store, there is a method of automatically identifying the displayed objects from the captured image information by photographing the display shelves with a photographing device. For example, there is a method of using image recognition technology for images of the display shelves in a store based on a specimen image of each object. Examples of such conventional technology include Patent Document 1 and Patent Document 2 below. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 5-342230 [Patent Document 2] Japanese Patent Application Publication No. 5-334409 Summary of the Invention [Problem to be solved by the invention]

[0005] The invention of Patent Document 1 is a system that supports even those without knowledge to know on which shelf an object should be displayed. Therefore, it is possible to know where an object should be displayed, but it does not specify the object that is displayed. Patent Document 2 is a system that supports the input of object images in a shelf allocation support system that supports the display of objects. However, the system of Patent Document 2 only supports the input of object images when using the shelf allocation support system, and even if this system is used, it is not possible to grasp the specific display status of objects.

[0006] Furthermore, in addition to Patent Document 1 and Patent Document 2, there is also a technology that uses image recognition processing technology to identify objects displayed on display shelves from image information captured on the shelves. This is useful in that it allows the actual display situation in a store to be grasped.

[0007] In conventional technology, when identifying an object using image recognition processing technology, a rectangular area where the object is thought to be located is detected from image information taken of a display shelf, and the rectangular area is matched with a sample image of the object, or a deep learning process is performed using the rectangular area as an input value to identify the object.

[0008] However, the shape (contour) of an object is not limited to a rectangle. When detecting an area where an object is thought to be located as a rectangle, parts of other objects may be reflected in the rectangular area. In addition, the background may be reflected in the rectangular area. Therefore, if such rectangular areas are used as training data for deep learning, images to be processed, or image matching processing, there is a problem that the accuracy of identifying objects may be reduced.

[0009] Furthermore, if an object is unknown due to being unlearned or the like, performing subsequent object identification processing will result in erroneous recognition, but performing identification processing of such erroneous objects is a waste of processing, and it is preferable to exclude such objects as early as possible. However, this has not been possible with conventional methods. In addition, it is difficult to detect an object as an unlearned object itself. [Means for solving the problem]

[0010] In view of the above problems, the present inventor has invented an information processing system that improves the accuracy of identifying an object when identifying an object captured in image information.

[0011] A first invention is an information processing system for identifying an object shown in image information, the information processing system having a first learning processing unit that performs machine learning using first annotation data in which data obtained by masking an inside of a closed region of the object's outline is associated with an attribute to create a first learning model, a second learning processing unit that performs machine learning using second annotation data in which image information of the object is associated with object identification information to create a second learning model, and a recognition processing unit that identifies object identification information of the object shown in the image information, the recognition processing unit using the image information and the first learning model to identify the outline of the object shown in the image information as an outline region, and Output the confidence level for the corresponding attribute of the contour region; The corresponding attributes of the specified contour area If the reliability of is below a certain threshold, If not, using the image information of the specified outline region and the second learning model, identify one or more object identification information of the object appearing in the specified outline region; Outputting a reliability of the identified object identification information; Object identification information of the identified object If the reliability of is below a certain threshold, If not, one or more object identification information of the object shown in the outline region is set as the identified object identification information, and the object identification information of the identified object is The reliability of is below a certain threshold. In this case, object identification information of the object shown in the outline region is unknown and identifying the object as being of the specified outer shape region. The reliability of is below a certain threshold. In this case, the object in the specified outline region is unknown It is an information processing system that identifies an object as being of a certain type.

[0012] As in the present invention, by identifying object identification information using a learning model that is machine-learned using the first annotation data based on data in which the inside of the closed region of the object's outline is masked, the outline region cut out from the image information can be changed from a conventional rectangular region to an area that follows the outline of the object. This can reduce the inclusion of other objects or the background, improving the accuracy of identifying the object.

[0013] In addition, when the attribute corresponding to the outline region of the identified object does not satisfy a predetermined condition, for example, when the reliability is lower than a predetermined threshold, the outline region can be identified as a predetermined object, such as an unknown object (for example, an object that has not been learned), without performing the identification process of the object identification information of the outline region. This makes it possible to detect unknown objects and reduce the processing load. Since the outline region is composed of an area that is the outline of an object, it is preferable to identify the object identification information by performing processing such as those of the inventions. Furthermore, if the object identification information of the identified object satisfies a predetermined condition, for example, if the reliability is lower than a predetermined threshold, the outline region is identified as a predetermined object, for example, an unknown object, regardless of the identified object identification information. This makes it possible to avoid the identification of object identification information with low reliability.

[0014] In the above-mentioned invention, the first learning processing unit can be configured as an information processing system that performs machine learning by image segmentation using the first annotation data to create the first learning model.

[0015] When performing machine learning, machine learning using image segmentation methods is preferred.

[0019] In the above-mentioned invention, the recognition processing unit can be configured as an information processing system that identifies object identification information of an object depicted in the outline region by performing image matching processing between image information of the identified outline region and specimen information of the object stored in a specimen information storage unit.

[0020] The process of identifying the object identification information of the object captured in the outline region may use image matching processing as in the present invention.

[0021] A fourth invention is an information processing system for identifying an object shown in image information, comprising: a first learning processing unit that performs machine learning using first annotation data in which data obtained by masking an inside of a closed region of an object's outline is associated with an attribute to create a first learning model; and a second learning processing unit that performs machine learning using second annotation data in which image information of the object is associated with object identification information to create a second learning model, and the system identifies the outline of the object shown in the image information as an outline region using the image information and the first learning model, Output the confidence level for the corresponding attribute of the contour region; The corresponding attributes of the specified contour area If the reliability of is below a certain threshold, If not, one or more object identification information of the object appearing in the specified outline region is identified using the image information of the specified outline region and the second learning model; outputting a reliability of the identified object identification information; Object identification information of the identified object If the reliability of is below a certain threshold, If not, one or more object identification information of the object shown in the outline region is set as the identified object identification information, and the object identification information of the identified object is The reliability of is below a certain threshold. In this case, object identification information of the object shown in the outline region is unknown and identifying the object as being an object of the specified outline region. The reliability of is below a certain threshold. In this case, the object in the specified outline region is unknown It is an information processing system that identifies an object as being of a certain type.

[0022] A fifth invention is an information processing system for identifying an object shown in image information, the information processing system including: an image information input reception processing unit that receives an input of image information obtained by photographing an object; and an object recognition processing unit that identifies object identification information of the object shown in the image from the received image information or image information obtained by orthogonalizing the image information. The object recognition processing unit identifies the outline of the object shown in the image as an outline area by using a first learning model created by machine learning using first annotation data in which data obtained by masking an inside of a closed area of ​​the outline of the object is associated with an attribute, and the received image information or image information obtained by orthogonalizing the image information. Outputting a reliability of the attribute corresponding to the specified outline region;The corresponding attributes of the specified contour area If the reliability of is below a certain threshold, If not, using a second learning model created by machine learning using second annotation data in which image information of the object and object identification information are associated with each other, and the image information of the specified outer region, identify one or more object identification information of the object appearing in the specified outer region; Outputting a reliability of the identified object identification information; Object identification information of the identified object If the reliability of is below a certain threshold, If not, one or more object identification information of the object shown in the outline region is set as the identified object identification information, and the object identification information of the identified object is The reliability of is below a certain threshold. In this case, object identification information of the object shown in the outline region is unknown and identifying the object as being of the specified outer shape region. The reliability of is below a certain threshold. In this case, the object in the specified outline region is unknown It is an information processing system that identifies an object as being of a certain type.

[0023] Even if these inventions are configured in the manner described above, the same technical effects as those of the first invention can be obtained.

[0024] The first invention can be realized by loading the program of the present invention into a computer and executing it. That is, the information processing program causes a computer to function as a first learning processing unit that performs machine learning using first annotation data in which data obtained by masking the inside of a closed region of an object's outline is associated with an attribute to create a first learning model, a second learning processing unit that performs machine learning using second annotation data in which image information of an object is associated with object identification information to create a second learning model, and a recognition processing unit that identifies object identification information of an object shown in image information, in which the recognition processing unit uses the image information and the first learning model to specify the outline of an object shown in the image information as an outline region, and Output the confidence level for the corresponding attribute of the contour region; The corresponding attributes of the specified contour area If the reliability of is below a certain threshold, If not, using the image information of the specified outline region and the second learning model, identify one or more object identification information of the object appearing in the specified outline region; Outputting a reliability of the identified object identification information; Object identification information of the identified object If the reliability of is below a certain threshold, If not, one or more object identification information of the object shown in the outline region is set as the identified object identification information, and the object identification information of the identified object is The reliability of is below a certain threshold. In this case, object identification information of the object shown in the outline region is unknown and identifying the object as being of the specified outer shape region. The reliability of is below a certain threshold. In this case, the object in the specified outline region is unknown It is an information processing program that identifies an object as being

[0025] The fourth invention can be realized by loading the program of the present invention into a computer and executing it. That is, the information processing program causes a computer to function as a first learning processing unit that performs machine learning using first annotation data in which data obtained by masking the inside of a closed region of an object's outline is associated with an attribute to create a first learning model, and a second learning processing unit that performs machine learning using second annotation data in which image information of an object is associated with object identification information to create a second learning model, and the information processing unit causes a computer to function as a first learning processing unit that performs machine learning using image information and the first learning model to identify the outline of an object shown in the image information as an outline region, and a second learning processing unit that performs machine learning using the image information and the first learning model to identify the outline of the object shown in the image information as an outline region. Output the confidence level for the corresponding attribute of the contour region; The corresponding attributes of the specified contour area If the reliability of is below a certain threshold, If not, one or more object identification information of the object appearing in the specified outline region is identified using the image information of the specified outline region and the second learning model; outputting a reliability of the identified object identification information; Object identification information of the identified object If the reliability of is below a certain threshold, If not, one or more object identification information of the object shown in the outline region is set as the identified object identification information, and the object identification information of the identified object is The reliability of is below a certain threshold. In this case, object identification information of the object shown in the outline region is unknown and identifying the object as being an object of the specified outline region. The reliability of is below a certain threshold. In this case, the object in the specified outline region is unknown It is an information processing program that identifies an object as being

[0026] The fifth invention can be realized by loading the program of the present invention into a computer and executing it. That is, the information processing program causes a computer to function as an image information input reception processing unit that receives input of image information of an object, and an object recognition processing unit that identifies object identification information of the object in the image from the received image information or image information obtained by orthogonalizing the image information, in which the object recognition processing unit specifies the outline of the object in the image as an outline area using a first learning model created by machine learning using first annotation data in which data obtained by masking the inside of a closed area of ​​the outline of the object is associated with an attribute, and the received image information or image information obtained by orthogonalizing the image information, Outputting a reliability of the attribute corresponding to the specified outline region; The corresponding attributes of the specified contour area If the reliability of is below a certain threshold, If not, using a second learning model created by machine learning using second annotation data in which image information of the object and object identification information are associated with each other, and the image information of the specified outer region, identify one or more object identification information of the object appearing in the specified outer region; Outputting a reliability of the identified object identification information; Object identification information of the identified object If the reliability of is below a certain threshold, If not, one or more object identification information of the object shown in the outline region is set as the identified object identification information, and the object identification information of the identified object is The reliability of is below a certain threshold. In this case, object identification information of the object shown in the outline region is unknown and identifying the object as being of the specified outer shape region. The reliability of is below a certain threshold. In this case, the object in the specified outline region is unknown It is an information processing program that identifies an object as being Effect of the Invention

[0027] By using the information processing system of the present invention, it is possible to improve the accuracy of identification when identifying an object captured in image information. [Brief description of the drawings]

[0028] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an information processing system according to the present invention. [Diagram 2] 2 is a block diagram illustrating an example of a configuration of an object recognition processing unit in the information processing system of the present invention. FIG. [Diagram 3] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer used in the information processing system of the present invention. [Figure 4] 10 is a flowchart showing an example of a learning process in the information processing system of the present invention. [Diagram 5] 5 is a flowchart showing an example of a processing process of recognition processing in the information processing system of the present invention. [Figure 6] FIG. 2 is a diagram illustrating an example of first annotation data. [Figure 7] FIG. 11 is a diagram illustrating another example of the first annotation data. [Figure 8] FIG. 10 is a diagram illustrating an example of second annotation data. [Figure 9] FIG. 11 is a diagram illustrating another example of the second annotation data. [Figure 10] FIG. 4 is a diagram showing an example of captured image information. [Figure 11] FIG. 11 is a diagram showing another example of captured image information. [Figure 12] 11 is a diagram showing an example of image information obtained by normalizing the captured image information of FIG. 10. FIG. [Figure 13] 12 is a diagram showing an example of image information obtained by normalizing the captured image information of FIG. 11. FIG. [Figure 14] FIG. 11 is a block diagram illustrating an example of a configuration of an information processing system according to a second embodiment. [Figure 15] FIG. 13 is a diagram showing a schematic diagram of a state in which input specifying a shelf area is received for upright image information obtained by orthogonally orthogonally orthogonalizing image information obtained by photographing a display shelf on which objects are displayed. [Figure 16]FIG. 13 is a schematic diagram showing a state in which input specifying a shelf area has been received for upright image information obtained by uprighting image information captured of a display shelf on which objects are hung and displayed. [Figure 17] 13 is a diagram showing an example of a case where an outer shape area is specified from image information of a shelf area. FIG. [Figure 18] 13 is a flowchart showing an example of a processing process of recognition processing in the information processing system according to the second embodiment. [Figure 19] FIG. 4 is a diagram showing an example of captured image information. [Figure 20] 20 is a diagram showing an example of normalized image information obtained by performing normalization processing on the captured image information of FIG. 19. FIG. [Figure 21] FIG. 11 is a block diagram illustrating an example of a configuration of an information processing system according to a third embodiment. [Figure 22] FIG. 4 is a diagram illustrating an example of specimen information stored in a specimen information storage unit. [Diagram 23] FIG. 13 is a block diagram illustrating an example of an object recognition processing unit according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] An example of the processing function of an information processing system 1 of the present invention is shown in block diagrams in Figures 1 and 2. The information processing system 1 uses a management terminal 2 and an image information input terminal 3. Figure 1 is a block diagram showing the overall function of the information processing system 1, and Figure 2 is a block diagram showing the function of an object recognition processing unit 213, which will be described later.

[0030] The management terminal 2 is a computer used by an organization such as a company that operates the information processing system 1. The image information input terminal 3 is a terminal for inputting image information including an object to be identified, for example, image information obtained by photographing a display shelf in a store on which the product to be identified is displayed.

[0031] The management terminal 2 and the image information input terminal 3 in the information processing system 1 are realized by using a computer. An example of the hardware configuration of a computer is shown in Fig. 3. The computer has a calculation device 70 such as a CPU that executes the calculation processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or a mouse that can input information, and a communication device 74 that transmits and receives the processing results of the calculation device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0032] In the case where the computer is equipped with a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited to these.

[0033] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display by a predetermined input device (such as a touch panel pen) or a finger.

[0034] In addition to the above devices, the image information input terminal 3 may be equipped with a photographing device such as a camera. As the image information input terminal 3, a portable communication terminal such as a mobile phone, a smartphone, or a tablet computer can also be used.

[0035] The functions of the means in the present invention are only logically distinct, and may be physically or practically the same area. The order of the processes in the means in the present invention may be changed as appropriate. Also, some of the processes may be omitted. For example, the normalization process described later may be omitted. In that case, the process may be performed on image information that has not been normalized.

[0036] The information processing system 1 has a learning processing unit 20, a recognition processing unit 21, and an object information storage unit 22. The learning processing unit 20 has a first learning processing unit 201 and a second learning processing unit 202.

[0037] The first learning processing unit 201 uses the first annotation data to perform a learning process by machine learning, preferably a learning process by an image segmentation method, on image information including an object to be identified, for example, image information obtained by photographing a display shelf displaying the product to be identified. This learning process is a learning process in machine learning, and executes a learning process by image segmentation to create a learning model using, for example, deep learning.

[0038] The first annotation data is data in which the contour of an object that may be the subject of identification is taken as the outer shape, data in which the inside of a closed area formed by the contour is masked, and a tag (label) that classifies the attributes of the contour of the object are associated with each other. The outer shape may be the contour of the object itself, or may be a shape that includes the contour of the object and follows the contour of the object, or a shape that has a width from the contour of the object, such as a shape that shows the contour of the object. In other words, the outer shape of the contour of the object is not limited to a rectangular area, and may be a non-rectangular area that reflects the contour of the object.

[0039] For example, in the case of an object being a product displayed on a display shelf, the first annotation data is data that corresponds to the contour of the object that may be displayed on the display shelf, with the inside of the closed area formed by the contour masked, and a tag (label) that classifies the attributes of the object's contour.

[0040] The first annotation data may not be one for one object, and may be multiple for one object. That is, data obtained by masking the inside of the outline of an object from multiple directions may be associated with attributes, and each may be used as the first annotation data for the object. The attributes are the types of objects. The types of objects include animal types (elephants, tigers, lions, etc.), bird types (chickens, parakeets, peacocks, etc.), insect types (beetles, stag beetles, butterflies, etc.), fish types (tuna, mackerel, sardines, etc.), plant types (trees, flowers, etc.) (cherry blossoms, plum blossoms, lilies, roses, chrysanthemums, tulips, etc.), types of moving objects (vehicles, aircraft, etc.) (automobiles, bicycles, motorcycles, airplanes, helicopters, drones, UAVs (Unmanned Aircraft Vehicles), ships, boats, etc.), as well as various types such as classification of containers of objects and object identification information of objects (JAN codes, etc.). The classification of containers may be by type of container, such as can, bottle, box, pouch container, etc., or may be further subdivided according to use, such as detergent container. In other words, the attribute may indicate how a closed area based on the contour of the object is classified. An example of the first annotation data is shown in Figs. 6 and 7. Figs. 6 and 7 show the case of a displayed commodity as the object, but as mentioned above, the object is not limited to this. The object identification information is not limited to the JAN code, and may be any information that can uniquely identify the object.

[0041] Fig. 6 shows a case where data obtained by masking the outline of a can and the inside of the closed area formed by the outline is associated with "can" as an attribute to generate first annotation data, and Fig. 7 shows a case where data obtained by masking the outline of a refillable shampoo and the inside of the closed area formed by the outline is associated with "pouch container" as an attribute to generate first annotation data. In cases where an object can be identified from the outline of the object itself, object identification information such as a JAN code may be used as an attribute in the first annotation data instead of the container classification.

[0042] A learning process for machine learning is executed using the first annotation data in the first learning processing unit 201, whereby a learning model (first learning model) for identifying the contour area of ​​an object is created from image information of an object that may be a target for identification. For example, a learning model (first learning model) for identifying the contour area of ​​an object is created from image information of an image of a display shelf. Note that the first learning model may be a learning model capable of identifying the attributes of an object in addition to the contour area of ​​the object.

[0043] The second learning processing unit 202 executes a machine learning learning process using the second annotation data to create a learning model (second learning model) for identifying object identification information of an object in a given image information, preferably image information of an outline region described later, from the area. As the learning process at this time, it is preferable to execute a learning process by an image classification method, but a method such as object detection, image classification and object location identification (Image Classification Localization) may also be used. Note that the second learning model may be a learning model in which attributes corresponding to the outline region, in addition to image information of the outline region, can be input as input values ​​of the second learning model.

[0044] The second annotation data is data in which image information of an object that may be identified is associated with the object identification information of the object as a tag (label). For example, when the object is a display commodity, the data is data in which image information of the commodity that may be displayed on the shelf is associated with the object identification information of the object as a tag (label). When the object is an animal, the data is data in which image information of the animal that may be identified is associated with identification information such as the animal's name (scientific name, common name, etc.) as a tag (label) of the object identification information. An example of the second annotation data is shown in FIG. 8 and FIG. 9. Although FIG. 8 and FIG. 9 show the case of a display commodity as the object, as described above, the second annotation data is not limited to this. As with the first annotation data, the second annotation data may be multiple, not limited to one for one object. That is, the object may be photographed from multiple directions, and the image information of the object from each direction may be associated with the object identification information to form the second annotation data. In addition, in the case where attributes corresponding to the external area can also be input as input values ​​in the second learning model, the second annotation data may be data that associates image information of an object that may be the subject of identification with the attributes of the object and the object identification information of the object as a tag (label).

[0045] Fig. 8 shows a case where image information of a can is associated with object identification information to generate second annotation data, and Fig. 9 shows a case where image information of a shampoo refill is associated with object identification information to generate second annotation data. The second annotation data in Fig. 8 corresponds to the first annotation data in Fig. 6, and the second annotation data in Fig. 8 corresponds to the second annotation data in Fig. 7.

[0046] The recognition processing unit 21 includes an image information input reception processing unit 210 , an image information storage unit 211 , an image information alignment processing unit 212 , and an object recognition processing unit 213 .

[0047] The image information input reception processing unit 210 receives input of image information (photographed image information) including an object to be identified, photographed by the image information input terminal 3, and stores it in the image information storage unit 211 described later. It is preferable to receive input of the photographed image information, as well as the photographed date and time, image information identification information for identifying the image information, and the like, from the image information input terminal 3. An example of photographed image information in the case where the object is a display commodity displayed on a display shelf is shown in Figs. 10 and 11. Figs. 10 and 11 show photographed image information in which the display shelf has three shelves and the object is displayed on the shelves. Although the processing is not particularly specified in the present invention, display shelves and shelves are often long in the horizontal direction. Therefore, in the processing, the image may be divided into certain widths and used as the processing target for each processing.

[0048] The image information storage unit 211 stores the captured image information, the capture date and time, the image information identification information, etc. received from the image information input terminal 3 in association with each other. The captured image information may be any image information to be processed by the present invention. In general, when simply capturing an image, it is difficult to capture an image of the subject facing the camera in a normal position. Therefore, it is advisable to execute a correction process to correct the image to a normal position, such as a keystone correction process. When multiple images of a single subject are captured, the image information includes image information in which the images are combined into one piece of image information. The captured image information also includes image information after a distortion correction process has been executed.

[0049] The image information orthogonalization processing unit 212 performs a process (orthogonalization process) for correcting the captured image information stored in the image information storage unit 211 so that the captured object is in a right-facing state, for example, a keystone correction process, to generate right-facing image information. The keystone correction process is a correction process performed so that the object captured in the captured image information is in a right-facing state. For example, in the case where the object is a displayed commodity, the correction process is performed so that the shelves on which the displayed commodity is displayed are horizontal. Orthogonalization is a process for transforming the image information so that the optical axis of the lens of the image capture device is aligned along the perpendicular direction of the plane of the captured object, so that the image information is the same as when the image is captured from a sufficiently distant location, for example, but not limited to, a keystone correction process.

[0050] The keystone correction process executed by the image information alignment processor 212 accepts input of designation of four vertices in the captured image information, and executes the keystone correction process using each of the vertices. The four vertices to be designated may be the four vertices of a shelf of the display shelf or the four vertices of the shelf position of the display shelf when the object is a commodity displayed on a display shelf. Alternatively, the four vertices may be the four vertices of a group of two or three shelf shelves. Any four points can be designated as the four vertices. Fig. 12 shows an example of captured image information (aligned image information) obtained by alignment of the captured image information of Fig. 10, and Fig. 13 shows an example of the captured image information of Fig. 11.

[0051] The object recognition processing unit 213 executes a process of recognizing an object appearing in image information, preferably captured image information or upright image information.

[0052] The object recognition processing unit 213 has an external shape specification processing unit 2131 and an object identification processing unit 2132 .

[0053] The outer shape specification processing unit 2131 specifies the area of ​​the outer shape of an object (outer shape area) shown in the normal image information.

[0054] The outer shape specification processing unit 2131 inputs the photographed image information and the upright image information as input values ​​to the learning model (first learning model) trained in the first learning processing unit 201, and specifies the outer shape region from the input image information. That is, the outer shape specification processing unit 2131 inputs the region to be processed, for example, the upright image information, to the learning model trained in the first learning processing unit 201 (a learning model in which weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized), and specifies the outer shape region based on the output value. Outer shape identification information that identifies the outer shape region is assigned to the specified outer shape region, and the region is stored in the object identification information storage unit 22 together with position information in the photographed image information and the upright image information (for example, coordinates in the image information).

[0055] The output value output by the contour identification processing unit 2131 may include the attribute tag to which the contour region corresponds and its reliability (probability). At this time, the attribute tag with the highest reliability is identified as the tag of the contour region, and if the reliability is equal to or lower than a certain threshold, the identified contour region is identified as a predetermined object such as an unknown object (an object that cannot be identified due to reasons such as not having been learned).

[0056] The object identification processing unit 2132 inputs the image information of the outline region as an input value to a learning model (second learning model) trained in the second learning processing unit 202, and identifies the identification information of the object in the region from the input image information. That is, the object identification processing unit 2132 inputs the image information of the outline region to be processed to the learning model trained in the second learning processing unit 202 (a learning model in which weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized), and identifies the object identification information of the object in the outline region based on the output value. At this time, in addition to the image information of the outline region as an input value, tag information of the attribute corresponding to the outline region specified by the outline specification processing unit 2131 may be input as an input value. At this time, one object identification information may be identified for the outline region, or multiple object identification information may be identified. When multiple object identification information is identified, candidates for the object identification information to be identified are output.

[0057] The reliability (probability) of the object identification information identified by the outline region is also output as an output value from the object identification processing unit 2132. At this time, the object identification information with the highest reliability is identified as the object identification information of the outline region, but if the highest reliability is equal to or less than a certain threshold, the input outline region is identified as an unknown object.

[0058] The object recognition processing unit 213 may execute the processing of the external shape specification processing unit 2131 and the object identification processing unit 2132 together by deep learning or the like.

[0059] The object identification information storage unit 22 stores information indicating object identification information of objects shown in the photographed image information and the upright image information. For example, the object identification information is stored in the object identification information storage unit 22 in association with the photographed date and time information, image information identification information of the photographed image information, image identification information of the upright image information, and external shape identification information for identifying the external shape. EXAMPLES

[0060] Next, an example of the processing of the information processing system 1 of the present invention will be described with reference to the flowcharts of FIGS.

[0061] First, a learning process for learning a learning model used in the recognition processing unit 21 of the information processing system 1 of the present invention will be described with reference to the flowchart of FIG.

[0062] First annotation data is created as teacher data for the learning model in the first learning processing unit 201 (S100). The first annotation data is image data in which the outline of an object that may be the subject of identification is formed and the closed area inside the outline is masked. Attributes are associated with this image data as tags to create the first annotation data.

[0063] Similarly, second annotation data is created as teacher data for the learning model in the second learning processing unit 202 (S110). The second annotation data is created by associating an image of an object that may be the subject of identification with object identification information of the object as a tag. The image information of the object in this case should correspond to the first annotation data, and should be image information having the contour of the object as the external shape.

[0064] Then, the created first annotation data is input as teacher data, and a learning process for machine learning is executed in the first learning processing unit 201 to create a learning model (first learning model) for identifying the contour area of ​​an object (S120).

[0065] In addition, the created second annotation data is input as training data, and a learning process for machine learning is executed in the second learning processing unit 202 to create a learning model (second learning model) for identifying object identification information of an object in the image information, preferably the outline area, from that area (S130).

[0066] By executing the above-mentioned processing, each learning model can be created.

[0067] Next, a recognition process for identifying object identification information of an object captured in image information obtained by photographing the object to be identified will be described with reference to the flowchart of FIG.

[0068] The photographed image information of the object to be identified is input from the image information input terminal 3, and the input is accepted by the image information input acceptance processing unit 210 of the management terminal 2 (S200). The image information input acceptance processing unit 210 also accepts input of the photographed date and time, and image information identification information of the photographed image information. Then, the image information input acceptance processing unit 210 stores the photographed image information, the photographed date and time, and the image information identification information of the photographed image information that have been accepted as input in the image information storage unit 211 in association with each other.

[0069] When a specified operation input is received in the management terminal 2, the image information alignment processing unit 212 extracts the captured image information to be stored in the image information storage unit 211, receives input of four points for performing alignment processing such as keystone correction processing, and executes the alignment processing (S210).

[0070] Then, the management terminal 2 receives a predetermined operation input for the upright image information, thereby executing a process of identifying the outer shape (S220). That is, the outer shape identification processing unit 2131 inputs a part or the whole area in the captured image information or the upright image information as an input value to a learning model (first learning model) trained by the first learning processing unit 201, and identifies the outer shape area from the input image information.

[0071] As described above, the contour regions of each object captured in the upright image information are specified, and the attributes (tags) of each object are identified. If the reliability of the identified tag is equal to or lower than a certain threshold (S230), the object in that contour region is identified as an unknown object (S240).

[0072] On the other hand, if the reliability of the identified tag is greater than a certain threshold (S230), the object identification processing unit 2132 inputs the image information of the contour region as an input value to the learning model (second learning model) trained in the second learning processing unit 202, and identifies the object identification information of the object appearing in the contour region (S250). If the reliability of the identified object identification information is equal to or less than a certain threshold (S260), the object in the contour region is identified as an unknown object (S240).

[0073] On the other hand, if the reliability of the identified object identification information is greater than a certain threshold (S260), the identified object identification information is stored in the object identification information storage unit 22 in correspondence with the shooting date and time, the image information identification information of the captured image information, the image information identification information of the upright image information, and the external shape identification information.

[0074] In addition, an input regarding a process for correcting the identified object identification information may be accepted.

[0075] By carrying out the above-mentioned processing, it is possible to identify the object identification information of the object captured in the captured image information. In addition, unlike the conventional system, the outer shape region is not a rectangular region, but the object is identified along the outer shape of the object's outline, so that unnecessary information contained in the outer shape region, such as noise from other objects, is eliminated, improving the recognition accuracy.

[0076] In addition, when object identification information is used as an attribute in the first annotation data (when an object can be identified from its external shape), the image information is input to the first learning model to identify the external shape region, thereby identifying the object identification information of the object. In this case, it is not necessary to execute the processing by the second learning processing unit 202 and the object identification processing unit 2132, and when the external shape region is identified by the external shape identification processing unit 2131, the object identification information of the object depicted in the external shape region may be identified by the object identification information of the attribute as an output result by the first learning model. EXAMPLES

[0077] In the information processing system 1 of the first embodiment described above, in the case of display products displayed on a display shelf as objects, the processing may be further executed for each shelf area (shelf area). An example of the processing function of the information processing system 1 in this case is shown in a block diagram in FIG.

[0078] In the information processing system 1 of the second embodiment, the recognition processing unit 21 further includes a shelf level identification processing unit 214 .

[0079] The shelf level identification processing unit 214 identifies the area of ​​the shelf level (shelf level area) of the display shelf where an object may be placed, from the upright image information obtained by performing keystone correction processing on the captured image information in the image information orientation processing unit 212. The captured image information and the upright image information show the display shelf, but the display shelf has a shelf level area where the object is displayed. Therefore, the shelf level area is identified from the upright image information. The shelf level area may be identified by the operator of the management terminal 2 manually specifying the shelf level area, which is then accepted by the shelf level identification processing unit 214, or the shelf level area may be identified automatically from the second time onwards, based on the information on the shelf level area that was manually input the first time.

[0080] Fig. 15 shows a schematic diagram of a state where an input for designating a shelf area is received for upright image information obtained by uprighting image information captured on a display shelf on which objects such as beverage cans are displayed. Fig. 16 also shows a schematic diagram of a state where an input for designating a shelf area is received for upright image information obtained by uprighting image information captured on a display shelf on which objects such as toothbrushes are hung and displayed.

[0081] When identifying the shelf level area, the shelf level identification processing unit 214 may identify the shelf level area using deep learning. In this case, the above-mentioned upright image information may be input to a learning model in which weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized, and the shelf level area may be identified based on the output value. In addition, the learning model may be one in which the shelf level area is given as correct answer data to various upright image information.

[0082] The shelf level area specified by the shelf level specification processing unit 214 specifies the image information as shelf level area image information. The shelf level specification processing unit 214 may actually cut out the image information, or may not actually cut out the image information, but may virtually cut out the image information of the area by specifying the coordinates or the like. If the display shelf has multiple shelf levels, each of them is cut out as shelf level area image information. The coordinates indicating the shelf level area are the coordinates of vertices for specifying the area, and may be, for example, four points in the upright image information, such as the coordinates of two points, the upper right and the lower left, or the upper left and the lower right. In addition, the coordinates are relative coordinates based on a predetermined location in the image information (for example, the upper left vertex of the display shelf) such as the display shelf in the upright image information. In this specification, cutting out the image information may actually cut out the image information, similar to the cutting out in the shelf level specification processing unit 214, or may not actually cut out the image information, but may virtually cut out the image information of the area by specifying the coordinates or the like.

[0083] The outer shape specification processing unit 2131 and the object identification processing unit 2132 in the object recognition processing unit 213 in the information processing system 1 of this embodiment may execute the following processing in addition to the processing in the first embodiment.

[0084] The outer shape specification processing unit 2131 specifies an outer shape area (outer shape area) for each shelf in the shelf area in the normal image information. The outer shape is an area in which an object is placed, regardless of whether the object is placed there or not. The size of the outer shape area is the same or approximately the same as the size of the object to be placed there.

[0085] The outer shape specification processing unit 2131 inputs the photographed image information, the upright image information, or the image information of the shelf area as input values ​​to the learning model (first learning model) trained in the first learning processing unit 201, and specifies the outer shape area from the input image information. That is, the outer shape specification processing unit 2131 inputs the image information of the area to be processed, for example, the shelf area, to the learning model trained in the first learning processing unit 201 (a learning model in which the weighting coefficients between the neurons of each layer of a neural network consisting of many intermediate layers are optimized), and specifies the outer shape area based on the output value. Outer shape identification information that identifies the outer shape area is assigned to the specified outer shape area, and the outer shape area is stored in the object identification information storage unit 22 together with position information (for example, coordinates in the image information) in the photographed image information, the upright image information, or the image information of the shelf area.

[0086] The output value output by the contour identification processing unit 2131 includes the attribute tag to which the contour region corresponds, for example, can, bottle, pouch container, etc., along with its reliability (probability). At this time, the attribute tag with the highest reliability is identified as the tag of the contour region, and if the reliability is equal to or lower than a certain threshold, the identified contour region is identified as an unknown object.

[0087] Fig. 17 shows an example of a case where an outer shape region is specified from image information of a shelf level region. Fig. 17(a) is an example of image information of a shelf level region input to a learning model trained by the first learning processing unit 201, and Fig. 17(b) is a diagram showing an example of a state where an outer shape region is specified using the learning model in the image information of the shelf level region that is used as an input value in Fig. 17(a). Fig. 17(b) shows a state where an outer shape region is specified in the shelf level region superimposed, but the image information of the specified outer shape region may be cut out and output as it is.

[0088] The object identification processing unit 2132 inputs the image information of the outline region as an input value to a learning model (second learning model) trained in the second learning processing unit 202, and identifies the identification information of the object in the area from the input image information. That is, the object identification processing unit 2132 inputs the image information of the outline region to be processed to the learning model trained in the second learning processing unit 202 (a learning model in which weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized), and identifies the object identification information of the object in the outline region based on the output value. At this time, in addition to the image information of the outline region, the tag of the attribute corresponding to the outline region identified by the outline identification processing unit 2131, for example, information on a can, bottle, or pouch container, may be input as an input value. At this time, one object identification information may be identified for the outline region, or multiple object identification information may be identified. When multiple object identification information is identified, candidates for the object identification information to be identified are output.

[0089] The reliability (probability) of the object identification information identified by the outline region is also output as an output value from the object identification processing unit 2132. At this time, the object identification information with the highest reliability is identified as the object identification information of the outline region, but if the highest reliability is equal to or less than a certain threshold, the input outline region is identified as an unknown object.

[0090] As in the first embodiment, the object recognition processing unit 213 may execute the processes of the external shape specification processing unit 2131 and the object identification processing unit 2132 together by deep learning or the like.

[0091] The object identification information storage unit 22 stores information indicating object identification information of objects displayed on the outer shape of each shelf of the display shelf. For example, the object identification information is stored in the object identification information storage unit 22 in association with the object identification information, in association with the shooting date and time information, the store information, the image information identification information of the photographed image information, the image identification information of the upright image information, and the outer shape identification information for identifying the outer shape.

[0092] Next, an example of the processing process of the information processing system 1 in this embodiment will be described with reference to the flowcharts of Fig. 4 and Fig. 18. In the following description of this embodiment, a case will be described in which object identification information of objects on display is identified from captured image information.

[0093] The learning process (FIG. 4) for learning the learning model used in the recognition processing unit 21 of the information processing system 1 in this embodiment is similar to that in the first embodiment, and therefore a description thereof will be omitted.

[0094] The recognition process for identifying object identification information of objects displayed on a display shelf from image information obtained by photographing the display shelf will be described with reference to the flowchart of FIG.

[0095] Captured image information of store display shelves is input from the image information input terminal 3, and the input is accepted by the image information input acceptance processing unit 210 of the management terminal 2 (S300). An example of captured image information is shown in Fig. 19. Also, input of the capture date and time, store identification information, and image information identification information of the captured image information is accepted. Then, the image information input acceptance processing unit 210 stores the captured image information, capture date and time, store identification information, and image information identification information of the captured image information that have been accepted as input in the image information storage unit 211 in association with each other.

[0096] When a predetermined operation input is received in the management terminal 2, the orthogonal image information orthogonalization processing unit 212 extracts the photographed image information stored in the image information storage unit 211, receives input of four shelf positions (positions of display shelves) that are vertices for performing orthogonalization processing such as keystone correction processing, and executes the orthogonalization processing (S310). An example of the photographed image information (orthogonal image information) that has been subjected to the orthogonalization processing in this manner is shown in FIG.

[0097] Then, by receiving a predetermined operation input in the management terminal 2 for the normal placement image information, the shelf level identification processing unit 214 identifies the shelf level position area (S320). That is, input of the shelf level area in the normal placement image information is received. Figures 15 and 16 are diagrams showing a state in which the shelf level area has been identified from the normal placement image information.

[0098] When the shelf level area is identified in the above manner, image information of the shelf level area is extracted from the upright image information. Then, a process of identifying an outline is executed for each shelf level in the shelf level area image information (S330). That is, the outline identification processing unit 2131 inputs the image information of the shelf level area as an input value to the learning model (first learning model) learned by the first learning processing unit 201, and identifies the outline area from the input image information.

[0099] As described above, the contour regions of each object captured in the upright image information are specified, and the attributes (tags) of each are identified. If the reliability of the identified tag is equal to or lower than a certain threshold (S340), the object in that contour region is identified as an unknown object (S350).

[0100] On the other hand, if the reliability of the identified tag is greater than a certain threshold (S340), the object identification processing unit 2132 inputs the image information of the contour region as an input value to the learning model (second learning model) trained in the second learning processing unit 202, and identifies the object identification information of the object appearing in the contour region (S360). If the reliability of the identified object identification information is equal to or less than a certain threshold (S370), the object in the contour region is identified as an unknown object (S350).

[0101] On the other hand, if the reliability of the identified object identification information is greater than a certain threshold (S370), the identified object identification information is stored in the object identification information storage unit 22 in correspondence with the shooting date and time, store identification information, image information identification information of the photographed image information, image information identification information of the upright image information, and external shape identification information.

[0102] Note that it is not always possible to identify object identification information for all outer shape regions. Therefore, for outer shape regions that cannot be identified, input of object identification information is accepted, and the accepted input object identification information is stored in the object identification information storage unit 22 in association with the photographing date and time, the store identification information, the image information identification information of the photographed image information, the image information identification information of the upright image information, and the outer shape identification information. Similarly, input may also be accepted for the correction process of the identified object identification information.

[0103] By carrying out the above-mentioned processing, it is possible to identify the object identification information of the object displayed on the shelf of the display shelf shown in the photographed image information. In addition, unlike the conventional system, the outer shape region is not a rectangular region, but the object is identified along the outer shape of the object's outline, so that unnecessary information included in the outer shape region, such as noise from other objects, is eliminated, improving the recognition accuracy.

[0104] In addition, when object identification information is used as an attribute in the first annotation data (when an object can be identified from its external shape), the image information is input to the first learning model to identify the external shape region, thereby identifying the object identification information of the object. In this case, it is not necessary to execute the processing by the second learning processing unit 202 and the object identification processing unit 2132, and when the external shape region is identified by the external shape identification processing unit 2131, the object identification information of the object depicted in the external shape region may be identified by the object identification information of the attribute as an output result by the first learning model. EXAMPLES

[0105] In the first and second embodiments, the configurations using machine learning in two processes, that is, specifying the outer shape region and identifying the object identification information from the outer shape region, have been described, but the process of identifying the object identification information from the outer shape region may use image matching processing. An example of the configuration of the information processing system 1 in this case is shown in FIG. 21. Note that FIG. 21 shows a case based on the configuration of the information processing system 1 in the second embodiment, but it goes without saying that the information processing system 1 may be based on the first embodiment.

[0106] In the information processing system 1 of this embodiment, the learning processing unit 20 does not need to include the second learning processing unit 202. Moreover, the recognition processing unit 21 includes a sample information storage unit 215 that stores sample information used in the image matching process.

[0107] The specimen information storage unit 215 stores specimen information for identifying which object is displayed on the shelf of the display shelf shown in the image information. The specimen information is image information obtained by photographing an object that may be displayed on the display shelf from multiple angles, such as up and down, left and right, and diagonally. FIG. 22 shows an example of specimen information stored in the specimen information storage unit 215. FIG. 22 shows a case where canned beer is photographed from various angles as specimen information, but is not limited to canned beer. The specimen information storage unit 215 stores the specimen information in association with object identification information.

[0108] The specimen information storage unit 215 may store information necessary for calculating similarity, such as paired information of image features and their positions, extracted from the specimen information together with or instead of the specimen information. The specimen information includes information necessary for calculating similarity. In this case, the object recognition processing unit 213 does not need to calculate image features for the specimen information every time when performing matching processing between image information of the outline region described below and the specimen information, and the calculation time can be reduced.

[0109] The specimen information stored in the specimen information storage unit may be image information of an object obtained by masking the outer shape of the object's contour in the first annotation data used in the learning process of the first learning processing unit 201. That is, when creating the first annotation data, image information of the object photographed from one or more directions or its image feature quantity is used as specimen information. Then, from the image information of the photographed object, the contour is used as the outer shape, and the inside of the closed region is masked, and the attribute is tagged to create the first annotation data. By such processing, the specimen information and the first annotation data can be created together.

[0110] The object identification processing unit 2132 in this embodiment executes a matching process between the image information of the contour area specified by the contour specification processing unit 2131 and the specimen information stored in the specimen information storage unit 215, and identifies the object identification information of the object displayed in the contour area. That is, the image information in the contour area of ​​a certain shelf (the contour identification information of this contour area is X) and each specimen information stored in the specimen information storage unit are used to calculate the image feature amount and obtain pairs of feature points, thereby determining the similarity. Then, the most similar specimen information is specified, and if the similarity at that time is equal to or greater than a predetermined threshold, the object identification information corresponding to that specimen information is identified based on the specimen information storage unit 215. Then, the identified object identification information is set as the object identification information of the object displayed in the contour of the contour identification information X. Note that, for a contour determined to be not similar to any specimen information, information indicating that the contour identification information is "empty" (information indicating that there is no object) is added to the contour identification information. The object identification processing unit 2132 stores the identified object identification information or information indicating that the object is “empty” in the object identification information storage unit 22 in correspondence with the shooting date and time, store identification information, image information identification information of the captured image information, image information identification information of the upright image information, and external shape identification information.

[0111] Specifically, as an example, the object identification processing unit 2132 executes the following process: First, it determines the similarity between the image information formed by the coordinates of the outer region to be processed and the specimen information stored in the specimen information storage unit 215, identifies the object identification information corresponding to the specimen information with the highest similarity, and if the identified similarity is equal to or greater than a predetermined threshold, identifies it as the object identification information of the object displayed in the outer region formed by the coordinates.

[0112] Here, to determine the similarity between the image information of the outer shape and the specimen information, the following process is carried out: First, in the process before the identification process of the object identification information in the object identification processing unit 2132, the image information of the outer shape region on the shelf of the upright image information and the specimen information have the same orientation (not turned sideways or upside down), and the sizes of each piece of image information are roughly the same (if the sizes of the image information differ beyond a predetermined range, the sizes of each piece of image information are adjusted so that they are within a predetermined range before determining the similarity).

[0113] To determine the similarity between the image information of the contour region and the specimen information, the object identification processing unit 2132 extracts feature points based on the image feature amount (e.g., local feature amount) of the image information of the contour and feature points based on the image feature amount (e.g., local feature amount) of the specimen information. Then, the most similar pair of feature points of the image information of the contour and feature points of the specimen information is detected, and the difference in coordinates of corresponding points between them is calculated. Then, the average value of the differences is calculated. The average value of the differences indicates the overall average amount of movement between the image information of the contour region and the specimen information. Then, the coordinate differences of all pairs of feature points are compared with the average coordinate difference, and pairs with a large degree of deviation are excluded. Then, the similarity is ranked according to the number of remaining corresponding points.

[0114] The above method allows the similarity between the image information of the outline region and the specimen information to be calculated. In order to improve the accuracy, the EMD (Earth Movers Distance) between the color histograms may be calculated and used as a measure of similarity. This allows a comparison of similarities that are relatively resistant to environmental changes such as brightness information of the captured image information to be performed, allowing for highly accurate identification.

[0115] Another method for determining similarity is to find the EMD between the signatures (a set of image features and weights) of the image information of each contour region, and use this as a measure of similarity. As the image feature of the signature, for example, the frequency distribution in the HSV color space of the image information of the contour region can be found, and grouped according to hue and saturation to determine the image feature according to the number of features and the area in the HSV color space. Grouping according to hue and saturation is done to reduce the dependency on brightness so that it is not significantly affected by the shooting conditions.

[0116] Also, in order to speed up processing, similarity such as L2 distance between image feature quantities such as color correlograms and color histograms of image information in an appropriate color space can be used instead of the signature and EMD.

[0117] The determination of similarity is not limited to the above. The identified object identification information is stored in the object identification information storage unit 22 in association with the photographing date and time information, the store information, the image information identification information of the photographed image information, the image identification information of the upright image information, and the external shape identification information.

[0118] For an outline for which the object identification information could not be identified, information indicating that the outline region is "empty" (information indicating that the object is not missing or the like) is stored in the object identification information storage unit 22.

[0119] As described above, even if image matching processing is used to identify an object from image information of its outline region, the image matching processing can be performed with high accuracy because the outline region is not a rectangular region. EXAMPLES

[0120] As a modification of the above-mentioned embodiment 2 or 3, a shelf level comparison processing unit 2143 is provided to detect changes in shelf levels, and if there is no change in shelf level, the previous recognition result can be used as is. An example of the object recognition processing unit 213 in this case is shown in FIG.

[0121] The shelf level comparison processing unit 2133 determines that the object identification information of each external shape of the shelf level or "empty" is the same if the similarity is high based on the image information of the shelf level area in the previous (N-1th) upright image information and the image information of the shelf level area in the current (Nth) upright image information. As described above, this similarity determination process may be a determination of similarity based on the image feature amount of the image information of the shelf level area in the previous (N-1th) upright image information and the image information of the shelf level area in the current (Nth) upright image information, or may use EMD between color histograms. In addition, it is not limited to these. Then, instead of performing a specification process for each external shape unit in the object identification processing unit 2132, the object identification processing unit 2132 stores the object identification information of each external shape of the shelf level in the Nth upright image information in the object identification information storage unit 22 as the same as the object identification information of each external shape of the same shelf level in the N-1th time. This makes it possible to omit processing for shelves that hardly change, such as shelves where there is little movement of objects or shelves that are managed in extremely short cycles. EXAMPLES

[0122] The processes of the above-mentioned first to fourth embodiments can be combined as appropriate. The order of each process is not limited to that described in the specification of the present invention, and can be changed as appropriate to the extent that the object is achieved. The process in the object recognition processing unit 213 is performed on the normalized image information obtained by performing the normalization process on the photographed image information, but it may be performed on the photographed image information. In that case, the normalized image information may be read as the photographed image information.

[0123] Furthermore, in the second and fourth embodiments, instead of specifying the shelf level area in the recognition processing unit 21 and then specifying the outer shape area described below from the shelf level area, it is also possible to configure the system so that the outer shape area described below is specified from the entire captured image information, upright image information, or image information of the shelf level area without specifying the shelf level area. In this case, the shelf level identification processing unit 214 does not need to be provided, and the system may be configured so as not to execute this processing. EXAMPLES

[0124] In the above-mentioned embodiments, the display shelves of convenience stores and supermarkets are described as examples, but the present invention is not limited thereto, and can be applied to medicines (objects) displayed on shelves (medicine shelves) that display medicines in a pharmacy. Similarly, the present invention can be applied to objects displayed on shelves in a warehouse.

[0125] The object may be something other than the displayed goods on the display shelf. For example, the object may be various objects to be identified, such as animals, birds, insects, fish, plants, moving objects, etc. In this way, image information obtained by photographing various objects may be applied to image information other than the displayed goods on the display shelf. [Industrial Applicability]

[0126] By using the information processing system 1 of the present invention, it is possible to improve the accuracy of identifying objects when identifying displayed objects from image information. [Explanation of symbols]

[0127] 1: Information processing system 2: Management terminal 3: Image information input terminal 20: Learning processing unit 21: Recognition processing section 22: Object identification information storage unit 70: Arithmetic device 71:Storage device 72:Display device 73: Input device 74:Communication equipment 201: First learning processing unit 202: Second learning processing unit 210: Image information input reception processing unit 211: Image information storage unit 212: Image information alignment processing unit 213: Object recognition processing unit 214: Shelf specific processing section 2131: Outline specific processing section 2132: Object identification processing unit 2133: Shelf comparison processing unit

Claims

1. An information processing system for identifying an object appearing in image information, comprising: a first learning processing unit that performs machine learning using first annotation data in which data obtained by masking the inside of a closed region of the outline of an object is associated with an attribute, to create a first learning model; a second learning processing unit that performs machine learning using second annotation data in which image information of an object is associated with object identification information to create a second learning model; a recognition processing unit that identifies object identification information of an object shown in the image information; It has The recognition processing unit is Using the image information and the first learning model, an outline of an object shown in the image information is identified as an outline region, and a reliability of an attribute corresponding to the identified outline region is output; If the reliability of the attribute corresponding to the specified outline region is not equal to or less than a predetermined threshold, Using the image information of the specified outline region and the second learning model, identify one or more object identification information of the object appearing in the specified outline region, and output a reliability of the identified object identification information; When the reliability of the object identification information of the identified object is not equal to or less than a predetermined threshold, one or more object identification information of the object depicted in the outline region is set as the identified object identification information; If the reliability of the object identification information of the identified object is equal to or less than a predetermined threshold, the object identification information of the object depicted in the outline region is identified as an unknown object; If the reliability of the attribute corresponding to the identified contour region is equal to or less than a predetermined threshold, the object depicted in the identified contour region is identified as an unknown object. An information processing system comprising:

2. The first learning processing unit includes: performing machine learning by image segmentation using the first annotation data to create the first learning model; 2. The information processing system according to claim 1 .

3. The recognition processing unit is performing an image matching process between the image information of the specified outline region and specimen information of the object stored in a specimen information storage unit, thereby identifying object identification information of the object depicted in the outline region; 3. The information processing system according to claim 1 or 2.

4. An information processing system for identifying an object appearing in image information, comprising: a first learning processing unit that performs machine learning using first annotation data in which data obtained by masking the inside of a closed region of the outline of an object is associated with an attribute, to create a first learning model; a second learning processing unit that performs machine learning using second annotation data in which image information of an object is associated with object identification information to create a second learning model, Using the image information and the first learning model, a contour of an object shown in the image information is identified as a contour region, and a reliability of a corresponding attribute of the identified contour region is output; If the reliability of the attribute corresponding to the specified outline region is not equal to or less than a predetermined threshold, Using the image information of the specified outline region and the second learning model, one or more pieces of object identification information of an object appearing in the specified outline region are identified, and a reliability of the identified object identification information is output; When the reliability of the object identification information of the identified object is not equal to or less than a predetermined threshold, one or more object identification information of the object depicted in the outline region is regarded as the identified object identification information; When the reliability of the object identification information of the identified object is equal to or less than a predetermined threshold, the object identification information of the object depicted in the outline region is identified as an unknown object; If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object shown in the specified outline region is identified as an unknown object. An information processing system comprising:

5. An information processing system for identifying an object appearing in image information, comprising: an image information input reception processing unit that receives input of image information obtained by photographing an object; an object recognition processing unit that identifies object identification information of an object captured in the image information from the received input image information or image information obtained by orthogonally orthogonally orthogonally orthogonally orthogonally; It has The object recognition processing unit includes: identifying an outline of the object as an outline region using a first learning model created by machine learning using first annotation data in which data obtained by masking the inside of a closed region of the outline of the object is associated with an attribute, and using the image information received as the input or image information obtained by orthogonally orthogonally orthogonally orthogonally orthogonally; Outputting a reliability of the attribute corresponding to the specified outline region; If the reliability of the attribute corresponding to the specified outline region is not equal to or less than a predetermined threshold, using a second learning model created by machine learning using second annotation data in which image information of an object and object identification information are associated with each other, and the image information of the specified outer region, to identify one or more object identification information of an object depicted in the specified outer region; Outputting a reliability of the identified object identification information; When the reliability of the object identification information of the identified object is not equal to or less than a predetermined threshold, one or more object identification information of the object depicted in the outline region is set as the identified object identification information; If the reliability of the object identification information of the identified object is equal to or less than a predetermined threshold, the object identification information of the object depicted in the outline region is identified as an unknown object; If the reliability of the attribute corresponding to the identified contour region is equal to or less than a predetermined threshold, the object depicted in the identified contour region is identified as an unknown object. An information processing system comprising:

6. Computer, a first learning processing unit that performs machine learning using first annotation data in which data obtained by masking the inside of a closed region of the outline of an object is associated with attributes, to create a first learning model; a second learning processing unit that performs machine learning using second annotation data in which image information of an object is associated with object identification information to create a second learning model; a recognition processing unit that identifies object identification information of an object shown in the image information; An information processing program that functions as The recognition processing unit is Using the image information and the first learning model, an outline of an object shown in the image information is identified as an outline region, and a reliability of an attribute corresponding to the identified outline region is output; If the reliability of the attribute corresponding to the specified outline region is not equal to or less than a predetermined threshold, Using the image information of the specified outline region and the second learning model, identify one or more object identification information of the object appearing in the specified outline region, and output a reliability of the identified object identification information; When the reliability of the object identification information of the identified object is not equal to or less than a predetermined threshold, one or more object identification information of the object depicted in the outline region is set as the identified object identification information; If the reliability of the object identification information of the identified object is equal to or less than a predetermined threshold, the object identification information of the object depicted in the outline region is identified as an unknown object; If the reliability of the attribute corresponding to the identified contour region is equal to or less than a predetermined threshold, the object depicted in the identified contour region is identified as an unknown object.

2. An information processing program comprising:

7. Computer, a first learning processing unit that performs machine learning using first annotation data in which data obtained by masking the inside of a closed region of the outline of an object is associated with attributes, to create a first learning model; a second learning processing unit that performs machine learning using second annotation data in which image information of an object is associated with object identification information to create a second learning model; An information processing program that functions as Using image information and the first learning model, a contour of an object shown in the image information is identified as a contour region, and a reliability for a corresponding attribute of the identified contour region is output; If the reliability of the attribute corresponding to the specified outline region is not equal to or less than a predetermined threshold, Using the image information of the specified outline region and the second learning model, one or more pieces of object identification information of an object appearing in the specified outline region are identified, and a reliability of the identified object identification information is output; When the reliability of the object identification information of the identified object is not equal to or less than a predetermined threshold, one or more object identification information of the object depicted in the outline region is regarded as the identified object identification information; When the reliability of the object identification information of the identified object is equal to or less than a predetermined threshold, the object identification information of the object depicted in the outline region is identified as an unknown object; If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object shown in the specified outline region is identified as an unknown object.

2. An information processing program comprising:

8. Computer, an image information input reception processing unit that receives input of image information obtained by photographing an object; an object recognition processing unit that identifies object identification information of an object shown in the image information from the received input image information or image information obtained by orthogonalizing the image information; An information processing program that functions as The object recognition processing unit includes: identifying an outline of the object as an outline region using a first learning model created by machine learning using first annotation data in which data obtained by masking the inside of a closed region of the outline of the object is associated with an attribute, and using the image information received as the input or image information obtained by orthogonally orthogonally orthogonally orthogonally orthogonally; Outputting a reliability of the attribute corresponding to the specified outline region; If the reliability of the attribute corresponding to the specified outline region is not equal to or less than a predetermined threshold, using a second learning model created by machine learning using second annotation data in which image information of an object and object identification information are associated with each other, and the image information of the specified outer shape region, to identify one or more object identification information of an object appearing in the specified outer shape region, and outputting a reliability of the identified object identification information; When the reliability of the object identification information of the identified object is not equal to or less than a predetermined threshold, one or more object identification information of the object depicted in the outline region is set as the identified object identification information; If the reliability of the object identification information of the identified object is equal to or less than a predetermined threshold, the object identification information of the object depicted in the outline region is identified as an unknown object; If the reliability of the attribute corresponding to the identified contour region is equal to or less than a predetermined threshold, the object depicted in the identified contour region is identified as an unknown object.

2. An information processing program comprising:

Images