Information Processing Systems

Abstract

To provide an information processing system configured to improve recognition accuracy in identifying an object from image information.SOLUTION: An information processing system for identifying an object included in image information comprises: a first learning processing unit which generates a first learning model by machine learning using first annotation data in which attributes are associated with data indicating an outline of the object; and a recognition processing unit which identifies object identification information of the object included in the image information. The recognition processing unit specifies, using the image information and the first learning model, the outline of the object included in the image information, as an outline region, identifies, when an attribute corresponding to the specified outline region does not satisfy a predetermined condition, one or more pieces of object identification information of the object included in the specified outline region, or identifies, when the attribute corresponding to the specified outline region satisfies the predetermined condition, the object included in the specified outline region as a predetermined object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system that does not reduce recognition accuracy when identifying an object from image information. [Background technology]

[0002] In convenience stores, supermarkets, and other stores, it is common for products and other objects to be sold on display shelves. This display method can involve displaying multiple objects horizontally to make them more visible to customers, or vertically so that even if one object is purchased, another person can purchase the same object. Managing where and how many objects are displayed on the display shelves is important in terms of sales strategies for the objects.

[0003] Therefore, in order to grasp the actual display status of objects in a store, there is a method of taking a picture of the display shelf with a camera and automatically identifying the displayed objects from the captured image information. For example, there is a method of using image recognition technology on images of the store's display shelf based on a specimen image of each object. Examples of such prior art include Patent Document 1 and Patent Document 2 below. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 5-342230 [Patent Document 2] Japanese Patent Application Publication No. 5-334409 Summary of the Invention [Problem to be solved by the invention]

[0005] The invention of Patent Document 1 is a system that assists even an uninformed person in determining on which shelf an object should be displayed. Therefore, while it is possible to determine where an object should be displayed, it does not identify the object that is displayed. Patent Document 2 is a system that assists in the input of object images in a shelf allocation support system that assists in the display of objects. However, the system of Patent Document 2 only assists in the input of object images when using the shelf allocation support system, and even if this system is used, it is not possible to grasp the specific display status of objects.

[0006] Furthermore, in addition to Patent Documents 1 and 2, there is also a technology that uses image recognition processing technology to identify displayed objects from image information captured on display shelves. This is beneficial in that it allows the actual display situation in a store to be grasped.

[0007] In conventional technology, when identifying an object using image recognition processing technology, a rectangular area where the object is thought to be located is detected from image information taken of a display shelf, and the object is identified by performing a matching process on that rectangular area with a sample image of the object, or by performing deep learning processing using the rectangular area as an input value.

[0008] However, the shape (outline) of an object is not necessarily rectangular. When detecting an area where an object is likely to be located as a rectangle, parts of other objects may be reflected in the rectangular area. In addition, the background may also be reflected in the rectangular area. Therefore, when such rectangular areas are used as training data for deep learning, as images to be processed, or in image matching processing, there is a problem that the accuracy of object identification may be reduced.

[0009] Furthermore, if an object is unknown due to unlearning or other reasons, any subsequent object identification process will result in an erroneous recognition. However, performing the identification process for such an erroneous object is a waste of processing, and it is preferable to eliminate it as early as possible. However, this has not been possible with conventional methods. Furthermore, it is difficult to detect an unlearned object itself. [Means for solving the problem]

[0010] In view of the above-mentioned problems, the present inventors have invented an information processing system that improves the accuracy of identifying an object when identifying an object captured in image information.

[0011] A first invention is an information processing system for identifying an object shown in image information, the information processing system including: a first learning processing unit that performs machine learning using first annotation data in which data indicating the outer shape of the object is associated with attributes to create a first learning model; and a recognition processing unit that identifies object identification information of the object shown in the image information, the recognition processing unit using the image information and the first learning model to identify the outer shape of the object shown in the image information as an outer shape region, and Output the confidence level for the corresponding attribute of the contour region; The corresponding attributes of the specified contour area If the reliability of is below a certain threshold, If not, using a second learning model or image matching process that has been machine-learned using image information of the specified outer shape region and second annotation data that associates image data of the object with object identification information, Identifying one or more object identification information of an object shown in the specified outline region, and The reliability of is below a predetermined threshold. In this case, the object in the specified outline area is unknown It is an information processing system that identifies an object as being of a certain type.

[0012] As in the present invention, by identifying object identification information using a learning model that has been machine-learned using the first annotation data, the contour area cut out from the image information can be changed from a conventional rectangular area to an area that follows the contour of the object. This reduces the inclusion of other objects and the background, thereby improving the accuracy of object identification.

[0013] Furthermore, if the attribute corresponding to the outline region of the identified object does not satisfy a predetermined condition, for example, if the reliability is lower than a predetermined threshold, the outline region can be identified as a predetermined object, such as an unknown object (for example, an object that has not been learned), without performing identification processing of the object identification information of the outline region. This makes it possible to detect unknown objects and reduce the processing load.

[0014] In the above-mentioned invention, the first learning processing unit can be configured as an information processing system that performs machine learning by image segmentation using the first annotation data to create the first learning model.

[0015] When performing machine learning, image segmentation methods are preferred.

[0016] In the above invention, the information processing system The aforementioned Second, machine learning is performed using annotation data. The aforementioned a second learning processing unit that creates a second learning model; have It can be configured like an information processing system.

[0017] Because the outline region is composed of an area that represents the outline of an object, it is preferable to identify object identification information by performing processing such as those of these inventions. Furthermore, if the object identification information of the identified object satisfies a predetermined condition, such as if its reliability is lower than a predetermined threshold, the outline region is identified as a predetermined object, such as an unknown object, regardless of the identified object identification information. This makes it possible to avoid identifying object identification information with low reliability.

[0018] In the above-mentioned invention, the recognition processing unit can be configured as an information processing system that identifies object identification information of the object depicted in the outline region by performing image matching processing on the image information of the identified outline region and specimen information of the object stored in the specimen information storage unit.

[0019] As in the present invention, image matching processing may be used to identify the object identification information of the object captured in the contour area.

[0020] A fifth invention is an information processing system for identifying an object shown in image information, the information processing system including a first learning processing unit that performs machine learning using first annotation data in which data indicating the outer shape of the object is associated with attributes, to create a first learning model, and that uses the image information and the first learning model to identify the outer shape of the object shown in the image information as an outer shape region, Output the confidence level for the corresponding attribute of the contour region; The corresponding attributes of the specified contour area If the reliability of is below a certain threshold, If not, using a second learning model or image matching process that has been machine-learned using image information of the specified outer shape region and second annotation data that associates image data of the object with object identification information, Identifying object identification information of an object shown in the specified outline region, and The reliability of is below a predetermined threshold. In this case, the object in the specified outline area is unknown It is an information processing system that identifies an object as such.

[0021] A sixth invention is an information processing system for identifying an object shown in image information, the system including: an image information input reception processing unit that receives input of image information of an object; and an object recognition processing unit that identifies object identification information of the object shown in the image from the received input image information or image information obtained by orthogonalizing the image information. The object recognition processing unit identifies the outline of the object shown in the image as an outline region using a first learning model created by machine learning using first annotation data that associates data indicating the outline of the object with attributes, and the received input image information or image information obtained by orthogonalizing the image information. Output the confidence level for the corresponding attribute of the contour region; The corresponding attributes of the specified contour area If the reliability of is below a certain threshold,If not, using a second learning model or image matching process that has been machine-learned using image information of the specified outer shape region and second annotation data that associates image data of the object with object identification information, Identifying object identification information of an object shown in the specified outline region, and The reliability of is below a predetermined threshold. In this case, the object in the specified outline area is unknown It is an information processing system that identifies an object as being of a certain type.

[0022] Even with the configurations of these inventions, the same technical effects as those of the first invention can be obtained.

[0023] The first invention can be realized by loading the program of the present invention into a computer and executing it. That is, the information processing program causes a computer to function as a first learning processing unit that performs machine learning using first annotation data in which data indicating the outer shape of an object is associated with attributes, to create a first learning model, and a recognition processing unit that identifies object identification information of an object shown in image information, wherein the recognition processing unit uses the image information and the first learning model to identify the outer shape of the object shown in the image information as an outer shape region, and Output the confidence level for the corresponding attribute of the contour region; The corresponding attributes of the specified contour area If the reliability of is below a certain threshold, If not, using a second learning model or image matching process that has been machine-learned using image information of the specified outer shape region and second annotation data that associates image data of the object with object identification information, Identifying one or more object identification information of an object shown in the specified outline region, and The reliability of is below a predetermined threshold. In this case, the object in the specified outline area is unknown It is an information processing program that identifies an object as being of the same type.

[0024] The fifth invention can be realized by loading the program of the present invention into a computer and executing it. That is, the information processing program causes a computer to function as a first learning processing unit that performs machine learning using first annotation data in which data indicating the outer shape of an object is associated with attributes, and creates a first learning model, Image informationand the first learning model, identifies the outline of an object shown in the image information as an outline region, outputs a reliability for an attribute corresponding to the identified outline region, and, if the reliability of the attribute corresponding to the identified outline region is not below a predetermined threshold, identifies object identification information of the object shown in the identified outline region using a second learning model or image matching process that has been machine-learned using the image information of the identified outline region and second annotation data that associates image data of the object with object identification information, and, if the reliability of the attribute corresponding to the identified outline region is below a predetermined threshold, identifies the object shown in the identified outline region as an unknown object.

[0025] The sixth invention can be realized by loading the program of the present invention into a computer and executing it. That is, the information processing program causes a computer to function as an image information input reception processing unit that receives input of image information of an object, and an object recognition processing unit that identifies object identification information of the object in the image from the received image information or image information obtained by orthogonalizing the image information, wherein the object recognition processing unit identifies the outline of the object in the image as an outline region using a first learning model created by machine learning using first annotation data that associates data indicating the outline of the object with attributes, and the received image information or image information obtained by orthogonalizing the image information, and Output the confidence level for the corresponding attribute of the contour region; The corresponding attributes of the specified contour area If the reliability of is below a certain threshold, If not, using a second learning model or image matching process that has been machine-learned using image information of the specified outer shape region and second annotation data that associates image data of the object with object identification information, Identifying object identification information of an object shown in the specified outline region, and The reliability of is below a predetermined threshold. In this case, the object in the specified outline area is unknown It is an information processing program that identifies an object as being of the same type. [Effects of the Invention]

[0026] By using the information processing system of the present invention, it is possible to improve the accuracy of identification when identifying an object captured in image information. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a block diagram schematically illustrating an example of a configuration of an information processing system according to the present invention. [Figure 2] FIG. 2 is a block diagram schematically illustrating an example of a configuration of an object recognition processing unit in the information processing system of the present invention. [Figure 3] FIG. 2 is a block diagram schematically illustrating an example of a hardware configuration of a computer used in the information processing system of the present invention. [Figure 4] 10 is a flowchart showing an example of a learning process in the information processing system of the present invention. [Figure 5] 10 is a flowchart showing an example of a processing process for recognition processing in the information processing system of the present invention. [Figure 6] FIG. 2 is a diagram schematically illustrating an example of first annotation data. [Figure 7] FIG. 10 is a diagram schematically illustrating another example of the first annotation data. [Figure 8] FIG. 10 is a diagram schematically illustrating an example of second annotation data. [Figure 9] FIG. 10 is a diagram schematically illustrating another example of the second annotation data. [Figure 10] FIG. 10 is a diagram showing an example of captured image information. [Figure 11] FIG. 10 is a diagram showing another example of captured image information. [Figure 12] 11 is a diagram showing an example of image information obtained by normalizing the captured image information of FIG. 10. FIG. [Figure 13] 12 is a diagram showing an example of image information obtained by normalizing the captured image information of FIG. 11. FIG. [Figure 14] FIG. 10 is a block diagram illustrating an example of the configuration of an information processing system according to a second embodiment. [Figure 15]FIG. 10 is a diagram showing a schematic diagram of a state in which input specifying a shelf area is received for upright image information obtained by orthogonally ... [Figure 16] FIG. 10 is a diagram showing a schematic diagram of a state in which input specifying a shelf area is received for upright image information obtained by orthogonally ... [Figure 17] FIG. 10 is a diagram showing an example of a case where an outer shape area is identified from image information of a shelf area. [Figure 18] 10 is a flowchart illustrating an example of a processing process of recognition processing in the information processing system according to the second embodiment. [Figure 19] FIG. 10 is a diagram showing an example of captured image information. [Figure 20] 20 is a diagram showing an example of normalized image information obtained by performing normalization processing on the photographed image information of FIG. 19. FIG. [Figure 21] FIG. 10 is a block diagram illustrating an example of the configuration of an information processing system according to a third embodiment. [Figure 22] FIG. 4 is a diagram illustrating an example of specimen information stored in a specimen information storage unit. [Figure 23] FIG. 10 is a block diagram illustrating an example of an object recognition processing unit according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0028] 1 and 2 are block diagrams showing an example of the processing functions of an information processing system 1 of the present invention. The information processing system 1 uses a management terminal 2 and an image information input terminal 3. Fig. 1 is a block diagram showing the overall functions of the information processing system 1, and Fig. 2 is a block diagram showing the functions of an object recognition processing unit 213, which will be described later.

[0029] The management terminal 2 is a computer used by an organization such as a company that operates the information processing system 1. The image information input terminal 3 is a terminal for inputting image information including an object to be identified, for example, image information obtained by photographing a display shelf in a store where the product to be identified is displayed.

[0030] The management terminal 2 and the image information input terminal 3 in the information processing system 1 are realized using a computer. An example of the hardware configuration of a computer is shown in Fig. 3. The computer has a calculation device 70 such as a CPU that executes the calculation processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or a mouse that can input information, and a communication device 74 that sends and receives the processing results of the calculation device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0031] If the computer is equipped with a touch panel display, the display device 72 may be integrated with the input device 73. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited to these.

[0032] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

[0033] In addition to the above devices, the image information input terminal 3 may also be equipped with a photographing device such as a camera. The image information input terminal 3 may also be a portable communication terminal such as a mobile phone, a smartphone, or a tablet computer.

[0034] The functions of the various means in the present invention are only logically distinct, and may be physically or practically the same area. The order of the processes in the various means of the present invention may be changed as appropriate. Also, some of the processes may be omitted. For example, the normalization process described below may be omitted. In this case, the process may be performed on image information that has not been normalized.

[0035] The information processing system 1 includes a learning processing unit 20, a recognition processing unit 21, and an object information storage unit 22. The learning processing unit 20 includes a first learning processing unit 201 and a second learning processing unit 202.

[0036] The first learning processing unit 201 uses the first annotation data to perform a learning process by machine learning, preferably a learning process by image segmentation, on image information including an object to be identified, such as image information obtained by photographing a display shelf displaying products to be identified. This learning process is a learning process in machine learning, and executes a learning process by image segmentation to create a learning model using, for example, deep learning.

[0037] The first annotation data is data in which the contour of an object that may be the target of identification is used as the contour, and data in which the inside of a closed area formed by the contour is masked is associated with a tag (label) that classifies attributes of the contour of the object. Note that the contour may be the contour of the object itself, or may be a shape that includes the contour of the object and follows the contour of the object, or a shape that has a width from the contour of the object, such as a shape that indicates the contour of the object. In other words, the contour of the object's contour is not limited to a rectangular area, and may also be a non-rectangular area that reflects the contour of the object.

[0038] For example, if the object is a product displayed on a display shelf, the first annotation data is data that corresponds to the outline of the object that may be displayed on the display shelf, with the inside of the closed area formed by the outline masked, and a tag (label) that classifies the attributes of the object's outline.

[0039] The first annotation data does not have to be one per object, and may be multiple per object. In other words, data obtained by masking the interior of an object's outline from multiple directions may be associated with attributes, and each of these may be used as the first annotation data for the object. The attribute refers to the type of object. Examples of object types include animal types (e.g., elephant, tiger, lion), bird types (e.g., chicken, parakeet, peacock), insect types (e.g., rhinoceros beetle, stag beetle, butterfly), fish types (e.g., tuna, mackerel, sardine), plant types (e.g., cherry blossom, plum, lily, rose, chrysanthemum, tulip), and mobile object types (e.g., vehicle, aircraft) (e.g., automobile, bicycle, motorcycle, airplane, helicopter, drone, UAV (Unmanned Aircraft Vehicle), ship, boat), as well as various other types such as the classification of the object's container and the object's object identification information (e.g., JAN code). The classification of containers may be by type of container, such as can, bottle, box, or pouch container, or may be further subdivided according to use, such as detergent container. In other words, the attribute may indicate how the closed area defined by the contour of the object is classified. Examples of the first annotation data are shown in FIGS. 6 and 7. Although FIGS. 6 and 7 show the case of displayed products as objects, as mentioned above, the object identification information is not limited to JAN codes and may be any information that can uniquely identify an object.

[0040] Fig. 6 shows a case where data obtained by masking the outline of a can and the inside of the closed area formed by the outline is associated with "can" as an attribute to form first annotation data, and Fig. 7 shows a case where data obtained by masking the outline of a refillable shampoo and the inside of the closed area formed by the outline is associated with "pouch container" as an attribute to form first annotation data. In cases where an object can be identified from the outline itself, object identification information such as a JAN code may be used as the attribute in the first annotation data instead of the container classification.

[0041] By executing a learning process for machine learning using the first annotation data in the first learning processing unit 201, a learning model (first learning model) for identifying the contour area of ​​an object is created from image information of an object that may be a target for identification. For example, a learning model (first learning model) for identifying the contour area of ​​an object is created from image information of an image of a display shelf. Note that the first learning model may be a learning model that can identify the contour area of ​​an object as well as the attributes of the object.

[0042] The second learning processing unit 202 executes a machine learning learning process using the second annotation data to create a learning model (second learning model) for identifying object identification information of an object in a predetermined image information, preferably image information of an outline region described below, from that region. The learning process in this case is preferably performed using an image classification method, but may also be a method such as object detection, image classification, or object localization. The second learning model may be a learning model that can input attributes corresponding to the outline region as input values ​​for the second learning model, in addition to image information of the outline region.

[0043] The second annotation data is data in which image information of an object that may be identified is associated with the object identification information of the object as a tag (label). For example, if the object is a displayed commodity, the second annotation data is data in which image information of the commodity that may be displayed on a shelf is associated with the object identification information of the object as a tag (label). If the object is an animal, the second annotation data is data in which image information of the animal that may be identified is associated with identification information such as the animal's name (scientific name, common name, etc.) as a tag (label) of the object identification information. Examples of the second annotation data are shown in FIGS. 8 and 9. Although FIGS. 8 and 9 show the case of a displayed commodity as the object, as mentioned above, the second annotation data is not limited to this. Like the first annotation data, the second annotation data may be multiple and not just one for each object. That is, the object may be photographed from multiple directions, and the image information of the object from each direction may be associated with the object identification information to generate the second annotation data. Furthermore, if the second learning model allows attributes corresponding to the outer shape area to be input as input values, the second annotation data may be data that associates image information of an object that may be the subject of identification with the attributes of that object and the object identification information of that object as tags (labels).

[0044] Fig. 8 shows a case where image information of a can is associated with object identification information to form second annotation data, and Fig. 9 shows a case where image information of a shampoo refill is associated with object identification information to form second annotation data. The second annotation data in Fig. 8 corresponds to the first annotation data in Fig. 6, and the second annotation data in Fig. 8 corresponds to the second annotation data in Fig. 7.

[0045] The recognition processing unit 21 includes an image information input reception processing unit 210 , an image information storage unit 211 , an image information alignment processing unit 212 , and an object recognition processing unit 213 .

[0046] The image information input reception processing unit 210 receives input of image information (photographed image information) including an object to be identified, photographed by the image information input terminal 3, and stores the information in the image information storage unit 211 (described later). It is preferable to receive input from the image information input terminal 3, in addition to the photographed image information, along with the photographed date and time, image information identification information for identifying the image information, and the like. An example of photographed image information in the case where the object is a product displayed on a display shelf is shown in FIGS. 10 and 11. In FIGS. 10 and 11, the display shelf has three shelves, and the photographed image information shows the object being displayed on the shelves. While the present invention does not specifically specify the processing, display shelves and shelves are often long in the horizontal direction. Therefore, in the processing, the image may be divided into sections at a certain width and used as the processing target for each process.

[0047] The image information storage unit 211 stores captured image information, shooting date and time, image information identification information, etc. received from the image information input terminal 3 in association with each other. The captured image information may be any image information to be processed in the present invention. Generally, when simply shooting, it is difficult to shoot a photograph of the subject facing the camera in a straight-on position. Therefore, it is advisable to perform a correction process to correct the photograph to a straight-on position, such as a keystone correction process. When multiple images of a single subject are taken, the captured image information also includes image information obtained by combining these images into a single piece of image information. The captured image information also includes image information after distortion correction processing has been performed.

[0048] The image information orthogonalization processing unit 212 performs a process (orthogonalization process) on the captured image information stored in the image information storage unit 211 to correct the captured object so that it is facing upright, for example, to generate orthogonalized image information by performing keystone correction process. Keystone correction process is a correction process performed to orient the object captured in the captured image information so that it is facing upright. For example, if the object is a displayed commodity, this is a correction process performed so that the shelves on which the displayed commodity is displayed are horizontal. Orthogonalization is a process of transforming the image information so that the optical axis of the lens of the image capture device is aligned perpendicular to the plane of the object to be captured, so that it is the same as if it were captured from a sufficiently distant location; for example, keystone correction process is used, but is not limited to this.

[0049] The keystone correction process executed by the image information alignment processor 212 accepts input specifying four vertices in the captured image information, and executes the keystone correction process using each of those vertices. If the object is a commodity displayed on a display shelf, the four vertices to be specified may be the four vertices of the shelf level of the display shelf, or the four vertices of the shelf position on the display shelf. Alternatively, they may be the four vertices of a group of two or three shelf levels. Any four points can be specified as the four vertices. Fig. 12 shows an example of captured image information (aligned image information) obtained by alignment of the captured image information of Fig. 10, and Fig. 13 shows an example of captured image information obtained by alignment of the captured image information of Fig. 11.

[0050] The object recognition processing unit 213 executes processing to recognize an object appearing in image information, preferably captured image information or upright image information.

[0051] The object recognition processing unit 213 includes an outer shape specification processing unit 2131 and an object identification processing unit 2132 .

[0052] The outer shape specification processing unit 2131 specifies the area of ​​the outer shape of the object (outer shape area) shown in the normal image information.

[0053] The contour identification processing unit 2131 inputs the captured image information and the upright image information as input values ​​to a learning model (first learning model) trained in the first learning processing unit 201, and identifies a contour region from the input image information. That is, the contour identification processing unit 2131 inputs a region to be processed, for example, upright image information, to the learning model trained in the first learning processing unit 201 (a learning model in which weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized), and identifies a contour region based on the output value. The identified contour region is assigned contour identification information that identifies the contour region, and is stored in the object identification information storage unit 22 together with position information in the captured image information and upright image information (for example, coordinates in the image information).

[0054] The attribute tag corresponding to the contour region and its reliability (probability) may also be output as output values ​​from the contour identification processing unit 2131. At this time, the attribute tag with the highest reliability is identified as the tag of the contour region, and if the reliability is below a certain threshold, the identified contour region is identified as a predetermined object such as an unknown object (an object that cannot be identified for reasons such as not having been learned).

[0055] The object identification processing unit 2132 inputs the image information of the outline region as an input value to a learning model (second learning model) trained in the second learning processing unit 202, and identifies the identification information of the object in that region from the input image information. That is, the object identification processing unit 2132 inputs the image information of the outline region to be processed into the learning model trained in the second learning processing unit 202 (a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized), and identifies the object identification information of the object in the outline region based on the output value. At this time, in addition to the image information of the outline region, tag information of the attribute corresponding to the outline region identified by the outline identification processing unit 2131 may also be input as an input value. Note that at this time, one object identification information may be identified for the outline region, or multiple object identification information may be identified. When multiple object identification information is identified, candidates for object identification information to be identified are output.

[0056] The reliability (probability) of the object identification information identified by the outline region is also output as an output value from the object identification processing unit 2132. At this time, the object identification information with the highest reliability is identified as the object identification information of the outline region, but if the highest reliability is equal to or less than a certain threshold, the input outline region is identified as an unknown object.

[0057] The object recognition processing unit 213 may perform the processing of the outer shape specification processing unit 2131 and the object identification processing unit 2132 together by deep learning or the like.

[0058] The object identification information storage unit 22 stores information indicating the object identification information of objects shown in the photographed image information and the normal-position image information. For example, the object identification information is stored in the object identification information storage unit 22 in association with the photographed date and time information, image information identification information of the photographed image information, image identification information of the normal-position image information, and external shape identification information for identifying the external shape. [Example]

[0059] Next, an example of the processing process of the information processing system 1 of the present invention will be described with reference to the flowcharts of FIGS.

[0060] First, the learning process for learning a learning model used in the recognition processing unit 21 of the information processing system 1 of the present invention will be described with reference to the flowchart of FIG.

[0061] First annotation data is created as training data for the learning model in the first learning processing unit 201 (S100). The first annotation data is image data in which the outline of an object that may be the target of identification is formed and the closed area inside the outline is masked. Attributes are associated with this image data as tags.

[0062] Similarly, second annotation data is created as training data for the learning model in the second learning processing unit 202 (S110). The second annotation data is created by associating an image of an object that may be the target of identification with the object identification information of the object as a tag. The image information of the object in this case preferably corresponds to the first annotation data and is preferably image information that shows the outline of the object.

[0063] Then, the created first annotation data is input as training data, and a learning process for machine learning is executed in the first learning processing unit 201 to create a learning model (first learning model) for identifying the contour area of ​​the object (S120).

[0064] In addition, the created second annotation data is input as training data, and a learning process for machine learning is executed in the second learning processing unit 202 to create a learning model (second learning model) for identifying object identification information of an object in the image information, preferably the outline area, from that area (S130).

[0065] By performing the above-described processing, each learning model can be created.

[0066] Next, a recognition process for identifying object identification information of an object captured in image information obtained by capturing an object to be identified will be described with reference to the flowchart of FIG.

[0067] Photographed image information of an object to be identified is input from the image information input terminal 3, and the input is accepted by the image information input acceptance processing unit 210 of the management terminal 2 (S200). The image information input acceptance processing unit 210 also accepts input of the photographed date and time and image information identification information of the photographed image information. Then, the image information input acceptance processing unit 210 associates the accepted input photographed image information, photographed date and time, and image information identification information of the photographed image information, and stores them in the image information storage unit 211.

[0068] When a predetermined operation input is received at the management terminal 2, the image information alignment processing unit 212 extracts the captured image information stored in the image information storage unit 211, receives input of four points for performing alignment processing such as trapezoid correction processing, and executes the alignment processing (S210).

[0069] Then, the management terminal 2 receives a predetermined operation input for the normal position image information, thereby executing a process for identifying the outer shape (S220). That is, the outer shape identification processing unit 2131 inputs a part or all of the area in the captured image information or the normal position image information as input values ​​to the learning model (first learning model) learned by the first learning processing unit 201, and identifies the outer shape area from the input image information.

[0070] As described above, the contour regions of each object in the upright image information are identified, and their attributes (tags) are identified. If the reliability of the identified tag is below a certain threshold (S230), the object in that contour region is identified as an unknown object (S240).

[0071] On the other hand, if the reliability of the identified tag is greater than a certain threshold (S230), the object identification processing unit 2132 inputs the image information of the contour region as an input value to a learning model (second learning model) trained in the second learning processing unit 202, and identifies the object identification information of the object appearing in the contour region (S250). If the reliability of the identified object identification information is equal to or less than a certain threshold (S260), the object in the contour region is identified as an unknown object (S240).

[0072] On the other hand, if the reliability of the identified object identification information is greater than a certain threshold (S260), the identified object identification information is stored in the object identification information storage unit 22 in association with the shooting date and time, image information identification information of the photographed image information, image information identification information of the upright image information, and external shape identification information.

[0073] It is also possible to accept input regarding a process for correcting the identified object identification information.

[0074] By performing the above-described processing, it is possible to identify the object identification information of the object captured in the captured image information. Furthermore, unlike conventional systems, the outer shape region is not a rectangular region, but the object is identified along the outer shape of the object's outline, so unnecessary information contained in the outer shape region, such as noise from other objects, is eliminated, thereby improving recognition accuracy.

[0075] Note that when object identification information is used as an attribute in the first annotation data (when an object can be identified from its outer shape), the object identification information of the object can be identified by inputting image information into the first learning model and specifying the outer shape region. In this case, it is not necessary to perform processing by the second learning processing unit 202 and the object identification processing unit 2132. Once the outer shape region is specified by the outer shape specification processing unit 2131, the object identification information of the object shown in the outer shape region may be identified by the object identification information of the attribute as an output result from the first learning model. [Example]

[0076] In the information processing system 1 of the first embodiment described above, in the case of displayed products displayed on a display shelf as objects, the processing may be further executed for each shelf area (shelf area). An example of the processing function of the information processing system 1 in this case is shown in a block diagram in FIG.

[0077] In the information processing system 1 of the second embodiment, the recognition processing unit 21 further includes a shelf level identification processing unit 214 .

[0078] The shelf level identification processing unit 214 identifies the area of ​​a shelf on a display shelf (shelf level area) where an object may be placed, from the upright image information obtained by performing keystone correction processing on the captured image information in the image information alignment processing unit 212. The captured image information and the upright image information show a display shelf, but the display shelf has a shelf level area where an object is displayed. Therefore, the shelf level area is identified from the upright image information. To identify the shelf level area, the operator of the management terminal 2 may manually specify the shelf level area, which the shelf level identification processing unit 214 may accept, or the shelf level area may be identified automatically from the second time onwards based on the information on the shelf level area that was manually input the first time.

[0079] Fig. 15 shows a schematic diagram of a state in which an input for specifying a shelf area has been received for normal-positioned image information obtained by normalizing image information obtained by photographing a display shelf on which objects such as beverage cans are displayed. Fig. 16 also shows a schematic diagram of a state in which an input for specifying a shelf area has been received for normal-positioned image information obtained by normalizing image information obtained by photographing a display shelf on which objects such as toothbrushes are hung and displayed.

[0080] When identifying the shelf level area, the shelf level identification processing unit 214 may use deep learning to identify the shelf level area. In this case, the above-mentioned upright image information may be input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the shelf level area may be identified based on the output value. Furthermore, the learning model may be one in which the shelf level area is assigned as correct answer data to various upright image information.

[0081] The shelf level area identified by the shelf level identification processing unit 214 identifies its image information as shelf level area image information. The shelf level identification processing unit 214 may actually cut out the image information, or may virtually cut out the image information of the area by specifying coordinates or the like without actually cutting out the image information. If the display shelf has multiple shelf levels, each shelf level is cut out as shelf level area image information. The coordinates indicating the shelf level area are the coordinates of vertices for specifying the area, and may be, for example, four points in the normal image information, such as the coordinates of the upper right and lower left, or two points in the upper left and lower right. Furthermore, these are relative coordinates based on a predetermined location in the image information (for example, the upper left vertex of the display shelf) such as the display shelf in the normal image information. In this specification, cutting out image information may mean actually cutting out the image information, similar to cutting out by the shelf level identification processing unit 214, or may mean virtually cutting out the image information of the area by specifying coordinates or the like without actually cutting out the image information.

[0082] The outer shape specification processing unit 2131 and the object identification processing unit 2132 in the object recognition processing unit 213 in the information processing system 1 of this embodiment may execute the following processing in addition to the processing of the first embodiment.

[0083] The contour identification processing unit 2131 identifies a contour area (contour area) for each shelf in the shelf area in the normal image information. The contour is an area where an object is placed, regardless of whether the object is placed there or not. The size of the contour area is the same as or approximately the same as the size of the object to be placed there.

[0084] The contour identification processing unit 2131 inputs the captured image information, the upright image information, or the image information of the shelf area as input values ​​to a learning model (first learning model) trained by the first learning processing unit 201, and identifies a contour area from the input image information. That is, the contour identification processing unit 2131 inputs the image information of the area to be processed, for example, the shelf area, to the learning model trained by the first learning processing unit 201 (a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized), and identifies a contour area based on the output value. The identified contour area is assigned contour identification information that identifies the contour area, and is stored in the object identification information storage unit 22 together with position information (for example, coordinates in the image information) in the captured image information, the upright image information, or the image information of the shelf area.

[0085] The output value output by the contour identification processing unit 2131 includes the attribute tag corresponding to the contour region, for example, can, bottle, pouch container, etc., along with its reliability (probability). At this time, the attribute tag with the highest reliability is identified as the tag for the contour region, and if the reliability is below a certain threshold, the identified contour region is identified as an unknown object.

[0086] Fig. 17 shows an example of a case where an outline region is identified from image information of a shelf level region. Fig. 17(a) is an example of image information of a shelf level region that is input to a learning model trained by the first learning processing unit 201, and Fig. 17(b) is a diagram showing an example of a state where an outline region is identified using the learning model in the image information of the shelf level region that is used as an input value in Fig. 17(a). Fig. 17(b) shows a state where an outline region is identified in the shelf level region superimposed, but the image information of the identified outline region may also be cut out and output as is.

[0087] The object identification processing unit 2132 inputs the image information of the outline region as an input value to a learning model (second learning model) trained in the second learning processing unit 202, and identifies the identification information of the object in that region from the input image information. That is, the object identification processing unit 2132 inputs the image information of the outline region to be processed into the learning model trained in the second learning processing unit 202 (a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized), and identifies the object identification information of the object in the outline region based on the output value. At this time, in addition to the image information of the outline region, attribute tags corresponding to the outline region identified by the outline identification processing unit 2131, such as information on a can, bottle, or pouch container, may be input as input values. Note that at this time, one or more object identification information may be identified for the outline region. When multiple object identification information is identified, candidates for object identification information to be identified are output.

[0088] The reliability (probability) of the object identification information identified by the outline region is also output as an output value from the object identification processing unit 2132. At this time, the object identification information with the highest reliability is identified as the object identification information of the outline region, but if the highest reliability is equal to or less than a certain threshold, the input outline region is identified as an unknown object.

[0089] As in the first embodiment, the object recognition processing unit 213 may perform the processing of the outer shape specification processing unit 2131 and the object identification processing unit 2132 together by deep learning or the like.

[0090] The object identification information storage unit 22 stores information indicating the object identification information of the objects displayed on the outer shapes of the shelves of the display shelves. For example, the object identification information storage unit 22 stores, in association with the object identification information, information on the date and time of photographing, store information, image information identification information of the photographed image information, image identification information of the upright image information, and outer shape identification information for identifying the outer shape.

[0091] Next, an example of the processing process of the information processing system 1 in this embodiment will be described with reference to the flowcharts of Figures 4 and 18. In the following description of this embodiment, a case will be described in which object identification information of displayed objects is identified from captured image information.

[0092] The learning process (FIG. 4) for learning the learning model used in the recognition processing unit 21 of the information processing system 1 in this embodiment is the same as that in the first embodiment, and therefore a description thereof will be omitted.

[0093] The recognition process for identifying the object identification information of the objects displayed on the display shelves from the image information obtained by photographing the display shelves will be described with reference to the flowchart of FIG.

[0094] Captured image information of a store's display shelves is input from the image information input terminal 3, and the input is accepted by the image information input acceptance processing unit 210 of the management terminal 2 (S300). Fig. 19 shows an example of captured image information. Also accepted are input of the date and time of capture, store identification information, and image information identification information of the captured image information. Then, the image information input acceptance processing unit 210 associates the accepted input captured image information, date and time of capture, store identification information, and image information identification information of the captured image information, and stores them in the image information storage unit 211.

[0095] When a predetermined operation input is received in the management terminal 2, the normalized image information normalization processing unit 212 extracts the photographed image information stored in the image information storage unit 211, receives input of four shelf positions (positions of display shelves) that are vertices for performing normalization processing such as keystone correction processing, and executes normalization processing (S310). An example of photographed image information (normalized image information) that has been normalized in this way is shown in Fig. 20.

[0096] Then, by receiving a predetermined operation input in the management terminal 2 for the normal placement image information, the shelf level identification processing unit 214 identifies the shelf level position area (S320). That is, the input of the shelf level area in the normal placement image information is received. Figures 15 and 16 are diagrams showing the state in which the shelf level area has been identified from the normal placement image information.

[0097] Once the shelf level area is identified in the above manner, image information of the shelf level area is extracted from the normal image information. Then, a process of identifying the outline of each shelf level in the shelf level area image information is executed (S330). That is, the outline identification processing unit 2131 inputs the image information of the shelf level area as an input value to the learning model (first learning model) trained by the first learning processing unit 201, and identifies the outline area from the input image information.

[0098] As described above, the contour regions of each object in the upright image information are identified, and their attributes (tags) are identified. If the reliability of the identified tag is below a certain threshold (S340), the object in that contour region is identified as an unknown object (S350).

[0099] On the other hand, if the reliability of the identified tag is greater than a certain threshold (S340), the object identification processing unit 2132 inputs the image information of the outline region as an input value to a learning model (second learning model) trained in the second learning processing unit 202, and identifies the object identification information of the object appearing in the outline region (S360). If the reliability of the identified object identification information is equal to or less than a certain threshold (S370), the object in the outline region is identified as an unknown object (S350).

[0100] On the other hand, if the reliability of the identified object identification information is greater than a certain threshold (S370), the identified object identification information is stored in the object identification information storage unit 22 in association with the shooting date and time, store identification information, image information identification information of the photographed image information, image information identification information of the upright image information, and external shape identification information.

[0101] Note that it is not always possible to identify object identification information for all outline regions. Therefore, for outline regions that cannot be identified, input of object identification information is accepted, and the accepted input object identification information is stored in the object identification information storage unit 22 in association with the shooting date and time, store identification information, image information identification information of the photographed image information, image information identification information of the upright image information, and outline identification information. Similarly, input may also be accepted for correction processing of identified object identification information.

[0102] By performing the above-described processing, it is possible to identify the object identification information of the object displayed on the shelf of the display shelf shown in the captured image information. Furthermore, unlike conventional systems, the outer shape region is not a rectangular region, but the object is identified along the outer shape of the object's outline, so unnecessary information contained in the outer shape region, such as noise from other objects, is eliminated, thereby improving recognition accuracy.

[0103] Note that when object identification information is used as an attribute in the first annotation data (when an object can be identified from its outer shape), the object identification information of the object can be identified by inputting image information into the first learning model and specifying the outer shape region. In this case, it is not necessary to perform processing by the second learning processing unit 202 and the object identification processing unit 2132. Once the outer shape region is specified by the outer shape specification processing unit 2131, the object identification information of the object shown in the outer shape region may be identified by the object identification information of the attribute as an output result from the first learning model. [Example]

[0104] In the first and second embodiments, a configuration has been described in which machine learning is used in two processes, that is, specifying the outline region and identifying the object identification information from the outline region, but image matching processing may be used for the process of identifying the object identification information from the outline region. An example of the configuration of the information processing system 1 in this case is shown in Fig. 21. Note that Fig. 21 shows a case based on the configuration of the information processing system 1 in the second embodiment, but it goes without saying that the information processing system 1 may be based on the configuration of the first embodiment.

[0105] In the information processing system 1 of this embodiment, the learning processing unit 20 does not need to be provided with the second learning processing unit 202. Furthermore, the recognition processing unit 21 includes a sample information storage unit 215 that stores sample information used in the image matching process.

[0106] The specimen information storage unit 215 stores specimen information for identifying which object is displayed on the shelf of the display shelf shown in the image information. The specimen information is image information of an object that may be displayed on the display shelf, photographed from multiple angles, such as from above, below, left, right, or diagonally. FIG. 22 shows an example of specimen information stored in the specimen information storage unit 215. FIG. 22 shows the specimen information of canned beer photographed from various angles, but it is not limited to canned beer. The specimen information storage unit 215 stores specimen information in association with object identification information.

[0107] The sample information storage unit 215 may store, together with or instead of the sample information, information extracted from the sample information that is necessary for calculating similarity, such as information on pairs of image features and their positions. The sample information also includes information necessary for calculating similarity. In this case, when performing a matching process between image information of an outline region (described later) and the sample information, the object recognition processing unit 213 does not need to calculate image features for the sample information every time, thereby reducing the calculation time.

[0108] The specimen information stored in the specimen information storage unit may be image information of an object in which the outer shape of the object's contour in the first annotation data used during the learning process by the first learning processing unit 201 has been masked. That is, when creating the first annotation data, image information of the object photographed from one or more directions or its image feature quantities is used as specimen information. Then, from the image information of the photographed object, the contour is used as the outer shape, and the inside of the closed region is masked and tagged with attributes to create the first annotation data. By this process, specimen information and first annotation data can be created together.

[0109] The object identification processing unit 2132 in this embodiment performs a matching process between the image information of the contour region identified by the contour identification processing unit 2131 and the specimen information stored in the specimen information storage unit 215, and identifies the object identification information of the object displayed in that contour region. That is, the image information of the contour region of a certain shelf (the contour identification information of this contour region is designated X) is used to calculate image feature amounts from each of the pieces of specimen information stored in the specimen information storage unit, and pairs of feature points are found to determine similarity. The most similar specimen information is then identified, and if the similarity at that time is equal to or greater than a predetermined threshold, the object identification information corresponding to that specimen information is identified based on the specimen information storage unit 215. The identified object identification information is then used as the object identification information of the object displayed in the contour of the contour identification information X. Note that for contours determined to be not similar to any specimen information, information indicating that the contour identification information is "empty" (information indicating that no object is present) is added to the contour identification information. The object identification processing unit 2132 stores the identified object identification information or information indicating that the object is "empty" in the object identification information storage unit 22 in correspondence with the shooting date and time, store identification information, image information identification information of the photographed image information, image information identification information of the upright image information, and external shape identification information.

[0110] Specifically, as an example, the object identification processing unit 2132 executes the following process: First, it determines the similarity between the image information formed by the coordinates of the outline region to be processed and the specimen information stored in the specimen information storage unit 215, identifies the object identification information corresponding to the specimen information with the highest similarity, and if the identified similarity is equal to or greater than a predetermined threshold, identifies it as the object identification information of the object displayed in the outline region formed by the coordinates.

[0111] Here, to determine the similarity between the image information of the outer shape and the specimen information, the following processing is performed: First, in the processing before the identification processing of the object identification information in the object identification processing unit 2132, the image information of the outer shape region on the shelf of the upright image information and the specimen information are oriented in the same direction (not turned sideways or upside down), and the sizes of the respective image information are roughly the same (if the sizes of the image information differ by more than a predetermined range, the sizes of the respective image information are adjusted so that they are within a predetermined range before determining the similarity).

[0112] To determine the similarity between the image information of the contour region and the specimen information, the object identification processing unit 2132 extracts feature points based on the image feature amounts (e.g., local feature amounts) of the contour image information and feature points based on the image feature amounts (e.g., local feature amounts) of the specimen information. Then, it detects the most similar pair of feature points in the contour image information and feature points in the specimen information, and calculates the difference in coordinates of corresponding points between them. It then calculates the average of these differences. The average difference indicates the overall average amount of movement between the image information of the contour region and the specimen information. It then compares the coordinate differences of all feature point pairs with the average coordinate difference, and excludes pairs with a high degree of discrepancy. It then ranks the similarity based on the number of remaining corresponding points.

[0113] The above method can calculate the similarity between the image information of the outline region and the specimen information. Furthermore, to improve accuracy, the Earth Movers Distance (EMD) between the color histograms can be calculated and used as a measure of similarity. This allows for a comparison of similarities that are relatively resistant to environmental changes, such as brightness information in the captured image information, and allows for highly accurate identification.

[0114] Another way to determine similarity is to calculate the EMD between the image information signatures (a set of image features and weights) of each contour region and use this as a measure of similarity. For example, the image feature of the signature can be calculated by calculating the frequency distribution of the image information of the contour region in the HSV color space, grouping it based on hue and saturation, and using the number of features and the image feature based on the area in the HSV color space. Grouping based on hue and saturation is done to reduce dependence on brightness so that it is not significantly affected by the shooting conditions.

[0115] Furthermore, in order to speed up processing, similarity such as L2 distance between image feature quantities such as color correlograms and color histograms of image information in an appropriate color space can also be used instead of the signature and EMD.

[0116] The determination of similarity is not limited to the above. The identified object identification information is stored in the object identification information storage unit 22 in association with the photographing date and time information, store information, image information identification information of the photographed image information, image identification information of the upright image information, and external shape identification information.

[0117] For an outline for which object identification information could not be identified, information indicating that the outline area is "empty" (information indicating that the object is not missing or the like) is stored in the object identification information storage unit 22.

[0118] As described above, even when image matching processing is used to identify an object from image information of the outline region, the image matching processing can be performed with high accuracy because the outline region is not a rectangular region. [Example]

[0119] As a modification of the above-described second or third embodiment, a shelf level comparison processor 2143 is provided to detect changes in shelf levels, and if there is no change in shelf levels, the previous recognition result can be used as is. An example of the object recognition processor 213 in this case is shown in FIG.

[0120] The shelf level comparison processing unit 2133 determines that the object identification information of each outer shape of the shelf level in the previous (N-1) upright image information and the image information of the shelf level area in the current (N) upright image information are identical if there is a high degree of similarity between them. As described above, this similarity determination process may be a determination of similarity based on the image feature amount of the image information of the shelf level area in the previous (N-1) upright image information and the image information of the shelf level area in the current (N) upright image information, or may be a determination using EMD between color histograms. Furthermore, it is not limited to these. Then, rather than performing a specification process for each outer shape unit in the object identification processing unit 2132, the object identification processing unit 2132 stores the object identification information of each outer shape of the shelf level in the N-th upright image information as the same as the object identification information of each outer shape of the same shelf level in the N-1th upright image information in the object identification information storage unit 22. This makes it possible to omit processing for shelves that rarely change, such as shelves where there is little movement of objects or shelves that are managed in extremely short cycles. [Example]

[0121] The processes of the above-described first to fourth embodiments can be combined as appropriate. The order of each process is not limited to the order described in the present specification, and can be changed as appropriate to the extent that the object is achieved. The object recognition processing unit 213 performs the process on normalized image information obtained by performing normalization processing on photographed image information, but it may also perform the process on photographed image information. In this case, normalized image information should be read as photographed image information.

[0122] Furthermore, in Examples 2 and 4, instead of specifying the shelf level area in the recognition processing unit 21 and then specifying the outline area (described later) from there, it is also possible to configure the recognition processing unit 21 to specify the outline area (described later) from the entire captured image information, upright image information, or image information of the shelf level area without specifying the shelf level area. In that case, the shelf level specification processing unit 214 may not be provided, and the processing may be configured not to be executed. [Example]

[0123] In the above-described embodiments, the display shelves of convenience stores, supermarkets, etc. have been described as examples, but the present invention is not limited to these, and can be applied to, for example, medicines (objects) displayed on display shelves (medicine shelves) in pharmacies. Similarly, the present invention can be applied to objects displayed on display shelves in warehouses.

[0124] Furthermore, the object may be something other than merchandise displayed on a display shelf. For example, the object may be any of various objects to be identified, such as animals, birds, insects, fish, plants, moving objects, etc. In this way, image information obtained by photographing various objects may be applied to image information other than merchandise displayed on a display shelf. [Industrial Applicability]

[0125] By using the information processing system 1 of the present invention, it is possible to improve the accuracy of identifying objects when identifying displayed objects from image information. [Explanation of symbols]

[0126] 1: Information processing system 2: Management terminal 3: Image information input terminal 20: Learning processing unit 21: Recognition processing section 22: Object identification information storage unit 70: Arithmetic device 71:Storage device 72:Display device 73: Input device 74:Communication equipment 201: First learning processing unit 202: Second learning processing unit 210: Image information input reception processing unit 211: Image information storage unit 212: Image information alignment processing unit 213: Object recognition processing unit 214: Shelf specific processing unit 2131: Outline identification processing section 2132: Object identification processing unit 2133: Shelf comparison processing unit

Claims

1. An information processing system for identifying an object appearing in image information, a first learning processing unit that performs machine learning using first annotation data in which data indicating the external shape of an object is associated with attributes to create a first learning model; a recognition processing unit that identifies object identification information of an object shown in the image information; It has The recognition processing unit Using the image information and the first learning model, an outline of an object shown in the image information is identified as an outline region, and a reliability of an attribute corresponding to the identified outline region is output; If the reliability of the attribute corresponding to the specified outer region is not equal to or less than a predetermined threshold, one or more pieces of object identification information of the object appearing in the specified outer region are identified using a second learning model or image matching process that has been machine-learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information of the object; If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object shown in the specified outline region is identified as an unknown object. An information processing system comprising:

2. The first learning processing unit performing machine learning by image segmentation using the first annotation data to create the first learning model; 2. The information processing system according to claim 1, wherein:

3. The information processing system includes: a second learning processing unit that performs machine learning using the second annotation data to create the second learning model; 3. The information processing system according to claim 1, further comprising:

4. The recognition processing unit performing an image matching process between the image information of the specified outline region and specimen information of the object stored in a specimen information storage unit, thereby identifying object identification information of the object shown in the outline region; 3. The information processing system according to claim 1 or 2.

5. An information processing system for identifying an object appearing in image information, a first learning processing unit that performs machine learning using first annotation data in which data indicating the external shape of an object is associated with attributes, to create a first learning model; It has Using the image information and the first learning model, an outline of an object shown in the image information is identified as an outline region, and a reliability of an attribute corresponding to the identified outline region is output; If the reliability of the attribute corresponding to the specified outer region is not equal to or less than a predetermined threshold, the object identification information of the object shown in the specified outer region is identified using a second learning model or image matching process that has been machine-learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information; If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object shown in the specified outline region is identified as an unknown object. An information processing system comprising:

6. An information processing system for identifying an object appearing in image information, an image information input reception processing unit that receives input of image information obtained by photographing an object; an object recognition processing unit that identifies object identification information of an object shown in the image information from the received input image information or image information obtained by orthogonally positioning the image information; It has The object recognition processing unit using a first learning model created by machine learning using first annotation data in which data indicating the outline of an object is associated with an attribute, and the received input image information or image information obtained by orthogonally positioning the image information, to identify the outline of the object as an outline region, and output a reliability for the attribute corresponding to the identified outline region; If the reliability of the attribute corresponding to the specified outer region is not equal to or less than a predetermined threshold, the object identification information of the object shown in the specified outer region is identified using a second learning model or image matching process that has been machine-learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information; If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object shown in the specified outline region is identified as an unknown object. An information processing system comprising:

7. Computer, a first learning processing unit that performs machine learning using first annotation data in which data indicating the external shape of an object is associated with attributes, to create a first learning model; a recognition processing unit that identifies object identification information of an object shown in the image information; An information processing program that functions as The recognition processing unit Using the image information and the first learning model, an outline of an object shown in the image information is identified as an outline region, and a reliability of an attribute corresponding to the identified outline region is output; If the reliability of the attribute corresponding to the specified outer region is not equal to or less than a predetermined threshold, one or more pieces of object identification information of the object appearing in the specified outer region are identified using a second learning model or image matching process that has been machine-learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information of the object; If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object shown in the specified outline region is identified as an unknown object. An information processing program characterized by:

8. Computer, a first learning processing unit that performs machine learning using first annotation data in which data indicating the external shape of an object is associated with attributes, to create a first learning model; An information processing program that functions as Using image information and the first learning model, an outline of an object shown in the image information is identified as an outline region, and a reliability for an attribute corresponding to the identified outline region is output; If the reliability of the attribute corresponding to the specified outer region is not equal to or less than a predetermined threshold, the object identification information of the object shown in the specified outer region is identified using a second learning model or image matching process that has been machine-learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information; If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object shown in the specified outline region is identified as an unknown object. An information processing program characterized by:

9. Computer, an image information input reception processing unit that receives input of image information obtained by photographing an object; an object recognition processing unit that identifies object identification information of an object shown in the image information from the received input image information or image information obtained by orthogonally positioning the image information; An information processing program that functions as The object recognition processing unit using a first learning model created by machine learning using first annotation data in which data indicating the outline of an object is associated with an attribute, and the received input image information or image information obtained by orthogonally positioning the image information, to identify the outline of the object as an outline region, and output a reliability for the attribute corresponding to the identified outline region; If the reliability of the attribute corresponding to the specified outer region is not equal to or less than a predetermined threshold, the object identification information of the object shown in the specified outer region is identified using a second learning model or image matching process that has been machine-learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information; If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object shown in the specified outline region is identified as an unknown object. An information processing program characterized by:

Images