Object tracking device, object tracking method, and program

Abstract

Provided is an object recognition device, wherein an image acquisition means acquires an image which includes a plurality of objects. An object recognition means acquires recognition results corresponding to each of the plurality of objects included in the image, by performing object recognition processing. A connection relationship identification means identifies, on the basis of the recognition results, the presence or absence of a connection relationship for a plurality of object regions corresponding respectively to the plurality of objects. A region relationship acquisition means acquires region relationship information, which is information about the relationship of the object regions identified as having a connection relationship. A class relationship acquisition means acquires class relationship information, which is information indicating the relationship of a plurality of classes which have been preset in order to obtain the recognition results. A recognition results correction means acquires a plurality of corrected recognition results by correcting the recognition results on the basis of the region relationship information and the class relationship information. An evaluation means acquires final recognition results about the classes to which each of the plurality of objects belong, by evaluating the recognition results using the plurality of corrected recognition results.

Description

[Technical Field]

[0001] The present disclosure relates to recognizing objects in images. [Background technology]

[0002] A method has been proposed for managing product allocation using images of product shelves in a store.

[0003] Specifically, for example, Patent Document 1 discloses a method of recognizing a product represented by one product area image in an image obtained by photographing a product shelf on which multiple products are arranged as a first product, recognizing a product represented by another product area image other than the one product area image as a second product, and further determining the validity of recognizing the one product area image as the first product based on the relevance between the first product and the second product. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. WO2019 / 107157 Summary of the Invention [Problem to be solved by the invention]

[0005] However, according to the viewpoint disclosed in Patent Document 1, the validity of the recognition result is determined based on the relevance between the multiple products, which causes a problem in that the recognition accuracy of each individual product among the multiple products may decrease.

[0006] An object of the present disclosure is to provide an object identification device capable of improving the accuracy of recognizing each of a plurality of objects included in an image. [Means for solving the problem]

[0007] In one aspect of the present disclosure, an object recognition device includes: image capture means for capturing an image including a plurality of objects; an object recognition means for performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; a connection relationship specifying means for performing processing to specify whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects, based on the recognition result obtained by the object recognition processing; an area relationship acquisition means for acquiring area relationship information, which is information relating to the relationship between each object area identified as having the connection relationship; Class relationship information is information indicating relationships between a plurality of classes that are set in advance to obtain the recognition result by the object recognition processing. object name relationship information, which is information indicating whether or not the name of an object when it is assumed that the object actually belongs to one of the plurality of classes matches the name of the object recognized by the object recognition processing, A class relationship acquisition means for acquiring the class relationship; a recognition result correction means for performing a recognition result correction process for correcting the recognition result obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; and an evaluation means for evaluating the recognition results obtained by the object recognition process using the plurality of corrected recognition results, thereby obtaining a final recognition result relating to the class to which each of the plurality of objects belongs. In another aspect of the present disclosure, an object recognition apparatus includes: image capture means for capturing an image including a plurality of objects; an object recognition means for performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; a connection relationship specifying means for performing processing to specify whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects, based on the recognition result obtained by the object recognition processing; an area relationship acquisition means for acquiring area relationship information, which is information relating to the relationship between each object area identified as having the connection relationship; a class relationship acquisition means for acquiring height relationship information, which is information indicating the relationship between a height of an object when it is assumed that the object actually belongs to one of the plurality of classes, and a height of an object recognized by the object recognition processing, as class relationship information, which is information indicating the relationship between a plurality of classes that are preset in order to obtain the recognition result by the object recognition processing; and a recognition result correction means for performing a recognition result correction process for correcting the recognition result obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; and an evaluation means for evaluating the recognition results obtained by the object recognition process using the plurality of corrected recognition results, thereby obtaining a final recognition result relating to the class to which each of the plurality of objects belongs.

[0008] The present disclosure moreover In another aspect, the object recognition method comprises: Acquire an image containing multiple objects; performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; performing a process for identifying whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects based on the recognition result obtained by the object recognition process; acquiring area relationship information that is information relating to the relationship between each of the object areas identified as having the connection relationship; Class relationship information is information indicating relationships between a plurality of classes that are set in advance to obtain the recognition result by the object recognition processing. object name relationship information, which is information indicating whether or not the name of an object when it is assumed that the object actually belongs to one of the plurality of classes matches the name of the object recognized by the object recognition processing, Get performing a recognition result correction process for correcting the recognition result obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; The recognition results obtained by the object recognition process are evaluated using the plurality of corrected recognition results, thereby obtaining a final recognition result relating to the class to which each of the plurality of objects belongs. In yet another aspect of the present disclosure, a method for object recognition includes: Acquire an image containing multiple objects; performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; performing a process for identifying whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects based on the recognition result obtained by the object recognition process; acquiring area relationship information that is information relating to the relationship between each of the object areas identified as having the connection relationship; height relationship information is acquired as class relationship information that indicates a relationship between a plurality of classes that are preset for obtaining the recognition result by the object recognition processing, the height relationship information being information that indicates a relationship between a height of an object when it is assumed that the object actually belongs to one of the plurality of classes and a height of an object recognized by the object recognition processing; performing a recognition result correction process for correcting the recognition results obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; The recognition results obtained by the object recognition process are evaluated using the plurality of corrected recognition results, thereby obtaining a final recognition result relating to the class to which each of the plurality of objects belongs.

[0009] In yet another aspect of the disclosure, a program includes: Acquire an image containing multiple objects; performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; performing a process for identifying whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects based on the recognition result obtained by the object recognition process; acquiring area relationship information that is information relating to the relationship between each of the object areas identified as having the connection relationship; Class relationship information is information indicating relationships between a plurality of classes that are set in advance to obtain the recognition result by the object recognition processing. object name relationship information, which is information indicating whether or not the name of an object when it is assumed that the object actually belongs to one of the plurality of classes matches the name of the object recognized by the object recognition processing, Get performing a recognition result correction process for correcting the recognition results obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; The computer is caused to perform a process of obtaining a final recognition result relating to the class to which each of the plurality of objects belongs by evaluating the recognition result obtained by the object recognition process using the plurality of corrected recognition results. In yet another aspect of the disclosure, a program includes: Acquire an image containing multiple objects; performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; performing a process for identifying whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects based on the recognition result obtained by the object recognition process; acquiring area relationship information that is information relating to the relationship between each of the object areas identified as having the connection relationship; height relationship information is acquired as class relationship information that indicates a relationship between a plurality of classes that are preset for obtaining the recognition result by the object recognition processing, the height relationship information being information that indicates a relationship between a height of an object when it is assumed that the object actually belongs to one of the plurality of classes and a height of an object recognized by the object recognition processing; performing a recognition result correction process for correcting the recognition results obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; The computer is caused to perform a process of obtaining a final recognition result relating to the class to which each of the plurality of objects belongs by evaluating the recognition result obtained by the object recognition process using the plurality of corrected recognition results. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to provide an object identification device that can improve the recognition accuracy of each object among multiple objects included in an image. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing an outline of an object recognition device according to a first embodiment. [Figure 2] FIG. 1 is a block diagram showing the hardware configuration of an object recognition device according to a first embodiment. [Figure 3] FIG. 1 is a block diagram showing the functional configuration of an object recognition device according to a first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of an image used in processing by the object recognition device according to the first embodiment. [Figure 5]3A and 3B are diagrams for explaining a product area and an empty area detected by processing of the object recognition device according to the first embodiment. [Figure 6A] FIG. 2 is a diagram for explaining a process for identifying a connection relationship performed in the object recognition device according to the first embodiment. [Figure 6B] FIG. 2 is a diagram for explaining a process for identifying a connection relationship performed in the object recognition device according to the first embodiment. [Figure 6C] FIG. 2 is a diagram for explaining a process for identifying a connection relationship performed in the object recognition device according to the first embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of attribute information used when acquiring class relationship information. [Figure 8] FIG. 10 is a diagram for explaining an example of information that can be included as class relationship information. [Figure 9] FIG. 10 is a diagram for explaining an example of information that can be included as class relationship information. [Figure 10] 4 is a flowchart for explaining processing performed in the object recognition device according to the first embodiment. [Figure 11] FIG. 10 is a block diagram showing the functional configuration of an object recognition device according to a second embodiment. [Figure 12] 10 is a flowchart for explaining processing performed in an object recognition device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. ~ For convenience, letters with " will be represented as "A~" (where "A" is any letter).

[0013] First Embodiment [Schematic configuration] FIG. 1 is a diagram illustrating an outline of an object recognition device according to a first embodiment. The object recognition device 100 is configured as a portable terminal device such as a tablet terminal. The object recognition device 100 recognizes individual products displayed on a store shelf from an image obtained by photographing the shelf. The object recognition device 100 also acquires, as a final recognition result, a processing result obtained by performing processing such as correction on the recognition results for each product.

[0014] [Hardware configuration] Fig. 2 is a block diagram showing the hardware configuration of the object recognition device according to the first embodiment. As shown in Fig. 2, the object recognition device 100 includes an interface (IF) 111, a processor 112, a memory 113, a recording medium 114, a database (DB) 115, a camera 116, and a touch panel 117.

[0015] The IF 111 inputs and outputs data to and from an external device. Furthermore, the final recognition result obtained by the object recognition device 100 is output to the external device via the IF 111 as necessary.

[0016] The processor 112 is a computer such as a CPU (Central Processing Unit), and executes a program prepared in advance to control the entire object recognition device 100. Specifically, the processor 112 performs processes such as object recognition processing and recognition result correction processing.

[0017] The memory 113 is configured by a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 113 is also used as a working memory while the processor 112 is executing various processes.

[0018] Recording medium 114 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from object recognition device 100. Recording medium 114 records various programs to be executed by processor 112. When object recognition device 100 executes various processes, the programs recorded on recording medium 114 are loaded into memory 113 and executed by processor 112.

[0019] The database 115 stores, for example, information input via the IF 111, processing results obtained by the processing of the processor 112, images obtained by the camera 116, and the like.

[0020] Camera 116 captures an image of a product shelf on which multiple products are displayed. Note that in this embodiment, the multiple products may be placed on fixtures other than product shelves, as long as they are placed at approximately equal distances from camera 116. Also, in this embodiment, camera 116 may be, for example, a depth camera that can acquire both an image and depth information when capturing images, provided in object recognition device 100. If a depth camera is provided in object recognition device 100, appropriate information can be acquired as size information SZJ, which will be described later, even if the multiple products are not placed at approximately equal distances from camera 116.

[0021] The touch panel 117 has a function capable of displaying, for example, an image captured by the camera 116 and a final recognition result obtained by processing by the processor 112. The touch panel 117 also has a function capable of inputting, for example, instructions and information in response to a touch operation by the user.

[0022] [Function configuration] Fig. 3 is a block diagram showing the functional configuration of the object recognition device according to the first embodiment. As shown in Fig. 3, the object recognition device 100 includes an image acquisition unit 21, an object recognition unit 22, a connection relationship identification unit 23, a region relationship acquisition unit 24, a class relationship acquisition unit 25, a recognition result correction unit 26, an evaluation unit 27, and an output unit 28.

[0023] The image acquisition unit 21 acquires an image IMT by photographing a product shelf on which a plurality of products are displayed. Note that the image acquisition unit 21 is not limited to acquiring an image IMT by photographing a product shelf, and may acquire the image IMT from, for example, a database or the like in which a group of images of product shelves are stored in advance.

[0024] The object recognition unit 22 performs object recognition processing on the image IMT using, for example, a trained object recognition model constructed using a neural network or the like, and obtains recognition results corresponding to each of the multiple products included in the image IMT.

[0025] Specifically, the object recognition unit 22 detects rectangular areas corresponding to each of the multiple products included in the image IMT as multiple product areas SA, and detects areas in the image IMT within a certain distance from which no products exist as empty areas EA. Furthermore, based on the extraction results obtained by extracting the positions, sizes, and feature amounts of the multiple product areas SA, the object recognition unit 22 acquires a recognition score, which is a value indicating the likelihood of each class when each of the products included in the multiple product areas SA is classified into one of multiple preset classes.

[0026] The connection relationship identification unit 23 performs processing to identify the presence or absence of a connection relationship between the multiple product areas SA based on the multiple product areas SA detected by the object recognition unit 22. In other words, the connection relationship identification unit 23 performs processing to identify the presence or absence of a connection relationship between the multiple product areas SA corresponding to each of the multiple products based on the recognition result obtained by the object recognition processing of the object recognition unit 22.

[0027] The area relationship acquisition unit 24 analyzes the image IMT to acquire area relationship information ARJ, which is information related to the relationship between each product area SA identified by the connection relationship identification unit 23 as having a connection relationship. Specifically, the area relationship acquisition unit 24 analyzes the image IMT to acquire, for example, area relationship information ARJ related to the relationship between two adjacent product areas SA from among each product area SA identified by the connection relationship identification unit 23 as having a connection relationship.

[0028] The class relationship acquisition unit 25 performs processing to acquire class relationship information CRJ indicating the relationships between multiple classes that are pre-set in order to obtain recognition results in the object recognition unit 22, based on the attribute information ATJ stored in the attribute information storage unit 25a.

[0029] The recognition result correction unit 26 corrects the recognition results obtained by the object recognition unit 22 by performing a recognition result correction process based on the connection relationships of the multiple product areas SA obtained by the connection relationship identification unit 23, the area relationship information ARJ obtained by the area relationship acquisition unit 24, and the class relationship information CRJ obtained by the class relationship acquisition unit 25. Then, the recognition result correction unit 26 acquires multiple corrected recognition results according to the number of classes recognized by the object recognition unit 22 and the number of product areas SA identified by the connection relationship identification unit 23 as having a connection relationship. In other words, the recognition result correction unit 26 acquires multiple corrected recognition results by performing a recognition result correction process for correcting the recognition results obtained by the object recognition process of the object recognition unit 22 based on the area relationship information ARJ and the class relationship information CRJ.

[0030] The evaluation unit 27 uses the multiple corrected recognition results obtained by the recognition result correction unit 26 to perform a process of evaluating the recognition results obtained by the object recognition process of the object recognition unit 22, thereby obtaining a final recognition result related to the class to which each of the multiple products included in the multiple product areas SA belongs.

[0031] The output unit 28 generates a display screen for displaying the final recognition result obtained by the evaluation unit 27, and outputs the generated display screen to a display device. The output unit 28 also outputs data including the final recognition result obtained by the evaluation unit 27 to an external device.

[0032] [Specific example of processing performed in an object recognition device] Next, a specific example of the processing performed in the object recognition device according to the first embodiment will be described.

[0033] The image acquisition unit 21 acquires an image IMT by photographing a product shelf on which a plurality of products are displayed. Specifically, the image acquisition unit 21 acquires, as the image IMT, an image of a state in which products such as bottled drinks are lined up in a row on a product shelf PS, as shown in Fig. 4. Fig. 4 is a diagram showing an example of an image used in processing by the object recognition device according to the first embodiment.

[0034] The object recognition unit 22 detects a plurality of product areas SA and empty areas EA by performing object recognition processing on the image IMT. According to this processing, the product areas SA and empty areas EA are detected as shown in Fig. 5, for example. Fig. 5 is a diagram for explaining the product areas and empty areas detected by processing by the object recognition device according to the first embodiment.

[0035] Furthermore, the object recognition unit 22 acquires a plurality of recognition scores, which are values ​​indicating the probability for each class when each of the products included in the plurality of product areas SA is classified into one of a plurality of preset classes. Specifically, for example, when four classes, classes A to D, are set in advance, the object recognition unit 22 acquires, as a recognition result for one product included in one product area SA, a recognition score RA indicating the probability when the product is classified into class A, a recognition score RB indicating the probability when the product is classified into class B, a recognition score RC indicating the probability when the product is classified into class C, and a recognition score RD indicating the probability when the product is classified into class D. Furthermore, when four classes, classes A to D, are set in advance, the object recognition unit 22 acquires the recognition scores RA to RD for all product areas SA detected in the image IMT. Note that, according to this embodiment, the object recognition unit 22 performs a process of adjusting the range of output values ​​output through the object recognition process using a softmax function or the like. Therefore, in this embodiment, the description will be given assuming that the total value of the multiple recognition scores acquired by the object recognition unit 22 is 1, and that each of the multiple recognition scores has a value of 0 or more and 1 or less.

[0036] The connection relationship specification unit 23 performs processing to specify whether or not there is a connection relationship between the multiple product areas SA based on the multiple product areas SA and the empty areas EA detected by the object recognition unit 22.

[0037] Here, for example, as shown in Fig. 6A, a process of the connection relationship identification unit 23 will be described when a product area SAK corresponding to a product K placed on a product shelf PS and a product area SAL corresponding to a product L placed on the same shelf board as the product K on the product shelf PS are detected by the object recognition unit 22. Figs. 6A to 6C are diagrams for explaining the process related to identification of the connection relationship performed in the object recognition device according to the first embodiment.

[0038] First, the connection relationship identification unit 23 sets a rectangular area SAKA having the same size as the product area SAK at a position adjacent to the product area SAK. Specifically, the connection relationship identification unit 23 sets the rectangular area SAKA at a position adjacent to the right side of the product area SAK, for example, as shown in FIG. 6B .

[0039] Next, the connection relationship identification unit 23 detects an overlapping area TRA where the product area SAL and the rectangular area SAKA overlap, and calculates a ratio RKL of the overlapping area TRA to the product area SAL. The overlapping area TRA is represented as an area such as that shown in FIG. 6C, for example.

[0040] Thereafter, the connection relationship identifying unit 23 determines whether or not the product area SAK and the product area SAL are adjacent to each other based on the ratio RKL and the threshold value THA.

[0041] If the ratio RKL is less than the threshold value THA, the connection relationship identification unit 23 determines that the product area SAK and the product area SAL are not adjacent to each other. Specifically, if the ratio RKL calculated according to the overlapping area TRA in Fig. 6C is less than the threshold value THA, the connection relationship identification unit 23 determines that the product area SAK and the product area SAL are not adjacent to each other on the right side of the product area SAK. When such a determination is made, the connection relationship identification unit 23 identifies that the product area SAK and the product area SAL arranged on the product shelf PS do not have a connection relationship in the left-right direction of the product shelf PS.

[0042] Furthermore, if the ratio RKL is equal to or greater than the threshold value THA, the connection relationship identification unit 23 determines that the product area SAK and the product area SAL are adjacent in the direction in which the rectangular area SAKA is set. Specifically, if the ratio RKL calculated according to the overlapping area TRA in Fig. 6C is equal to or greater than the threshold value THA, the connection relationship identification unit 23 determines that the product area SAK and the product area SAL are adjacent to each other on the right side of the product area SAK. If such a determination is made, the connection relationship identification unit 23 identifies that the product area SAK and the product area SAL arranged on the product shelf PS have a connection relationship in the left-right direction of the product shelf PS.

[0043] In addition, for example, if there is an empty area EA between the product area SAK and the product area SAL, the connection relationship identification unit 23 may identify that the product area SAK and the product area SAL do not have a connection relationship (in the direction of the empty area EA) without performing the processing described above.

[0044] The area relationship acquisition unit 24 analyzes the image IMT to acquire area relationship information ARJ relating to the relationship between two adjacent product areas SA among each product area SA that have been identified by the connection relationship identification unit 23 as having a connection relationship.

[0045] Specifically, the area relationship acquisition unit 24 acquires, as area relationship information ARJ corresponding to the product areas SAK and SAL, for example, of each product area SA identified by the connection relationship identification unit 23 as having a connection relationship, appearance similarity information GSJ, which is information related to the similarity of appearance, and size information SZJ, which is information related to the relative size relationship.

[0046] The area relationship acquisition unit 24 acquires, as the appearance similarity information GSJ, an appearance similarity GSD, which is a value indicating the similarity between, for example, a feature vector SAKV calculated based on the color and pattern of product K included in the product area SAK and a feature vector SALV calculated based on the color and pattern of product L included in the product area SAL. In this embodiment, the appearance similarity GSD is acquired as a cosine similarity taking a value in the range of 0 to 1. Therefore, for example, when the feature vectors SAKV and SALV are close to each other, i.e., when the product areas SAK and SAL are similar to each other, the appearance similarity GSD is acquired as a relatively large value. On the other hand, when the feature vectors SAKV and SALV are far from each other, i.e., when the product areas SAK and SAL are dissimilar to each other, the appearance similarity GSD is acquired as a relatively small value.

[0047] The area relationship acquisition unit 24 acquires information related to a comparison result obtained by comparing, as size information SZJ, for example, the vertical height HK of the product area SAK and the vertical height HL of the product area SAL. Specifically, the area relationship acquisition unit 24 acquires, as size information SZJ, information indicating any one of, for example, the height HK being greater than the height HL (HK>HL), the height HK and the height HL being equal (HK=HL), or the height HK being less than the height HL (HK<HL). According to the present embodiment, the area relationship acquisition unit 24, for example, when the products K and L are arranged on the same shelf board of the product shelf PS and at least a lower portion of at least one of the products K and L is hidden by a shield such as an advertisement and a price tag, may acquire information related to the result of comparing the coordinate value of the uppermost part in the vertical direction of the product area SAK and the coordinate value of the uppermost part in the vertical direction of the product area SAL as the size information SZJ.

[0048] The class relationship acquisition unit 25 performs a process for acquiring class relationship information CRJ indicating the relationships of classes A to D preset for obtaining a recognition result in the object recognition unit 22 based on the attribute information ATJ stored in the attribute information storage unit 25a.

[0049] The attribute information ATJ may be created, for example, as information as shown in FIG. 7 when four classes A to D are set in the object recognition unit 22. FIG. 7 is a diagram for explaining an example of the attribute information used when acquiring the class relationship information.

[0050] According to the "product name" of the attribute information ATJ, it is shown that the name of the product belonging to class A is "PNA", the name of the product belonging to class B is "PNB", the name of the product belonging to class C is "PNC", and the name of the product belonging to class D is "PND". Furthermore, according to the "height" of the attribute information ATJ, it is shown that the height of the product belonging to class A is "15 cm", the height of the product belonging to class B is "15 cm", the height of the product belonging to class C is "18 cm", and the height of the product belonging to class D is "8 cm".

[0051] When the attribute information ATJ of Fig. 7 is stored in the attribute information storage unit 25a, the class relationship acquisition unit 25 performs processing to acquire, as class relationship information CRJ, for example, product name relationship information NRJ as shown in Fig. 8 and height relationship information HRJ as shown in Fig. 9. Figs. 8 and 9 are diagrams for explaining examples of information that can be included as class relationship information.

[0052] The product name relationship information NRJ in Fig. 8 corresponds to information indicating whether or not the name of a product assumed to actually belong to one of the four classes A to D matches the name of a product recognized by the object recognition unit 22. Specifically, the product name relationship information NRJ in Fig. 8 indicates, for example, that the name of a product assumed to actually belong to class A matches the name of a product recognized by the object recognition unit 22 as belonging to class A, and does not match the name of a product recognized by the object recognition unit 22 as belonging to any of class B, class C, and class D.

[0053] According to this embodiment, the class relationship acquisition unit 25 may acquire, as the product name relationship information NRJ, the value of the inter-class similarity CSD calculated based on images of products belonging to classes A to D, instead of the information shown in Fig. 8. Details of the inter-class similarity CSD will be described later.

[0054] The height relationship information HRJ in Fig. 9 corresponds to information indicating the relationship between the height of a product when it is assumed that the product actually belongs to one of the four classes A to D, and the height of a product recognized by the object recognition unit 22. Specifically, the height relationship information HRJ in Fig. 9 indicates, for example, that the height of a product assumed to actually belong to class A is the same as the height of a product recognized by the object recognition unit 22 as being either class A or class B, is smaller than the height of a product recognized by the object recognition unit 22 as being class C, and is larger than the height of a product recognized by the object recognition unit 22 as being class D.

[0055] The recognition result correction unit 26 corrects the recognition result obtained by the object recognition unit 22 by performing a recognition result correction process based on the connection relationships of the multiple product areas SA obtained by the connection relationship identification unit 23, the area relationship information ARJ obtained by the area relationship acquisition unit 24, and the class relationship information CRJ obtained by the class relationship acquisition unit 25.

[0056] Here, a specific example of the recognition result correction process will be described. In the following, a case will be described in which, for product areas SAK and SAL identified by the connection relationship identification unit 23 as having a connection relationship, the recognition result of product L included in the product area SAL is corrected using the recognition result of product K included in the product area SAK as a reference. In the following, it is assumed that the object recognition unit 22 has acquired recognition scores RLA, RLB, RLC, and RLD, which indicate the likelihood that the product L will be classified into classes A to D, as a recognition result of the product L included in the product area SAL. In the following, it is assumed that area relationship information ARJ, which includes appearance similarity information GSJ and size information SZJ, has been acquired. In the following, it is assumed that class relationship information CRJ, which includes product name relationship information NRJ illustrated in FIG. 8 and height relationship information HRJ illustrated in FIG. 9, has been acquired.

[0057] First, a process for obtaining a correction value based on the appearance similarity information GSJ included in the area relationship information ARJ and the product name relationship information NRJ included in the class relationship information CRJ will be described. Note that, hereinafter, the correction values ​​obtained by such a process will be collectively referred to as the correction value HVA.

[0058] The recognition result correction unit 26 acquires a correction value HVA for correcting the recognition scores RLA, RLB, RLC, and RLD based on the appearance similarity information GSJ and the product name relationship information NRJ, assuming that, for example, product K actually belongs to class A.

[0059] Specifically, for example, when the appearance similarity GSD included in the appearance similarity information GSJ is a large value (1 or a value close to 1), the recognition result correction unit 26 determines, based on the product name relationship information NRJ, that the probability that the product L belongs to class A is high and the probability that the product L belongs to any of classes B to D is low. When making such a determination, the recognition result correction unit 26 acquires 0 as the correction value HVA when the product K belongs to class A and the product L also belongs to class A. When making the determination as described above, the recognition result correction unit 26 acquires a value (-GSD) obtained by converting the appearance similarity GSD into a negative value as the correction value HVA when the product K belongs to class A and the product L belongs to any of classes B to D.

[0060] Furthermore, for example, when the appearance similarity GSD included in the appearance similarity information GSJ is a small value (0 or a value close to 0), the recognition result correction unit 26 determines, based on the product name relationship information NRJ, that the probability that the product L belongs to class A is low and that the product L is highly likely to belong to one of classes B to D. When making such a determination, the recognition result correction unit 26 acquires a value (-GSD) obtained by converting the appearance similarity GSD into a negative value as the correction value HVA when the product K belongs to class A and the product L also belongs to class A. When making the determination as described above, the recognition result correction unit 26 acquires 0 as the correction value HVA when the product K belongs to class A and the product L belongs to one of classes B to D.

[0061] The recognition result correction unit 26 performs processing similar to that described above to obtain correction values ​​HVA for correcting the recognition scores RLA, RLB, RLC, and RLD for each of the cases where it is assumed that the product K actually belongs to class B, where it is assumed that the product K actually belongs to class C, and where it is assumed that the product K actually belongs to class D.

[0062] That is, according to the processing described above, the recognition result correction unit 26 acquires a correction value HVA for correcting the recognition score obtained by the object recognition processing of the object recognition unit 22, based on the appearance similarity information GSJ and the product name relationship information NRJ. Furthermore, according to the processing described above, if there is no contradiction between the magnitude of the appearance similarity GSD value and the relationship between classes A to D indicated by the product name relationship information NRJ, 0 is acquired as the correction value HVA. Furthermore, according to the processing described above, if there is a contradiction between the magnitude of the appearance similarity GSD value and the relationship between classes A to D indicated by the product name relationship information NRJ, -GSD is acquired as the correction value HVA.

[0063] According to the present embodiment, the recognition result correction unit 26 may acquire the correction value HVA using a function that calculates a value of 0 or less corresponding to the difference between the value of the inter-class similarity CSD obtained based on the relationship between classes A to D indicated by the product name relationship information NRJ or the like and the value of the appearance similarity GSD. The inter-class similarity CSD may be set to 1 when the product names of products K and L match, and to 0 when the product names of products K and L do not match. Alternatively, the inter-class similarity CSD may be set to a value of 0 or more and 1 or less that indicates the similarity between two feature vectors among a feature vector calculated from an image of a product that actually belongs to class A, a feature vector calculated from an image of a product that actually belongs to class B, a feature vector calculated from an image of a product that actually belongs to class C, and a feature vector calculated from an image of a product that actually belongs to class D. Furthermore, the above-mentioned function may be realized, for example, by using a machine learning model configured with a neural network and trained in advance to output a correction value HVA in response to inputs of the comparison results of product names in the product areas SAK and SAL and the relationships between classes A to D indicated by the product name relationship information NRJ. According to the above-described configuration, for example, when CSD = 0.7 and GSD = 0.7, the correction value HVA can be obtained as 0. According to the above-described configuration, for example, when CSD = 0.7 and GSD = 0.8, the correction value HVA can be obtained as -0.1. According to the above-described configuration, for example, when CSD = 0.7 and GSD = 0.4, the correction value HVA can be obtained as -0.3.

[0064] Next, we will explain the process of obtaining a correction value based on the size information SZJ included in the area relationship information ARJ and the height relationship information HRJ included in the class relationship information CRJ. Note that, hereinafter, the correction values ​​obtained by such a process will be collectively referred to as the correction value HVB.

[0065] Based on the size information SZJ and the height relationship information HRJ, the recognition result correction unit 26 obtains a correction value HVB for correcting the recognition scores RLA, RLB, RLC, and RLD, for example, assuming that product K actually belongs to class A.

[0066] Specifically, for example, when the size information SZJ includes information indicating that HK=HL, the recognition result correction unit 26 determines, based on the height relationship information HRJ, that there is a high probability that product L belongs to either class A or B, and that there is a low probability that product L belongs to either class C or D. When making such a determination, the recognition result correction unit 26 acquires 0 as the correction value HVB when product K belongs to class A and product L belongs to either class A or B. When making the above-mentioned determination, the recognition result correction unit 26 acquires −1 as the correction value HVB when product K belongs to class A and product L belongs to either class C or D.

[0067] Furthermore, when the size information SZJ includes information indicating that HK>HL, for example, the recognition result correction unit 26 determines, based on the height relationship information HRJ, that there is a high probability that the product L belongs to class D and that there is a low probability that the product L belongs to any of classes A, B, or C. When making such a determination, the recognition result correction unit 26 acquires 0 as the correction value HVB when the product K belongs to class A and the product L belongs to class D. When making the above-mentioned determination, the recognition result correction unit 26 acquires −1 as the correction value HVB when the product K belongs to class A and the product L belongs to any of classes A to C.

[0068] Further, when information indicating that HK < HL, for example, is included in the size information SZJ, the recognition result correction unit 26 determines that, based on the height relationship information HRJ, the probability that the product L belongs to the class C is high, and the probability that the product L belongs to any of the classes A, B, or D is low. When such a determination is made, the recognition result correction unit 26 obtains 0 as the correction value HVB when the product K belongs to the class A and the product L belongs to the class C. Further, when such a determination is made, the recognition result correction unit 26 obtains -1 as the correction value HVB when the product K belongs to the class A and the product L belongs to any of the classes A, B, or D.

[0069] The recognition result correction unit 26 performs the same processing as described above to obtain correction values HVB for correcting the recognition scores RLA, RLB, RLC, and RLD for the cases where the product K is assumed to actually belong to the class B, the product K is assumed to actually belong to the class C, and the product K is assumed to actually belong to the class D.

[0070] That is, according to the processing described above, the recognition result correction unit 26 obtains a correction value HVB for correcting the recognition score obtained by the object recognition processing of the object recognition unit 22 based on the size information SZJ and the height relationship information HRJ. Further, according to the processing as described above, when there is no contradiction between the comparison result of the heights HK and HL included in the size information SZJ and the relationship of the classes A to D indicated by the height relationship information HRJ, is obtained as the correction value HVB. Further, according to the processing as described above, when there is a contradiction between the comparison result of the heights HK and HL included in the size information SZJ and the relationship of the classes A to D indicated by the height relationship information HRJ, -1 is obtained as the correction value HVB.

[0071] According to this embodiment, the recognition result correction unit 26 may acquire the correction value HVB using a function that calculates a value greater than or equal to 0 when there is no contradiction and calculates a negative value when there is a contradiction. The function may be implemented using a machine learning model that includes a neural network and is trained in advance to output the correction value HVB according to the input of the comparison result of the sizes of the product areas SAK and SAL and the relationship between classes A to D indicated by the height relationship information HRJ.

[0072] The recognition result correction unit 26 corrects the recognition result obtained by the object recognition unit 22 by performing a recognition result correction process using the correction values ​​HVA and HVB.

[0073] Specifically, as the recognition result correction process, the recognition result correction unit 26 performs a process of adding correction values ​​HVA and HVB to the recognition scores RLA, RLB, RLC, and RLD, respectively. This process obtains a corrected recognition score ARLA equivalent to RLA+HVA+HVB, a corrected recognition score ARLB equivalent to RLB+HVA+HVB, a corrected recognition score ARLC equivalent to RLC+HVA+HVB, and a corrected recognition score ARLD equivalent to RLD+HVA+HVB. The corrected recognition scores ARLA to ARLD are obtained for each class to which the product K is assumed to belong. That is, the recognition result correction unit 26 obtains the corrected recognition scores ARLA to ARLD for each of classes A to D to which the product K is assumed to belong.

[0074] The evaluation unit 27 uses the multiple corrected recognition results obtained by the recognition result correction unit 26 to perform a process of evaluating the recognition results obtained by the object recognition unit 22, thereby obtaining a final recognition result related to the class to which each of the products included in the multiple product areas SA belongs.

[0075] Here, a specific example of processing related to evaluation of the recognition result obtained by the object recognition unit 22 will be described. Note that the following description will be given on the assumption that the object recognition unit 22 has acquired recognition scores RKA, RKB, RKC, and RKD indicating the likelihood that the product K will be classified into classes A to D as a recognition result of the product K included in the product area SAK. Furthermore, the following description will mainly focus on a case where processing is performed using the recognition scores RKA to RKD and the corrected recognition scores ARLA to ARLD.

[0076] For example, the evaluation unit 27 performs a process of adding the recognition score RKA to each of the corrected recognition scores ARLA to ARLD obtained when it is assumed that the product K actually belongs to class A. Then, according to this process, an evaluation value EVAA corresponding to RKA+ARLA, an evaluation value EVAB corresponding to RKA+ARLB, an evaluation value EVAC corresponding to RKA+ARLC, and an evaluation value EVAD corresponding to RKA+ARLD are obtained.

[0077] Furthermore, the evaluation unit 27 performs a process of adding the recognition score RKB to each of the corrected recognition scores ARLA to ARLD obtained when it is assumed that the product K actually belongs to class B. Then, according to this process, an evaluation value EVBA corresponding to RKB+ARLA, an evaluation value EVBB corresponding to RKB+ARLB, an evaluation value EVBC corresponding to RKB+ARLC, and an evaluation value EVBD corresponding to RKB+ARLD are obtained.

[0078] Furthermore, the evaluation unit 27 performs a process of adding the recognition score RKC to each of the corrected recognition scores ARLA to ARLD obtained when it is assumed that the product K actually belongs to class C. Then, according to this process, an evaluation value EVCA corresponding to RKC+ARLA, an evaluation value EVCB corresponding to RKC+ARLB, an evaluation value EVCC corresponding to RKC+ARLC, and an evaluation value EVCD corresponding to RKC+ARLD are acquired.

[0079] Furthermore, the evaluation unit 27 performs a process of adding the recognition score RKD to each of the corrected recognition scores ARLA to ARLD obtained when it is assumed that the product K actually belongs to class D. Then, according to this process, an evaluation value EVDA corresponding to RKD+ARLA, an evaluation value EVDB corresponding to RKD+ARLB, an evaluation value EVDC corresponding to RKD+ARLC, and an evaluation value EVDD corresponding to RKD+ARLD are acquired.

[0080] The evaluation unit 27 compares the 16 evaluation values ​​EVAA-EVAD, EVBA-EVBD, EVCA-EVCD, and EVDA-EVDD obtained by the above-described processing to identify the evaluation value EVM having the largest value. Then, the evaluation unit 27 obtains information indicating the classes of products K and L corresponding to the evaluation value EVM as the final recognition result. Specifically, for example, if EVM=EVAB, the evaluation unit 27 obtains information indicating that product K belongs to class A and product L belongs to class B as the final recognition result.

[0081] On the other hand, in this embodiment, when, for example, N (N≧2) products are arranged in a horizontal row on a product shelf PS, that is, when N product areas have a connection relationship in the left-right direction, the evaluation unit 27 performs processing using dynamic programming according to the following formulas (1) and (2) to obtain, as the final recognition result, information indicating the class of each of the N products that maximizes the evaluation value EV of the following formula (1).

[0082]

number

number

[0083] In the above formula (1), x1 indicates the class to which the first product from the left on the product shelf PS is assumed to actually belong, and s1(x1) indicates the recognition score corresponding to the class of x1. Also, in the above formulas (1) and (2), s~j (x j-1 ,x j ) indicates the corrected recognition score for the combination of the class to which the j-1 (2≦j≦N)th product from the left of the product shelf PS is actually estimated to belong and the class to which the jth product from the left of the product shelf PS is recognized to belong. Also, in the above formula (2), x j indicates the class to which the jth (2≦j≦N) product from the left on the shelf PS is estimated to belong, and s j (x j ) is the x j In addition, in the above formula (2), h size (x j-1 ,x j ) corresponds to the correction value HVB calculated by applying the above-mentioned method to the j-1th and jth products from the left of the product shelf. sim (x j-1 ,x j ) corresponds to the correction value HVA calculated by applying the above-mentioned method to the j-1th and jth products from the left on the shelf.

[0084] Here, an outline of the process using dynamic programming using the above equations (1) and (2) will be explained.

[0085] First, the evaluation unit 27 performs processing using the above formulas (1) and (2) for the first product SH1 from the left on the product shelf PS and the second product SH2 from the left on the product shelf PS to obtain 16 evaluation values ​​EV similar to the above-mentioned evaluation values ​​EVAA to EVDD, and identifies the evaluation value EVM having the largest value among the 16 evaluation values ​​EV. Then, for example, if EVM=EVAB, the evaluation unit 27 obtains an estimation result that product SH1 belongs to class A and product SH2 belongs to class B.

[0086] Next, the evaluation unit 27 performs processing using the above formulas (1) and (2) on the product SH2 and the product SH3, which is third from the left on the product shelf PS. When performing processing using the above formulas (1) and (2), if the evaluation unit 27 has previously obtained an estimation result that the product SH2 belongs to class B, for example, the evaluation unit 27 obtains four evaluation values ​​EV similar to the above-mentioned evaluation values ​​EVBA to EVBD, and identifies the evaluation value EVM having the largest value among the four evaluation values ​​EV. Then, if EVM=EVBD, for example, the evaluation unit 27 obtains an estimation result that the product SH3 belongs to class D.

[0087] Thereafter, the evaluation unit 27 performs the processing using the above formulas (1) and (2) sequentially from the left product to the right product on the product shelf PS, thereby obtaining estimation results relating to the class to which each of the N products placed on the product shelf PS belongs.

[0088] That is, according to the processing described above, the evaluation unit 27 acquires as the final recognition result the estimation result relating to the class to which each of the N products placed on the product shelf PS belongs, which is acquired so that the evaluation value EV of the above formula (1) becomes the maximum value.

[0089] The output unit 28 generates a display screen for displaying the final recognition result obtained by the evaluation unit 27, and outputs the generated display screen to a display device. The output unit 28 also outputs data including the final recognition result obtained by the evaluation unit 27 to an external device.

[0090] In this embodiment, for the class to which each of the N products arranged on the product shelf PS belongs, the class that corresponds to the final recognition result obtained by the evaluation unit 27 is not necessarily displayed, but for example, the class that corresponds to the recognition result before correction obtained by the object recognition unit 22 may also be displayed.

[0091] In addition, in this embodiment, a display screen may be displayed that enables correction of the final recognition result for the class to which each of the N products arranged on the product shelf PS belongs, for example, based on the user's subjective opinion or on the processing results obtained by performing processing such as character recognition. Furthermore, in this embodiment, when the final recognition result is corrected, for example, the processing of the recognition result correction unit 26 and the evaluation unit 27 may be performed again with the class of each product that has been corrected in the corrected recognition result fixed.

[0092] In addition, in this embodiment, when the final recognition result is corrected based on the user's subjective opinion, a re-corrected recognition result obtained by further correcting the corrected recognition result through processing such as character recognition may be displayed. Furthermore, in this embodiment, a dialog box or the like may be displayed that allows the user to decide whether or not to accept the re-corrected recognition result.

[0093] [Processing flow] Next, the flow of processing performed in the object recognition device will be described. Fig. 10 is a flowchart for explaining processing performed in the object recognition device according to the first embodiment.

[0094] First, the image acquisition unit 21 captures an image of a product shelf on which a plurality of products are displayed (step S11).

[0095] Next, the object recognition unit 22 performs object recognition processing on the image obtained in step S11 to obtain a recognition result corresponding to each of the multiple products included in the image (step S12). Specifically, the recognition result includes, for example, multiple product areas and a recognition score, which is a value indicating the probability for each class when the products included in the multiple product areas are classified into one of multiple preset classes.

[0096] Next, the connection relationship identifying unit 23 performs a process for identifying whether or not there is a connection relationship between a plurality of product regions in the recognition result obtained in step S12 (step S13).

[0097] Next, based on the image obtained in step S11, the area relationship acquisition unit 24 acquires area relationship information relating to the relationship between two adjacent product areas from among the product areas identified as having a connection relationship in step S13 (step S14).

[0098] Next, the class relationship acquisition unit 25 performs processing to acquire class relationship information indicating the relationships between multiple classes that are set in advance to obtain a recognition result by the object recognition processing of step S12, based on the attribute information stored in the attribute information storage unit 25a (step S15).

[0099] Next, the recognition result correction unit 26 corrects the recognition score included in the recognition result obtained in step S12 by performing a recognition result correction process based on the connection relationships of the multiple product areas identified in step S13, the area relationship information obtained in step S14, and the class relationship information obtained in step S15 (step S16). According to this process, the recognition result correction unit 26 obtains multiple corrected recognition results according to the number of classes recognized by the object recognition unit 22 and the number of product areas SA identified by the connection relationship identification unit 23 as having a connection relationship.

[0100] Next, the evaluation unit 27 performs a process of evaluating the recognition results obtained in step S12 using the multiple corrected recognition results obtained in step S16, thereby obtaining a final recognition result related to the class to which each of the products included in the multiple product areas belongs (step S17).

[0101] Finally, the output unit 28 outputs the final recognition result obtained in step S17 to the display device, an external device, etc. (step S18).

[0102] As described above, according to this embodiment, object recognition processing is performed on an image including multiple objects to obtain recognition results for the multiple objects, the recognition results obtained by the object recognition processing are corrected based on area relationship information and class relationship information to obtain multiple corrected recognition results, and the recognition results obtained by the object recognition processing are evaluated using the multiple corrected recognition results to obtain a final (optimized) recognition result. Therefore, according to this embodiment, it is possible to improve the recognition accuracy of each of multiple objects included in an image.

[0103] [Variations]

[0104] Modifications of the above embodiment will be described below. For simplicity, specific descriptions of parts to which the above-described processes can be applied will be omitted as appropriate.

[0105] (Variation 1) The connection relationship determination unit 23 may, for example, perform processing similar to that described above when a rectangular area SAKA is set at a position adjacent to the upper or lower side of the product area SAK, thereby determining whether the product area SAK and the product area SAL, which are arranged on either side of the shelf board of the product shelf PS, have a connection relationship in the vertical direction of the product shelf PS.

[0106] (Variation 2) The recognition result correction unit 26 may be configured as, for example, a trained machine learning model having a graph convolutional neural network, and may be configured to output a correction value according to graph data input to the machine learning model. Furthermore, the graph data may be configured as data in which, for example, a plurality of nodes corresponding to a plurality of products included in the image IMT are connected by edges, and information such as appearance similarity information GSJ and size information SZJ indicating the relationship between a plurality of product regions corresponding to the plurality of products is embedded as edge features.

[0107] (Variation 3) For example, when N products are arranged in a row on a product shelf PS, i.e., when N product areas are connected in the left-right direction, the evaluation unit 27 may perform processing using dynamic programming according to the following formulas (3) and (4) to obtain, as the final recognition result, information indicating the class of each of the N products that minimizes the cost value CV of the following formula (1).

[0108]

number

number

[0109] In the above formula (3), x1 indicates the class to which the first product from the left on the product shelf PS is assumed to actually belong, and r1(x1) indicates the value obtained by subtracting the recognition score corresponding to the class of x1 from 1.0. Also, in the above formulas (3) and (4), h~ k (x k-1 ,x k ) indicates the corrected recognition score for the combination of the class to which the k-1 (2≦k≦N)th product from the left of the product shelf PS is actually estimated to belong and the class to which the kth product from the left of the product shelf PS is recognized to belong. Also, in the above formula (4), x k indicates the class to which the kth product from the left on the shelf PS is estimated to belong, and r k (x k ) is 1.0 to the corresponding x k In addition, in the above formula (4), M size (x k-1 ,x k ) indicates a correction value that is 0 when the heights (sizes) of the k-1th and kth products from the left on the product shelf PS match, and is 1 when the heights (sizes) do not match. Also, in the above formula (4), M sim (x k-1 ,x k) indicates a correction value that is 0 for the k-1st and kth products from the left on the product shelf PS if they belong to the same class, and is a value according to the following formula (5) if they belong to different classes.

[0110]

number

[0111] In the above formula (5), (φ k-1 ,φ k ) indicates the angle between the feature vector of the product area corresponding to the k-1th product from the left on the product shelf PS and the feature vector of the product area corresponding to the kth product from the left on the product shelf PS.

[0112] Second Embodiment FIG. 11 is a block diagram showing the functional configuration of an object recognition device according to the second embodiment.

[0113] The object recognition device 100A according to this embodiment has the same hardware configuration as the object recognition device 100. The object recognition device 100A also has an image acquisition means 41, an object recognition means 42, a connection relationship identification means 43, an area relationship acquisition means 44, a class relationship acquisition means 45, a recognition result correction means 46, and an evaluation means 47.

[0114] FIG. 12 is a flowchart illustrating processing performed in the object recognition device according to the second embodiment.

[0115] The image acquisition means 41 acquires an image including a plurality of objects (step S41).

[0116] The object recognition means 42 performs object recognition processing on the image to obtain recognition results corresponding to each of a plurality of objects included in the image (step S42).

[0117] The connection relationship specifying means 43 performs processing to specify whether or not there is a connection relationship between a plurality of object regions corresponding to each of a plurality of objects, based on the recognition result obtained by the object recognition processing (step S43).

[0118] The area relation obtaining means 44 obtains area relation information, which is information relating to the relation between each object area identified as having a connection relation (step S44).

[0119] The class relationship acquisition means 45 acquires class relationship information, which is information indicating relationships between a plurality of classes that are set in advance in order to obtain a recognition result through object recognition processing (step S45).

[0120] The recognition result correction means 46 performs a recognition result correction process to correct the recognition results obtained by the object recognition process based on the area relationship information and class relationship information, thereby obtaining multiple corrected recognition results (step S46).

[0121] The evaluation means 47 evaluates the recognition results obtained by the object recognition process using the plurality of corrected recognition results, thereby obtaining a final recognition result relating to the class to which each of the plurality of objects belongs (step S47).

[0122] According to this embodiment, it is possible to improve the accuracy of recognizing each of a plurality of objects included in an image.

[0123] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0124] (Appendix 1) image capture means for capturing an image including a plurality of objects; an object recognition means for performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; a connection relationship specifying means for performing processing to specify whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects, based on the recognition result obtained by the object recognition processing; an area relationship acquisition means for acquiring area relationship information, which is information relating to the relationship between each object area identified as having the connection relationship; a class relationship acquisition means for acquiring class relationship information that indicates relationships between a plurality of classes that are set in advance to obtain the recognition result by the object recognition processing; a recognition result correction means for performing a recognition result correction process for correcting the recognition result obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; an evaluation means for evaluating the recognition results obtained by the object recognition processing using the plurality of corrected recognition results, thereby obtaining a final recognition result relating to a class to which each of the plurality of objects belongs; An object recognition device comprising:

[0125] (Appendix 2) The object recognition device of claim 1, wherein the object recognition means performs the object recognition processing on the image, and obtains, as the recognition result, a recognition score, which is a value indicating the probability for each class when each of the plurality of objects is classified into one of the plurality of classes.

[0126] (Appendix 3) The object recognition device of claim 2, wherein the region relationship acquisition means acquires, as the region relationship information corresponding to two object regions having the connection relationship, appearance similarity information, which is information related to the similarity in appearance between objects included in the two object regions.

[0127] (Appendix 4) The object recognition device of claim 3, wherein the class relationship acquisition means acquires, as the class relationship information, object name relationship information which is information indicating whether or not a name of an object when it is assumed that the object actually belongs to one of the plurality of classes matches a name of the object recognized by the object recognition processing.

[0128] (Appendix 5) 5. The object recognition device according to claim 4, wherein the recognition result correction means obtains a correction value for correcting the recognition score obtained by the object recognition process based on the appearance similarity information and the object name relationship information.

[0129] (Appendix 6) The object recognition device of Appendix 2, wherein the area relationship acquisition means acquires size information, which is information relating to the relative size relationship between objects included in the two object areas, as the area relationship information corresponding to the two object areas having the connection relationship.

[0130] (Appendix 7) The object recognition device of Appendix 6, wherein the class relationship acquisition means acquires, as the class relationship information, height relationship information which is information indicating the relationship between the height of an object when it is assumed that the object actually belongs to one of the plurality of classes and the height of the object recognized by the object recognition processing.

[0131] (Appendix 8) 8. The object recognition device according to claim 7, wherein the recognition result correction means obtains a correction value for correcting the recognition score obtained by the object recognition process based on the size information and the height relationship information.

[0132] (Appendix 9) Acquire an image containing multiple objects; performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; performing a process for identifying whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects based on the recognition result obtained by the object recognition process; acquiring area relationship information that is information relating to the relationship between each of the object areas identified as having the connection relationship; acquiring class relationship information that indicates relationships between a plurality of classes that are set in advance to obtain the recognition result by the object recognition processing; performing a recognition result correction process for correcting the recognition results obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; An object recognition method that obtains a final recognition result relating to the class to which each of the plurality of objects belongs by evaluating the recognition result obtained by the object recognition processing using the plurality of corrected recognition results.

[0133] (Appendix 10) Acquire an image containing multiple objects; performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; performing a process for identifying whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects based on the recognition result obtained by the object recognition process; acquiring area relationship information that is information relating to the relationship between each of the object areas identified as having the connection relationship; acquiring class relationship information that indicates relationships between a plurality of classes that are set in advance to obtain the recognition result by the object recognition processing; performing a recognition result correction process for correcting the recognition results obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; A recording medium having a program recorded thereon that causes a computer to execute a process of obtaining a final recognition result relating to the class to which each of the plurality of objects belongs by evaluating the recognition results obtained by the object recognition process using the plurality of corrected recognition results.

[0134] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. [Explanation of symbols]

[0135] 21 Image acquisition unit 22 Object recognition section 23 Connection relationship identification part 24 Area Relationship Acquisition Unit 25 Class Relationship Acquisition Unit 26 Recognition result correction section 27 Evaluation Department 28 Output section 100 Object recognition device

Claims

1. image capture means for capturing an image including a plurality of objects; an object recognition means for performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; a connection relationship specifying means for performing processing to specify whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects, based on the recognition result obtained by the object recognition processing; an area relationship acquisition means for acquiring area relationship information, which is information relating to the relationship between each object area identified as having the connection relationship; a class relationship acquisition means for acquiring object name relationship information, which is information indicating the relationship between a plurality of classes that are preset in order to obtain the recognition result by the object recognition processing, and which is information indicating whether or not the name of an object that is assumed to actually belong to one of the plurality of classes matches the name of the object recognized by the object recognition processing; a recognition result correction means for performing a recognition result correction process for correcting the recognition result obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; an evaluation means for evaluating the recognition results obtained by the object recognition processing using the plurality of corrected recognition results, thereby obtaining a final recognition result relating to a class to which each of the plurality of objects belongs; An object recognition device comprising:

2. 2. The object recognition device according to claim 1, wherein the object recognition means performs the object recognition processing on the image, and obtains, as the recognition result, a recognition score, which is a value indicating the probability for each class when each of the plurality of objects is classified into one of the plurality of classes.

3. 3. The object recognition device according to claim 2, wherein the region relationship acquisition means acquires, as the region relationship information corresponding to two object regions having the connection relationship, appearance similarity information, which is information relating to the similarity in appearance between objects included in the two object regions.

4. An object recognition device as described in Claim 3, wherein the recognition result correction means obtains a correction value for correcting the recognition score obtained by the object recognition processing based on the appearance similarity information and the object name relationship information.

5. An image acquisition means for acquiring an image including a plurality of objects; an object recognition means for performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; a connection relationship specifying means for performing processing to specify whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects, based on the recognition result obtained by the object recognition processing; a region relation acquisition means for acquiring region relation information, which is information relating to the relationship between each of the object regions identified as having the connection relationship; a class relationship acquisition means for acquiring height relationship information, which is information indicating the relationship between a height of an object when it is assumed that the object actually belongs to one of the plurality of classes, and a height of an object recognized by the object recognition processing, as class relationship information, which is information indicating the relationship between a plurality of classes that are preset in order to obtain the recognition result by the object recognition processing; and a recognition result correction means for performing a recognition result correction process for correcting the recognition result obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; an evaluation means for evaluating the recognition results obtained by the object recognition processing using the plurality of corrected recognition results, thereby obtaining a final recognition result relating to a class to which each of the plurality of objects belongs; An object recognition device comprising:

6. The object recognition device described in Claim 5, wherein the object recognition means performs the object recognition processing on the image, and obtains, as the recognition result, a recognition score which is a value indicating the probability of each class when each of the plurality of objects is classified into one of the plurality of classes.

7. An object recognition device as described in Claim 6, wherein the area relationship acquisition means acquires size information, which is information regarding the relative size relationship between objects contained in the two object areas, as the area relationship information corresponding to the two object areas having the connection relationship.

8. The object recognition device according to claim 7 , wherein the recognition result correction means acquires a correction value for correcting the recognition score obtained by the object recognition process based on the size information and the height relationship information.

9. Acquire an image containing multiple objects; performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; performing a process for identifying whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects based on the recognition result obtained by the object recognition process; acquiring area relationship information that is information relating to the relationship between each of the object areas identified as having the connection relationship; As class relationship information that indicates relationships between a plurality of classes that are preset in order to obtain the recognition result by the object recognition processing, object name relationship information that indicates whether or not the name of an object that is assumed to actually belong to one of the plurality of classes matches the name of the object recognized by the object recognition processing is acquired; performing a recognition result correction process for correcting the recognition results obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; An object recognition method that obtains a final recognition result relating to the class to which each of the plurality of objects belongs by evaluating the recognition result obtained by the object recognition processing using the plurality of corrected recognition results.

10. Acquiring an image including a plurality of objects; performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; performing a process for identifying whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects based on the recognition result obtained by the object recognition process; acquiring area relationship information that is information relating to the relationship between each of the object areas identified as having the connection relationship; height relationship information is acquired as class relationship information that indicates a relationship between a plurality of classes that are preset for obtaining the recognition result by the object recognition processing, the height relationship information being information that indicates a relationship between a height of an object when it is assumed that the object actually belongs to one of the plurality of classes and a height of an object recognized by the object recognition processing; performing a recognition result correction process for correcting the recognition results obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; An object recognition method that obtains a final recognition result relating to the class to which each of the plurality of objects belongs by evaluating the recognition result obtained by the object recognition processing using the plurality of corrected recognition results.

11. Acquiring an image including a plurality of objects; performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; performing a process for identifying whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects based on the recognition result obtained by the object recognition process; acquiring area relationship information that is information relating to the relationship between each of the object areas identified as having the connection relationship; As class relationship information that indicates relationships between a plurality of classes that are preset in order to obtain the recognition result by the object recognition processing, object name relationship information that indicates whether or not the name of an object that is assumed to actually belong to one of the plurality of classes matches the name of the object recognized by the object recognition processing is acquired; performing a recognition result correction process for correcting the recognition result obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; A program that causes a computer to execute a process of obtaining a final recognition result relating to the class to which each of the plurality of objects belongs by evaluating the recognition result obtained by the object recognition process using the plurality of corrected recognition results.

12. Acquiring an image including a plurality of objects; performing an object recognition process on the image to obtain a recognition result corresponding to each of the plurality of objects included in the image; performing a process for identifying whether or not there is a connection relationship between a plurality of object regions corresponding to each of the plurality of objects based on the recognition result obtained by the object recognition process; acquiring area relationship information that is information relating to the relationship between each of the object areas identified as having the connection relationship; height relationship information is acquired as class relationship information that indicates a relationship between a plurality of classes that are preset for obtaining the recognition result by the object recognition processing, the height relationship information being information that indicates a relationship between a height of an object when it is assumed that the object actually belongs to one of the plurality of classes and a height of an object recognized by the object recognition processing; performing a recognition result correction process for correcting the recognition result obtained by the object recognition process based on the area relationship information and the class relationship information, thereby obtaining a plurality of corrected recognition results; A program that causes a computer to execute a process of obtaining a final recognition result relating to the class to which each of the plurality of objects belongs by evaluating the recognition result obtained by the object recognition process using the plurality of corrected recognition results.

Images