Inference device

Abstract

This inference device comprises: an object position acquisition unit that acquires position information indicating the positions of objects included in image data in which a plurality of objects are set as subjects; a feature extraction unit that extracts feature amounts of objects included in the image data; and a feature conversion unit that enters a feature amount and position information extracted from a first object in the image data and a feature amount and position information of one or more objects different from the first object into a model that has been pre-trained through machine learning, thereby converting the feature amounts into relationship feature amounts, which are values that take into account the positional relationship between the objects as well as the appearance of the objects. This configuration allows the inference device to assist in optimizing the monitoring, etc. of an object such as a product.

Description

reasoning device

[0001] The present disclosure relates to an inference device, an inference method, a recording medium, and a learning device.

[0002] 2. Description of the Related Art Techniques are known that are used for on-shelf management, such as detecting out-of-stock items, using image data.

[0003] For example, Patent Literature 1 describes an item recognition device that extracts a plurality of item regions from a captured image of an item such as a commodity, extracts feature quantities indicating characteristics of the item region from each item region, and identifies items of the same type based on the feature quantities. Patent Literature 1 also discloses, as an example of a method for grouping items, clustering based on feature quantities in a feature space.

[0004] JP 2022-182844 A

[0005] Ashish Vaswani et al. “Attention Is All You Need” [Retrieved December 6, 2020], <URL: https: / / arxiv.org / pdf / 1706.03762.pdf> Kan Wu et al. “Rethinking and Improving Relative Position Encoding for Vision Transformer” [Retrieved December 6, 2020], <URL: https: / / arxiv.org / pdf / 2107.14222.pdff> Peter Shaw et al. “Self-Attention with Relative Position Representations” [Retrieved December 6, 2020], <URL: https: / / arxiv.org / pdf / 1803.02155.pdf>

[0006] In cases where there are similar items or identical items with different postures, it may be desirable to perform processing using image data, such as clustering or item identification processing, taking into account the mutual relationship between appearance and physical distance, such as when the appearance of an item resembles the item to the left but the physical distance is closer to the item to the right. However, the technology described in Patent Literature 1 does not take into account the mutual relationship between appearance and physical distance. As a result, there has been a problem in that it may be difficult to improve the accuracy when an information processing device performs processing such as clustering using image data.

[0007] Therefore, one of the objects of the present disclosure is to provide an inference device, an inference method, a recording medium, and a learning device that can solve the above-mentioned problems.

[0008] In order to achieve this object, the inference device of the present disclosure has a configuration including an object position acquisition unit that acquires position information indicating the positions of objects included in image data having multiple objects as subjects; a feature extraction unit that extracts feature amounts of the objects included in the image data; and a feature conversion unit that converts the feature amounts into relationship feature amounts, which are values ​​that take into account the positional relationships between the objects as well as the appearances of the objects, by inputting the feature amounts and position information extracted from a first object in the image data and the feature amounts and position information of one or more objects different from the first object into a model that has been trained in advance by machine learning.

[0009] Furthermore, the inference method in the present disclosure is configured such that an information processing device acquires position information indicating the positions of objects included in image data having a plurality of objects as subjects, extracts features of the objects included in the image data, and inputs the features and position information extracted from a first object in the image data and the features and position information of one or more objects different from the first object into a model previously trained by machine learning, thereby converting the features into relationship features, which are values ​​that take into account the positional relationships between the objects as well as the appearances of the objects.

[0010] Furthermore, the recording medium in the present disclosure is a computer-readable recording medium having recorded thereon a program for causing an information processing device to perform the following process: acquire position information indicating the positions of objects included in image data having a plurality of objects as subjects; extract features of the objects included in the image data; and input the features and position information extracted from a first object in the image data and the features and position information of one or more objects different from the first object into a model previously trained by machine learning, thereby converting the features into relationship features, which are values ​​that take into account the positional relationships between the objects as well as the appearances of the objects.

[0011] Furthermore, the learning device in the present disclosure has a configuration including an object position acquisition unit that acquires position information indicating the positions of objects included in image data having a plurality of objects as subjects; a feature extraction unit that extracts feature amounts of the objects included in the image data; a feature conversion unit that converts the feature amounts into relationship feature amounts that are values ​​that take into account the positional relationships between the objects as well as the appearances of the objects by inputting the feature amounts and position information extracted from a first object in the image data and the feature amounts and position information of one or more objects different from the first object into a model; and a learning unit that trains parameters of the model that the feature conversion unit uses when converting the feature amounts into relationship feature amounts, using the results of the conversion by the feature conversion unit and correct answer information stored in advance.

[0012] According to the above-described configurations, it is possible to improve the accuracy when an information processing device performs processing such as clustering using image data.

[0013] FIG. 1 is a diagram illustrating an example configuration of a monitoring system in the present disclosure. FIG. 2 is a block diagram illustrating an example configuration of a learning device. FIG. 3 is a diagram for explaining an example of acquiring an object position. FIG. 4 is a diagram for explaining an example of feature conversion processing. FIG. 5 is a diagram for explaining an example of feature conversion processing. FIG. 6 is a diagram for explaining an example of learning processing. FIG. 7 is a block diagram illustrating an example configuration of an inference device. FIG. 8 is a diagram illustrating an example of clustering processing. FIG. 9 is a diagram illustrating an example of utilizing a clustering result. FIG. 10 is a diagram illustrating an example of utilizing a clustering result. FIG. 11 is a flowchart illustrating an example operation of a learning device. FIG. 12 is a flowchart illustrating an example operation of an inference device. FIG. 13 is a block diagram illustrating another example configuration of an inference device. FIG. 14 is a diagram illustrating an example of recognition processing. FIG. 15 is a diagram illustrating an example hardware configuration of a second inference device in the present disclosure. FIG. 16 is a block diagram illustrating an example configuration of an inference device. FIG. 17 is a flowchart illustrating an example operation of the inference device. FIG. 18 is a block diagram illustrating an example configuration of a second learning device.

[0014] [First Embodiment] An example configuration of a monitoring system 100 according to the present disclosure will be described with reference to FIGS. 1 to 14. FIG. 1 is a diagram illustrating an example configuration of the monitoring system 100. FIG. 2 is a block diagram illustrating an example configuration of a learning device 300. FIG. 3 is a diagram illustrating an example of acquiring an object position. FIGS. 4 and 5 are diagrams illustrating an example of feature conversion processing. FIG. 6 is a diagram illustrating an example of learning processing. FIG. 7 is a block diagram illustrating an example configuration of an inference device 400. FIG. 8 is a diagram illustrating an example of clustering processing. FIGS. 9 and 10 are diagrams illustrating examples of using the results of the clustering processing. FIG. 11 is a flowchart illustrating an example operation of the learning device 300. FIG. 12 is a flowchart illustrating an example operation of the inference device 400. FIG. 13 is a block diagram illustrating another example configuration of the inference device 400. FIG. 14 is a diagram illustrating an example of recognition processing. Note that in the present disclosure, the drawings may be associated with one or more embodiments.

[0015] In a first embodiment of the present disclosure, a monitoring system 100 capable of performing processes such as clustering and identification of objects such as merchandise and other items using image data will be described. As will be described later, the monitoring system 100 acquires image data of objects displayed on a shelf 500 or the like, and acquires location information that indicates the location of each object. The monitoring system 100 also extracts feature quantities corresponding to the external characteristics of each object from the image data. The monitoring system 100 then uses the acquired location information and the extracted feature quantities to convert the extracted feature quantities into relationship feature quantities that are values ​​that take into account the external appearance of the objects as well as the positional relationships between the objects. The monitoring system 100 can then perform processes such as clustering and identification of objects in the image data using the converted relationship feature quantities.

[0016] In the present disclosure, a relationship feature refers to a value that takes into account the positional relationship between objects as well as the appearance of the objects. For example, the relationship feature can be acquired by inputting position information indicating the position of the object and feature amounts corresponding to the appearance of the object into a pre-trained model. In other words, the relationship feature can be acquired by inputting feature amounts and position information extracted from a first object in an image and feature amounts and position information of one or more objects different from the first object into a machine-learned model.

[0017] The monitoring system 100 described in the present disclosure may also be used to manage objects such as merchandise and other items sold in a retail store or the like. The monitoring system 100 may also be applied to the healthcare field, such as managing shelves 500 displaying pharmaceuticals. In other words, the monitoring system 100 may be used to optimize the monitoring of objects such as merchandise in the healthcare field or other retail fields. The monitoring system 100 may also be used in situations other than those exemplified above.

[0018] Fig. 1 shows an example configuration of a monitoring system 100. Referring to Fig. 1, the monitoring system 100 includes, for example, an imaging device 200, a learning device 300, and an inference device 400. As shown in Fig. 1, the learning device 300 and the inference device 400 can be connected to each other via a wired or wireless connection so that they can communicate with each other. Furthermore, the imaging device 200 and the inference device 400 can be connected to each other via a wired or wireless connection so that they can communicate with each other.

[0019] Note that the monitoring system 100 may have a configuration other than that illustrated in FIG. 1 . For example, the imaging device 200 and the learning device 300 may be connected so as to be able to communicate with each other. Furthermore, the learning device 300 and the inference device 400 do not necessarily need to be directly connected, and may be configured so as to be able to exchange data such as learning results via a recording medium or the like. Furthermore, the monitoring system 100 may include two or more imaging devices 200. Similarly, the monitoring system 100 may include two or more learning devices 300, or two or more inference devices 400.

[0020] The imaging device 200 is installed in advance at a predetermined location where a plurality of objects are displayed, such as in a store or warehouse, and acquires image data. The imaging device 200 may be a general device such as a surveillance camera. For example, the imaging device 200 acquires image data of at least a portion of a shelf 500 on which a plurality of objects are displayed as a subject. The imaging device 200 may acquire image data including a plurality of shelves 500. The imaging device 200 can also acquire image data in a time series with different image capture times for the same predetermined range.

[0021] The learning device 300 is an information processing device that trains a model for converting features into relational features by, for example, performing supervised learning using correct answer data. Fig. 2 shows an example configuration of the learning device 300. Referring to Fig. 2, the learning device 300 has, as main components, for example, an operation input unit 310, a screen display unit 320, a communication I / F unit 330, a storage unit 340, and a calculation processing unit 350.

[0022] 2 illustrates an example in which the functions of learning device 300 are realized using a single information processing device. However, learning device 300 may be realized using multiple information processing devices, for example, on the cloud. Learning device 300 may not include some of the components illustrated above, such as not having operation input unit 310 or screen display unit 320, or may have components other than those illustrated above.

[0023] Operation input unit 310 is made up of operation input devices such as a keyboard, a mouse, etc. Operation input unit 310 detects operations by an operator operating learning device 300 and outputs the operations to calculation processing unit 350.

[0024] The screen display unit 320 is composed of a screen display device such as a liquid crystal display, an organic electroluminescence (EL) display, etc. The screen display unit 320 can display various information stored in the storage unit 340 on the screen in response to instructions from the arithmetic processing unit 350.

[0025] The communication I / F unit 330 is composed of a data communication circuit, etc. The communication I / F unit 330 performs data communication with the inference device 400 and other external devices connected via a communication line.

[0026] The storage unit 340 is a storage device such as a hard disk or memory. The storage unit 340 stores processing information and a program 343 required for various processes in the arithmetic processing unit 350. The program 343 is read into the arithmetic processing unit 350 and executed to realize various processing units. The program 343 is read in advance from an external device or recording medium via a data input / output function such as the communication I / F unit 330, and is stored in the storage unit 340. Main information stored in the storage unit 340 includes, for example, learning data information 341 and correct answer information 342.

[0027] The learning data information 341 includes information to be learned. The learning data information 341 is acquired in advance by, for example, accepting input using the operation input unit 310 or accepting input from an external device via the communication I / F unit 330, and is stored in the storage unit 340. Note that at least a portion of the information included in the learning data information 341 may be acquired from the imaging device 200, etc.

[0028] For example, the learning data information 341 includes learning image data, as information to be learned, of a subject such as a shelf 500 on which a plurality of objects are displayed. The learning data information 341 may include a plurality of learning image data for different displayed objects, shelves 500, etc.

[0029] The training data information 341 may include information other than the above examples. For example, the training data information 341 may include position information indicating the position of each object in the image data. Furthermore, instead of image data, the training data information 341 may include information indicating the feature amount of each object extracted in advance from the image data and position information indicating the position of the object.

[0030] The correct answer information 342 includes information indicating whether each object included in the learning data information 341 is the same as another object, or whether they are of the same type, etc. The correct answer information 342 is acquired in advance by receiving an input using the operation input unit 310, receiving an input from an external device or the like via the communication I / F unit 330, or the like, and is stored in the storage unit 340.

[0031] For example, the correct answer information 342 includes an identification ID for identifying each object included in the learning image data. For example, an identification ID is assigned in advance to each object or each type of object. By checking the identification ID, the learning device 300 can determine whether an object is of the same type as other objects. In the correct answer information 342, the identification ID may be associated with position information or feature values ​​of each object in the learning image data. Note that the correct answer information 342 may include information other than the above examples.

[0032] The arithmetic processing unit 350 has an arithmetic device such as a CPU (Central Processing Unit) and its peripheral circuits. The arithmetic processing unit 350 reads and executes a program 343 from the storage unit 340, thereby causing the above hardware and the program 343 to cooperate to realize various processing units. Major processing units realized by the arithmetic processing unit 350 include, for example, a learning data acquisition unit 351, an object position acquisition unit 352, a feature extraction unit 353, a feature conversion unit 354, a learning unit 355, and an output unit 356.

[0033] In addition, instead of the above-mentioned CPU, the arithmetic processing unit 350 may have a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination of these.

[0034] The learning data acquisition unit 351 acquires information to be learned by, for example, referring to the learning data information 341. For example, the learning data acquisition unit 351 can acquire learning image data, which has as its subject a shelf 500 on which a plurality of objects are displayed, as the information to be learned. The learning data acquisition unit 351 may acquire position information and the like included in the learning data information 341 along with the image data. Note that the learning data acquisition unit 351 may acquire information to be learned, such as image data for learning, from an external device such as the imaging device 200.

[0035] The object position acquisition unit 352 acquires position information indicating the position of each object included in the learning image data. For example, the object position acquisition unit 352 can acquire position information when the learning data information 341 does not include position information.

[0036] As an example, the object position acquisition unit 352 detects object portions in the image data by inputting learning image data into a pre-trained model such as an object detector. At this time, the object position acquisition unit 352 may acquire rectangles surrounding each detected object as the object portion detection results, as shown in FIG. 3 . Furthermore, the object position acquisition unit 352 acquires position information indicating the position of each detected object according to the detection results. For example, the object position acquisition unit 352 acquires the center coordinates of the acquired rectangle as the position information. The object position acquisition unit 352 may also acquire rectangular coordinates indicating the coordinates of the upper left or lower right corner of the acquired rectangle as the position information. Note that the object position acquisition unit 352 may also acquire information indicating the absolute position of an object, such as coordinates in the image data, or information indicating a relative position, such as information indicating a positional relationship with an object of interest, other than the above examples.

[0037] Furthermore, the object position acquisition unit 352 may acquire position information using any other method instead of using an object detector, etc. For example, the object position acquisition unit 352 may acquire position information indicating the position of each object included in the learning image data using a method such as accepting input using the operation input unit 310 or accepting input from an external device or the like via the communication I / F unit 330.

[0038] The feature extraction unit 353 extracts feature amounts of an object. For example, the feature extraction unit 353 extracts feature amounts according to the appearance of the object, such as the shape, color, and pattern of the object, as feature amounts of the object. The feature extraction unit 353 may extract feature amounts of an object by inputting the rectangular portion acquired by the object position acquisition unit 352 into a feature extractor that has been trained in advance. The feature extraction unit 353 may also extract feature amounts using other known methods.

[0039] Note that the object detection and acquisition of position information by the object position acquisition unit 352 and the feature extraction by the feature extraction unit 353 may be performed in a single process. For example, the learning device 300 may acquire rectangles and position information, and extract features in a single process, for example, by inputting image data to a trained model that simultaneously detects objects and extracts features based on the input image data.

[0040] As shown in FIG. 4 , the feature conversion unit 354 uses the position information acquired by the object position acquisition unit 352 and the feature amounts extracted by the feature extraction unit 353 to convert the extracted feature amounts into relationship feature amounts, which are values ​​that take into account the positional relationships between objects as well as the appearances of the objects. For example, the feature conversion unit 354 adds the position information to the feature amounts using any method. Then, the feature conversion unit 354 converts the feature amounts into relationship feature amounts by inputting the result of the addition to a pre-trained model. For example, the feature conversion unit 354 may convert the feature amounts into relationship feature amounts by inputting the feature amounts and position information extracted from a first object in the image data and the feature amounts and position information of one or more objects different from the first object to a pre-trained model.

[0041] For example, the feature conversion unit 354 can add the feature and the position information using any method, such as adding 4-dimensional position information to 128-dimensional feature to obtain 132-dimensional information. When absolute position information is used as the position information, the feature conversion unit 354 may convert the position information to the same dimensions as the feature via an MLP (Multilayer Perceptron) or the like, and then add the converted result to the feature.

[0042] The feature conversion unit 354 also converts the feature into a relationship feature by inputting the summed result to a pre-trained model. FIG. 5 shows an example of processing when feature conversion processing is implemented using a technology called a Transformer Encoder. Referring to FIG. 5 , for example, the feature conversion unit 354 converts a feature obtained by summing location information into a relationship feature by performing conversion processing according to the relationship between a feature of interest and other feature amounts in response to input of multiple feature amounts and location information. In other words, the feature conversion unit 354 weights a vector called a value obtained by multiplying a feature of interest by a transformation matrix according to the similarity between the vector called a query obtained by multiplying a feature of interest by a transformation matrix and the vector called a key obtained by multiplying each input feature amount including the matrix of interest, and calculates a weighted sum. In other words, the feature conversion unit 354 weights the vectors obtained by converting each feature amount according to the degree of correlation between the feature of interest and all feature amounts, and calculates the weighted sum to obtain a relationship feature. In this way, the feature conversion unit 354 converts the feature into a relational feature so as to emphasize the feature determined to be relevant and to ignore the feature determined to be irrelevant to some extent. Note that the feature conversion unit 354 may have only one layer to be added together, or may have multiple layers.

[0043] For example, as described above, the feature conversion unit 354 focuses on a certain feature and converts the focused feature into a relationship feature. Furthermore, the feature conversion unit 354 can acquire a relationship feature corresponding to each object by performing the above conversion process when focusing on each object in the image data. The feature conversion unit 354 may calculate a weighted sum of feature quantities corresponding to the focused object and all other objects present in the image data, or may be configured to select feature quantities other than the focused feature using any means and calculate a weighted sum of the focused feature and the selected feature quantity. For example, when multiple shelves 500 are present in the image data, the feature conversion unit 354 may be configured to select an object displayed on the same shelf 500 as the focused object. Furthermore, the feature conversion unit 354 may be configured to select an object displayed on the same shelf as the focused object, or an object adjacent to the focused object on the left, right, top, or bottom. For example, as described above, the feature conversion unit 354 may convert features into relationship features by inputting the features and position information extracted from the first object in the image data and the features and position information of one or more selected objects different from the first object into a pre-trained model. The feature conversion unit 354 may be configured to select features using a method other than the above examples. Furthermore, the feature conversion unit 354 may convert features into relationship features using a long short-term memory (LSTM), a graph neural network (GNN), or the like, in addition to a transformer encoder.

[0044] In this way, the feature conversion unit 354 converts the extracted feature into a relationship feature using multiple pieces of position information and feature amounts. The feature conversion unit 354 may convert a feature into a relationship feature using a method other than the above-mentioned examples. In other words, the feature conversion unit 354 may use an encoding method other than the above-mentioned examples to indicate a method of expressing a position or a method of addition. For example, the feature conversion unit 354 may use a technique such as that described in Non-Patent Document 1 as an encoding method for expressing an absolute position. Furthermore, the feature conversion unit 354 may use a technique such as that described in Non-Patent Document 2 or Non-Patent Document 3 as an encoding method for expressing a relative position.

[0045] The learning unit 355 trains the parameters of a model used when the feature conversion unit 354 converts a feature into a relationship feature, using the result of the conversion by the feature conversion unit 354 and the correct answer information 342. The learning unit 355 may train the parameters used when defining a conversion matrix.

[0046] 6, the learning unit 355 selects a plurality of positive pairs and a plurality of negative pairs from the plurality of relationship feature amounts converted by the feature conversion unit 354 by referring to the correct answer information 342. Then, the learning unit 355 trains parameters used when the feature conversion unit 354 converts the feature amounts into relationship feature amounts so that the distance between the relationship feature amounts forming a positive pair becomes closer and the distance between the relationship feature amounts forming a negative pair becomes farther in the feature space.

[0047] The learning unit 355 can select a positive pair or a negative pair using any method. For example, the learning unit 355 can refer to the correct answer information 342 and select, as a positive pair, a pair of relationship feature quantities corresponding to objects assigned the same identification ID among the relationship feature quantities. The learning unit 355 can also refer to the correct answer information 342 and select, as a negative pair, a pair of relationship feature quantities corresponding to objects assigned different identification IDs among the relationship feature quantities. In this case, the learning unit 355 may select, as a negative pair, a pair of objects whose distance in the image data is equal to or less than a predetermined threshold, or a pair of objects whose distance in the feature space is equal to or less than a predetermined threshold, among the relationship feature quantities corresponding to objects assigned different identification IDs. The learning unit 355 may also select, as a negative pair, a random pair in image data assigned different identification IDs.

[0048] For example, as described above, the learning unit 355 performs machine learning using positive pairs and negative pairs selected with reference to the correct answer information 342. Note that the learning unit 355 may repeat training of parameters until a predetermined termination condition is satisfied, for example, by repeating the learning process for multiple steps.

[0049] The output unit 356 transmits the learning results of the learning unit 355 to the inference device 400. For example, the output unit 356 can transmit the parameters learned by the learning unit 355 to the inference device 400 via the communication I / F unit 330.

[0050] The above is an example configuration of the learning device 300. Note that the configuration of the learning device 300 may be other than that exemplified above. For example, the learning unit 355 may be configured to train the parameters of a feature extractor used by the feature extraction unit 353 when extracting features, in addition to training the feature conversion unit 354. Furthermore, the learning unit 355 may learn parameters used by the MLP and the feature conversion unit 354 when converting features into relationship features, such that a two-class classifier, such as an MLP, concatenates and inputs relationship features corresponding to two different objects, and outputs 1 if the two input objects are a positive pair with the same identification ID and 0 if the two input objects are a negative pair with different identification IDs. Furthermore, the learning unit 355 may input relationship features corresponding to multiple objects and position information of two different objects among the multiple objects to a Transformer Decoder, and train the Transformer Decoder to output 1 if the two objects whose position information is input have the same identification ID and 0 if the two objects have different identification IDs.

[0051] The feature conversion unit 354 may also be configured to input feature quantities and position information of objects such as products, as well as feature quantities and position information of other known objects, and convert them into relationship features. For example, the known objects may be at least some of the objects present around the shelf 500, such as price tags, shelf labels, separators, shelf levels, and containers. By inputting feature quantities and position information of objects such as products, as well as feature quantities and position information of other known objects, and converting them into relationship features, the feature conversion unit 354 can convert the features into relationship features that also take into account the position of the price tag, etc. As a result, the feature conversion unit 354 can convert objects sandwiched between separators into similar relationship features, or convert multiple objects placed close to a price tag into similar relationship features.

[0052] The inference device 400 is an information processing device that acquires relationship features in response to input of image data acquired by the imaging device 200. Furthermore, the inference device 400 can use the acquired relationship features to perform processes such as clustering and identification of objects in the image data acquired by the imaging device 200. FIG. 7 shows an example configuration of the inference device 400. Referring to FIG. 7, the inference device 400 has, as main components, for example, an operation input unit 410, a screen display unit 420, a communication I / F unit 430, a storage unit 440, and an arithmetic processing unit 450.

[0053] Note that inference device 400 may employ the same modified example as learning device 300. Furthermore, the configurations of operation input unit 410, screen display unit 420, and communication I / F unit 430 may be similar to those of operation input unit 310, screen display unit 320, and communication I / F unit 330 of learning device 300.

[0054] The storage unit 440 is a storage device such as a hard disk or memory. The storage unit 440 stores processing information and a program 443 required for various processes in the arithmetic processing unit 450. The program 443 is read into the arithmetic processing unit 450 and executed to realize various processing units. The program 443 is read in advance from an external device or recording medium via a data input / output function such as the communication I / F unit 430, and is stored in the storage unit 440. Main information stored in the storage unit 440 includes, for example, image data information 441 and clustering result information 442.

[0055] The image data information 441 includes image data from which relationship features are to be acquired. The image data information 441 is updated in response to the image data acquisition unit 451 acquiring image data from the imaging device 200 or other external devices.

[0056] The clustering result information 442 includes information indicating the results of processing by the clustering unit 455, which will be described later. The clustering result information 442 is updated as the clustering unit 455 performs clustering, etc.

[0057] The arithmetic processing unit 450 includes an arithmetic device such as a CPU and its peripheral circuits. The arithmetic processing unit 450 reads and executes a program 443 from the storage unit 440, thereby causing the hardware and the program 443 to cooperate to realize various processing units. Major processing units realized by the arithmetic processing unit 450 include, for example, an image data acquisition unit 451, an object position acquisition unit 452, a feature extraction unit 353, a feature conversion unit 454, a clustering unit 455, and an output unit 456. Note that, like the arithmetic processing unit 350 included in the learning device 300, the arithmetic processing unit 450 may include a GPU or the like instead of the CPU described above.

[0058] The image data acquisition unit 451 acquires image data from the imaging device 200. The image data acquisition unit 451 may acquire image data from an external device other than the imaging device 200. In addition, the image data acquisition unit 451 stores the acquired image data in the storage unit 440 as image data information 441.

[0059] The object position acquisition unit 452 acquires position information indicating the position of each object included in the image data acquired by the image data acquisition unit 451. For example, similar to the object position acquisition unit 352, the object position acquisition unit 452 detects object portions in the image data by inputting the image data into a pre-trained model such as an object detector. At this time, the object position acquisition unit 452 may acquire a rectangle surrounding each detected object as the detection result of the object portion. Furthermore, the object position acquisition unit 452 acquires position information indicating the position of each detected object according to the detection result. For example, the object position acquisition unit 452 can acquire the center coordinates or rectangular coordinates of the acquired rectangle as the position information. Similar to the object position acquisition unit 352, the object position acquisition unit 452 may acquire information indicating the absolute position or relative position of an object other than those exemplified above.

[0060] Similar to the feature extraction unit 353, the feature extraction unit 453 extracts feature amounts of the object detected by the object position acquisition unit 452. For example, the feature extraction unit 453 extracts feature amounts according to the appearance of the object, such as the shape, color, and pattern of the object, as feature amounts of the object. The feature extraction unit 453 may extract feature amounts of the object by inputting the rectangular portion acquired by the object position acquisition unit 452 into a feature extractor that has been trained in advance. The feature extraction unit 453 may also extract feature amounts using other known methods.

[0061] The object detection and acquisition of position information by the object position acquisition unit 452 and the feature extraction by the feature extraction unit 453 may be performed in a single process. For example, similar to the learning device 300, the inference device 400 may input image data to a trained model that simultaneously detects objects and extracts features based on the input image data, thereby acquiring rectangles, acquiring position information, and extracting features in a single process.

[0062] Similar to the feature conversion unit 354, the feature conversion unit 454 uses the position information acquired by the object position acquisition unit 452 and the feature extracted by the feature extraction unit 453 to convert the extracted feature into a relationship feature, which is a value that takes into account the positional relationship between objects as well as the appearance of the objects. For example, the feature conversion unit 454 adds the position information to the feature using any method. The feature conversion unit 454 may add the position information to the feature using the same method as the feature conversion unit 354. Furthermore, the feature conversion unit 454 converts the feature into a relationship feature by inputting the result of the addition to a model previously trained by the learning device 300. Similar to the feature conversion unit 354, the feature conversion unit 454 may perform a conversion process into a relationship feature using a feature of interest and a selected feature.

[0063] The clustering unit 455 functions as a processing unit that performs clustering using the relationship feature converted by the feature conversion unit 454. That is, as shown in Fig. 8, the clustering unit 455 classifies objects in the image data into a plurality of clusters using the relationship feature converted by the feature conversion unit 454. The clustering unit 455 also stores the results of classification into a plurality of clusters in the storage unit 440 as clustering result information 442. The clustering unit 455 may perform clustering using the relationship feature for each shelf level, or may perform clustering for multiple levels, such as for each shelf 500.

[0064] For example, the clustering unit 455 may perform clustering for dividing each object in the image data into a predetermined number of clusters, such as a preset number, based on the relationship feature. The clustering unit 455 may perform the clustering by comparing the relationship feature for each object in the image data in the feature space and classifying objects that are determined to be similar, such as those for which the similarity, such as the distance between the relationship feature amounts, is equal to or less than a predetermined threshold, into the same cluster.

[0065] The threshold value to be compared with the similarity may be a predetermined fixed value, or may be a value dynamically determined by the clustering unit 455 or the like. For example, when dynamically determining the threshold value, the clustering unit 455 may determine the threshold value based on the similarity between adjacent objects in the image data and the similarity between non-adjacent objects. For example, the clustering unit 455 calculates the similarity between adjacent objects in the image data and the similarity between non-adjacent objects. The clustering unit 455 can then determine the threshold value according to the distribution of the calculated similarities. For example, the clustering unit 455 may determine an arbitrary value between the average value of the similarity between adjacent objects and the average value of the similarity between non-adjacent objects as the threshold value. The clustering unit 455 may dynamically determine the threshold value using a method other than those exemplified above.

[0066] The output unit 356 outputs the results of the clustering process performed by the clustering unit 455. For example, the output unit 356 can display the clustering result information 442 on the screen display unit 420 or transmit it to an external device via the communication I / F unit 330.

[0067] The above is an example of the configuration of the inference device 400. Note that the inference device 400 may have the same modified examples as the learning device 300.

[0068] The results of the processing by the clustering unit 455 can be used, for example, when setting a target frame indicating the range in which various processes, such as missing item detection, are performed, as shown in FIG. 9 . For example, the inference device 400 can be configured to set a target frame based on the clustering results and shelf lines in the image data. In other words, the inference device 400 can be configured to set a target frame for a range identified using the clustering results and in which an item may be present. Note that the range in which an item may be present may be identified using not only shelf lines but also the results of object detection using any object detector. The inference device 400 can also be configured to calculate a missing item rate, such as that shown in FIG. 10 , using the target frame set as described above. For example, the inference device 400 may calculate the missing item rate based on the proportion of objects within the target frame in the image data to be detected and the proportion of objects within the target frame in image data without missing items. Note that at least some of the processes, such as the clustering process, the setting of the target frame, and the calculation of the missing item rate, may be executed by an external device connected to the inference device 400.

[0069] 11 and 12, an example of the operation of the learning device 300 and the inference device 400 that constitute the monitoring system 100 will be described. First, an example of the operation of the learning device 300 will be described with reference to FIG.

[0070] 11 is a flowchart showing an example of the operation of the learning device 300. Referring to FIG. 11, the learning data acquisition unit 351 acquires information to be learned by, for example, referring to the learning data information 341 (step S101). For example, the learning data acquisition unit 351 can acquire learning image data, as the information to be learned, that has as its subject a shelf 500 on which multiple objects are displayed.

[0071] The object position acquisition unit 352 acquires position information indicating the position of each object included in the learning image data (step S102). The object position acquisition unit 352 may acquire the position information using an object detector or the like, or may acquire the position information in response to input using the operation input unit 310. Note that if the information to be learned includes position information, the processing of step S102 may be omitted.

[0072] The feature extraction unit 353 extracts feature amounts of the object (step S103). For example, the feature extraction unit 353 extracts feature amounts according to the appearance of the object, such as the shape, color, and pattern of the object, as feature amounts of the object. The feature extraction unit 353 may extract the feature amounts of the object by inputting the rectangular portion acquired by the object position acquisition unit 352 into a feature extractor that has been trained in advance.

[0073] The feature conversion unit 354 uses the position information acquired by the object position acquisition unit 352 and the feature amounts extracted by the feature extraction unit 353 to convert the extracted feature amounts into relationship feature amounts (step S104). For example, the feature conversion unit 354 adds the position information to the feature amounts using any method. Thereafter, the feature conversion unit 354 converts the feature amounts into relationship feature amounts by inputting the result of the addition to a model that has been trained in advance.

[0074] The learning unit 355 uses the results of the conversion by the feature conversion unit 354 and the correct answer information 342 to train model parameters used by the feature conversion unit 354 when converting features into relationship features (step S105). For example, the learning unit 355 refers to the correct answer information 342 to select multiple positive pairs and multiple negative pairs from the multiple relationship features converted by the feature conversion unit 354. Then, the learning unit 355 trains parameters used by the feature conversion unit 354 when converting features into relationship features so that the distance between positive paired relationship features becomes closer and the distance between negative paired relationship features becomes farther in the feature space.

[0075] The output unit 356 transmits the learning results of the learning unit 355 to the inference device 400 (step S106). For example, the output unit 356 can transmit the parameters learned by the learning unit 355 to the inference device 400 via the communication I / F unit 330.

[0076] The above is an example of the operation of the learning device 300. Next, an example of the operation of the inference device 400 will be described with reference to FIG.

[0077] Fig. 12 is a flowchart showing an example of the operation of the inference device 400. Referring to Fig. 12, the image data acquisition unit 451 acquires image data from the imaging device 200 (step S201).

[0078] The object position acquisition unit 452 acquires position information indicating the position of each object included in the image data acquired by the image data acquisition unit 451 (step S202). For example, the object position acquisition unit 452 may acquire the position information using an object detector or the like, similar to the object position acquisition unit 352.

[0079] The feature extraction unit 453 extracts feature amounts of the object (step S203). For example, the feature extraction unit 453 extracts feature amounts according to the appearance of the object, such as the shape, color, and pattern of the object, as feature amounts of the object. The feature extraction unit 453 may extract the feature amounts of the object by inputting the rectangular portion acquired by the object position acquisition unit 452 into a feature extractor that has been trained in advance.

[0080] The feature conversion unit 454 converts the extracted feature into a relationship feature using the position information acquired by the object position acquisition unit 452 and the feature extracted by the feature extraction unit 453 (step S204). For example, the feature conversion unit 454 adds the position information to the feature using any method. The feature conversion unit 454 also converts the feature into a relationship feature by inputting the result of the addition to a model previously trained by the learning device 300.

[0081] The clustering unit 455 performs clustering using the relationship feature converted by the feature conversion unit 454 (step S205). That is, the clustering unit 455 classifies objects in the image data into multiple clusters using the relationship feature converted by the feature conversion unit 454.

[0082] The output unit 356 outputs the results of the clustering process performed by the clustering unit 455 (step S206). For example, the output unit 356 can display the clustering result information 422 on the screen display unit 420 or transmit it to an external device via the communication I / F unit 330.

[0083] The above is an example of the operation of the inference device 400.

[0084] As described above, the inference device 400 includes an object position acquisition unit 452, a feature extraction unit 453, and a feature conversion unit 454. With this configuration, the feature conversion unit 454 can use the position information acquired by the object position acquisition unit 452 and the feature amounts extracted by the feature extraction unit 453 to convert the extracted feature amounts into relationship feature amounts. As a result, the inference device 400 and other external devices can perform processing such as clustering using relationship feature amounts that take into account appearance and positional relationships. This makes it possible to improve the accuracy when the information processing device performs processing such as clustering using image data.

[0085] The inference device 400 may be configured to perform processing using relationship features other than clustering. For example, as shown in Fig. 13, the calculation processing unit 450 of the inference device 400 may have a recognition unit 457 that functions as a processing unit that performs processing using relationship features together with or instead of the clustering unit 455 by reading and executing a program 443. Furthermore, the storage unit 440 can store comparison target information 444.

[0086] The comparison target information 444 is information used when performing object recognition using relational features. For example, the comparison target information 444 includes features extracted from image data of each object to be recognized. The comparison target information 444 is acquired in advance by receiving input using the operation input unit 410, receiving input from an external device via the communication I / F unit 430, or the like, and is stored in the storage unit 440.

[0087] Furthermore, as shown in FIG. 14 , the recognition unit 457 recognizes an object by matching the relationship feature converted by the feature conversion unit 454 with pre-stored feature included in the comparison information 444. For example, the recognition unit 457 may perform object recognition by performing a matching process using any method, such as searching the comparison information 444 for feature values ​​whose distance from the relationship feature converted by the feature conversion unit 454 is equal to or less than a predetermined threshold. For example, in the above configuration, the learning unit 355 of the learning device 300 trains parameters so that the relationship feature values ​​approach the correct feature values ​​included in the comparison information 444. As a result, the feature conversion unit 454 has the role of correcting the feature values ​​extracted from the object by referring to adjacent objects, etc. For example, when performing recognition using the above-described relationship feature values, using rectangular coordinates as position information enables recognition processing that takes into account differences in size from surrounding objects. Furthermore, recognition processing that takes into account display position and adjacent objects becomes possible.

[0088] Second Embodiment Next, configuration examples of an inference device 600 and a learning device 700 will be described with reference to Figs. 15 to 18. Fig. 15 is a diagram showing an example of the hardware configuration of the inference device 600. Fig. 16 is a block diagram showing an example of the configuration of the inference device 600. Fig. 17 is a flowchart showing an example of the operation of the inference device 600. Fig. 18 is a block diagram showing an example of the configuration of the learning device 700.

[0089] In a second embodiment of the present disclosure, an inference device 600 will be described, which is an information processing device that converts feature quantities of objects extracted based on image data into relationship feature quantities, which are values ​​that take into account the positional relationships between the objects as well as the appearances of the objects. Also described will be a learning device 700 that trains parameters of a model used by the inference device 600 when converting feature quantities into relationship feature quantities. Fig. 15 shows an example of the hardware configuration of the inference device 600. Referring to Fig. 15, the inference device 600 has the following hardware configuration, as an example. CPU (Central Processing Unit) 601 (arithmetic device) ROM (Read Only Memory) 602 (storage device) RAM (Random Access Memory) 603 (storage device) Programs 604 loaded into RAM 603 Storage device 605 for storing the programs 604 Drive device 606 for reading and writing data from and to a storage medium 610 external to the information processing device Communication interface 607 for connecting to a communication network 611 external to the information processing device Input / output interface 608 for inputting and outputting data Bus 609 for connecting the various components

[0090] 16 by the CPU 601 acquiring and executing the group of programs 604. The group of programs 604 is stored in advance in the storage device 605 or the ROM 602, for example, and is loaded into the RAM 603 or the like by the CPU 601 for execution as needed. The group of programs 604 may be supplied to the CPU 601 via the communication network 611, or may be stored in advance in the recording medium 610, with the drive device 606 reading out the programs and supplying them to the CPU 601.

[0091] 15 shows an example of the hardware configuration of the inference device 600. The hardware configuration of the inference device 600 is not limited to the above-described case. For example, the inference device 600 may be configured with only a part of the above-described configuration, such as not including the drive device 606. Furthermore, the CPU 601 may be a GPU, as exemplified in the first embodiment.

[0092] The object position acquisition unit 621 acquires position information indicating the positions of objects included in image data having multiple objects as subjects. For example, the object position acquisition unit 621 can acquire position information based on the results of detecting objects from the image data using an object detector or the like.

[0093] The feature extraction unit 622 extracts feature amounts of an object included in the image data. For example, the feature extraction unit 622 extracts feature amounts according to the appearance of the object, such as the shape, color, and pattern of the object, as feature amounts of the object. The feature extraction unit 622 may extract feature amounts by inputting the image data to a feature extractor that has been trained in advance.

[0094] The feature conversion unit 623 converts the features into relationship features, which are values ​​that take into account the positional relationships between the objects as well as the appearance of the objects, by inputting the features and position information extracted from the first object in the image data and the features and position information of one or more objects different from the first object into a model that has been trained in advance using machine learning.

[0095] The above is an example of the configuration of the inference device 600. Next, an example of the operation of the inference device 600 will be described with reference to FIG.

[0096] Fig. 17 shows an example of the operation of the inference device 600. Referring to Fig. 17, the object position acquisition unit 621 acquires position information indicating the positions of objects included in image data having multiple objects as subjects (step S301).

[0097] The feature extraction unit 622 extracts feature amounts of an object included in the image data. For example, the feature extraction unit 622 extracts feature amounts according to the appearance of the object, such as the shape, color, and pattern of the object (step S302).

[0098] The feature conversion unit 623 converts the features into relationship features, which are values ​​that take into account the positional relationships between objects as well as the appearance of the objects, by inputting the features and position information extracted from the first object in the image data and the features and position information of one or more objects different from the first object into a model that has been trained in advance using machine learning (step S303).

[0099] As described above, the inference device 600 includes an object position acquisition unit 621, a feature extraction unit 622, and a feature conversion unit 623. With this configuration, the feature conversion unit 623 can use the position information acquired by the object position acquisition unit 621 and the feature amounts extracted by the feature extraction unit 622 to convert the extracted feature amounts into relationship feature amounts. As a result, any information processing device can perform processing such as clustering using relationship feature amounts that take into account appearance and positional relationships. This makes it possible to improve the accuracy when the information processing device performs processing such as clustering using image data.

[0100] The above-described inference device 600 can be realized by incorporating a predetermined program into an information processing device such as the inference device 600. Specifically, a program according to another embodiment of the present disclosure is a program for causing an information processing device to acquire position information indicating the positions of objects included in image data having a plurality of objects as subjects, extract feature amounts of the objects included in the image data, and input the feature amount and position information extracted from a first object in the image data and the feature amounts and position information of one or more objects different from the first object into a model previously trained by machine learning, thereby converting the feature amounts into relationship feature amounts that are values ​​that take into account the positional relationships between the objects as well as the appearances of the objects.

[0101] Furthermore, an inference method executed by an information processing device such as the inference device 600 described above is a method in which the information processing device acquires position information indicating the positions of objects included in image data having multiple objects as subjects, extracts feature quantities of the objects included in the image data, and inputs the feature quantities and position information extracted from a first object in the image data and the feature quantities and position information of one or more objects different from the first object into a model previously trained by machine learning, thereby converting the feature quantities into relationship feature quantities, which are values ​​that take into account the positional relationships between the objects as well as the appearances of the objects.

[0102] Any program having the above-described configuration, or a computer-readable recording medium having the program recorded thereon, or an inference method, etc., can achieve the same functions and effects as the above-described inference device 600, and therefore can achieve the above-described objective of the present disclosure.

[0103] Furthermore, the model used by the inference device 600 described above when converting features into relationship features can be learned using a learning device 700. Fig. 18 shows an example configuration of the learning device 700. The learning device 700 can realize the functions of an object position acquisition unit 721, a feature extraction unit 722, a feature conversion unit 723, and a learning unit 724 shown in Fig. 18 by having a CPU acquire and execute a group of programs. The hardware configuration of the learning device 700 may be the same as the hardware configuration of the inference device 600 described with reference to Fig. 15 .

[0104] The object position acquisition unit 721 acquires position information indicating the positions of objects included in image data having multiple objects as subjects. For example, the object position acquisition unit 721 can acquire position information based on the result of detecting an object from the image data using an object detector or the like. The object position acquisition unit 721 may acquire position information by a method other than the above example, such as by accepting input via an operation input unit.

[0105] The feature extraction unit 722 extracts feature amounts of an object included in the image data. For example, the feature extraction unit 722 extracts feature amounts according to the appearance of the object, such as the shape, color, and pattern of the object, as feature amounts of the object. The feature extraction unit 722 may extract feature amounts by inputting the image data to a feature extractor that has been trained in advance.

[0106] The feature conversion unit 723 inputs the features and position information extracted from a first object in the image data and the features and position information of one or more objects different from the first object into a model, and converts the features into relationship features, which are values ​​that take into account the positional relationships between the objects as well as the appearance of the objects.

[0107] The learning unit 724 uses the results of the conversion by the feature conversion unit 723 and pre-stored correct answer information to train the parameters of the model used by the feature conversion unit 723 when converting features into relationship features.

[0108] With this configuration, inference device 600 can perform conversion processing using the learning results from learning device 700. Therefore, learning device 700 having the above configuration can also achieve the object of the present invention, just like inference device 600.

[0109] <Supplementary Notes> Part or all of the above-described embodiments can also be described as follows: An outline of the inference device and the like in the present disclosure will be described below. However, the present disclosure is not limited to the following configuration.

[0110] (Supplementary Note 1) An inference device comprising: an object position acquisition unit that acquires position information indicating the positions of objects included in image data having a plurality of objects as subjects; a feature extraction unit that extracts feature amounts of the objects included in the image data; and a feature conversion unit that converts the feature amounts and position information extracted from a first object in the image data and the feature amounts and position information of one or more objects different from the first object into a model previously trained by machine learning, thereby converting the feature amounts into relationship feature amounts that are values ​​that take into account the positional relationships between the objects as well as the appearances of the objects. (Supplementary Note 2) The inference device according to Supplementary Note 1, further comprising: a processing unit that performs processing using the relationship feature amounts converted by the feature conversion unit, wherein the processing unit classifies the objects included in the image data into a plurality of clusters using the relationship feature amounts converted by the feature conversion unit. (Supplementary Note 3) The inference device according to Supplementary Note 1 or Supplementary Note 2, further comprising: a processing unit that performs processing using the relationship feature amounts converted by the feature conversion unit, wherein the processing unit recognizes objects in the image data by matching the relationship feature amounts converted by the feature conversion unit with pre-stored feature amounts. (Supplementary Note 4) The inference device according to any one of Supplementary Notes 1 to 3, wherein the feature conversion unit converts features into relationship features by inputting, into the model, feature amounts and position information extracted from the first object of interest in the image data and feature amounts and position information of one or more objects selected based on predetermined criteria from among the objects included in the image data and other than the first object of interest. (Supplementary Note 5) The inference device according to Supplementary Note 4, wherein the feature conversion unit selects, from among the objects included in the image data, an object that is displayed on the same shelf as the first object of interest. (Supplementary Note 6) The inference device according to Supplementary Note 4 or Supplementary Note 5, wherein the feature conversion unit selects, from among the objects included in the image data, an object adjacent to the first object of interest. (Supplementary Note 7) An inference device according to any one of Supplementary Notes 1 to 6, wherein the object position acquisition unit acquires a rectangle surrounding the object by performing object detection on the image data, and acquires the center coordinates of the acquired rectangle as position information.(Supplementary Note 8) The inference device according to any one of Supplementary Notes 1 to 7, wherein the object position acquisition unit acquires a rectangle surrounding an object by performing object detection on the image data, and acquires rectangular coordinates that are coordinates of a predetermined location in the acquired rectangle as position information. (Supplementary Note 9) The inference device according to any one of Supplementary Notes 1 to 8, wherein, in response to inputting features and position information extracted from a first object and features and position information of one or more objects different from the first object into the model, the feature conversion unit converts the features into relationship features by weighting the features of each object according to the degree of association between the features and the first object and calculating a weighted sum. (Supplementary Note 10) The inference device according to Supplementary Note 2, wherein the processing unit determines a threshold based on a similarity calculated based on the relationship features of adjacent objects in the image data and a similarity calculated based on the relationship features of non-adjacent objects, and classifies the objects included in the image data into a plurality of clusters using the determined threshold. (Supplementary Note 11) An inference method in which an information processing device acquires position information indicating the positions of objects included in image data having a plurality of objects as subjects, extracts feature amounts of the objects included in the image data, and inputs the feature amount and position information extracted from a first object in the image data and the feature amounts and position information of one or more objects different from the first object into a model trained in advance by machine learning, thereby converting the feature amount into a relationship feature amount that is a value taking into account the positional relationships between the objects as well as the appearances of the objects. (Supplementary Note 12) A program for realizing a process in which an information processing device acquires position information indicating the positions of objects included in image data having a plurality of objects as subjects, extracts feature amounts of the objects included in the image data, and inputs the feature amount and position information extracted from a first object in the image data and the feature amounts and position information of one or more objects different from the first object into a model trained in advance by machine learning, thereby converting the feature amount into a relationship feature amount that is a value taking into account the positional relationships between the objects as well as the appearances of the objects.(Supplementary Note 13) A learning device comprising: an object position acquisition unit that acquires position information indicating positions of objects included in image data having a plurality of objects as subjects; a feature extraction unit that extracts feature amounts of the objects included in the image data; a feature conversion unit that converts the feature amounts into relationship feature amounts that are values ​​that take into account the positional relationships between objects as well as the appearances of the objects by inputting the feature amounts and position information extracted from a first object in the image data and the feature amounts and position information of one or more objects different from the first object into a model; and a learning unit that trains parameters of the model that the feature conversion unit uses when converting the feature amounts into relationship feature amounts, using the results of the conversion by the feature conversion unit and correct answer information stored in advance.

[0111] Note that some or all of the configurations described in Supplementary Notes 2 to 10 that are dependent on the inference device described in Supplementary Note 1 may also be dependent in a similar dependent relationship on the inference method described in Supplementary Note 11, the program described in Process 12, etc. Furthermore, not limited to Supplementary Notes 11 and 12, but also to various hardware, software, various recording means for recording software, or systems, some or all of the configurations described as the Supplements may be made dependent on the same within the scope of each of the above-mentioned embodiments. Furthermore, the learning device described in Supplementary Note 13 may also have a configuration corresponding to Supplementary Notes 2 to 10.

[0112] The programs described in the above embodiments and appendices may be stored in a storage device or a computer-readable recording medium, such as a portable medium such as a flexible disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0113] Although the present disclosure has been described above with reference to the above-described embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0114] 100 Monitoring system 200 Imaging device 300 Learning device 310 Operation input unit 320 Screen display unit 330 Communication I / F unit 340 Memory unit 341 Learning data information 342 Correct answer information 343 Program 350 Arithmetic processing unit 351 Learning data acquisition unit 352 Object position acquisition unit 353 Feature extraction unit 354 Feature conversion unit 355 Learning unit 356 Output unit 400 Inference device 410 Operation input unit 420 Screen display unit 430 Communication I / F unit 440 Memory unit 441 Image data information 442 Clustering result information 443 Program 444 Comparison target information 450 Arithmetic processing unit 451 Image data acquisition unit 452 Object position acquisition unit 453 Feature extraction unit 454 Feature conversion unit 455 Clustering unit 456 Output unit 457 Recognition unit 500 Shelf 600 Inference device 601 CPU 602 ROM 603 RAM 604 Program group 605 Storage device 606 Drive device 607 Communication interface 608 Input / output interface 609 Bus 610 Recording medium 611 Communication network 621 Object position acquisition unit 622 Feature extraction unit 623 Feature conversion unit 700 Learning device 721 Object position acquisition unit 722 Feature extraction unit 723 Feature conversion unit 724 Learning unit

Claims

1. An object position acquisition unit that acquires position information indicating the position of an object included in image data of multiple objects, A feature extraction unit that extracts feature quantities of objects included in the aforementioned image data, A feature transformation unit inputs the feature quantities and positional information extracted from the first object in the image data, and the feature quantities and positional information of one or more objects different from the first object, into a model that has been pre-trained by machine learning, thereby transforming the feature quantities into relational feature quantities that take into account the positional relationships between objects as well as the appearance of the objects. has Reasoning device.

2. An inference device according to claim 1, The processing unit has a processing unit that performs processing using the relational feature quantities converted by the feature conversion unit, The processing unit classifies the objects contained in the image data into multiple clusters using the relational features converted by the feature conversion unit. Reasoning device.

3. An inference device according to claim 1, The processing unit has a processing unit that performs processing using the relational feature quantities converted by the feature conversion unit, The processing unit recognizes objects in image data by matching the relational features converted by the feature conversion unit with pre-stored features. Reasoning device.

4. An inference device according to claim 1, The feature conversion unit converts the feature quantities into relational features by inputting into the model the feature quantities and positional information extracted from the first object of interest within the image data, and the feature quantities and positional information of one or more objects selected from among the objects in the image data that are different from the first object of interest based on predetermined criteria. Reasoning device.

5. An inference device according to claim 4, The feature conversion unit selects objects from the image data that are displayed on the same shelf as the first object of interest. Reasoning device.

6. An inference device according to claim 4, The feature conversion unit selects objects adjacent to the first object of interest from among the objects included in the image data. Reasoning device.

7. An inference device according to claim 2, The processing unit determines a threshold based on the similarity calculated based on the relationship features of adjacent objects in the image data and the similarity calculated based on the relationship features of non-adjacent objects, and uses the determined threshold to classify the objects included in the image data into multiple clusters. Reasoning device.

8. Information processing device, We obtain location information indicating the position of objects included in image data with multiple objects as subjects. Extract the feature quantities of the object contained in the aforementioned image data, By inputting the feature quantities and positional information extracted from the first object within the aforementioned image data, along with the feature quantities and positional information of one or more objects different from the first object, into a model pre-trained by machine learning, the feature quantities are converted into relational features, which are values ​​that take into account the positional relationships between objects as well as the appearance of the objects. Reasoning method.

9. In an information processing device, We obtain location information indicating the position of objects included in image data with multiple objects as subjects. Extract the feature quantities of the object contained in the aforementioned image data, By inputting the feature quantities and positional information extracted from the first object within the aforementioned image data, along with the feature quantities and positional information of one or more objects different from the first object, into a model pre-trained by machine learning, the feature quantities are converted into relational features, which are values ​​that take into account the positional relationships between objects as well as the appearance of the objects. A program to perform a process.

10. An object position acquisition unit that acquires position information indicating the position of an object included in image data of multiple objects, A feature extraction unit that extracts feature quantities of objects included in the aforementioned image data, A feature transformation unit inputs feature quantities and positional information extracted from a first object within the aforementioned image data, and feature quantities and positional information of one or more objects different from the first object, into a model, thereby transforming the feature quantities into relational feature quantities that take into account the positional relationships between objects as well as the appearance of the objects. A learning unit that trains the parameters of the model used by the feature transformation unit when it transforms features into relational features, using the results transformed by the feature transformation unit and the ground truth information stored in advance. has Learning device.