Inference apparatus, inference method, and recording medium

Abstract

The inference device (3) is configured to include: an image signal acquisition unit (11) that acquires an image signal representing an inference target image that is an image of an existing detection target object, in one or more different cases in which a domain of the image is different from a learning image and / or a recognition task is different from a task learned in advance; and a feature quantity extraction unit (12) that supplies the image signal acquired by the image signal acquisition unit (11) to a learned model (1a) of the learning image, and acquires an inference-time feature quantity that is a feature quantity in which a plurality of feature quantities of the detection target object represented in the inference target image are combined after being blurred, from the learned model (1a). In addition, the inference device (3) includes an object recognition unit (15) that recognizes the detection target object represented in the inference target image, based on a representative feature quantity that is a registered feature quantity of the detection target object represented in a transformation image that is an object in which the domain of the image and the recognition task are the same as those of the inference target image, and the inference-time feature quantity acquired by the feature quantity extraction unit (12).

Description

Technical Field

[0001] This disclosure relates to inference devices, inference methods, and recording media. Background Technology

[0002] When there is limited learning data in the adapted domain, learning devices for learning multi-layer neural networks (hereinafter referred to as "multi-layer NNs") are available as efficient learning devices for neural networks (see, for example, Patent Document 1). The domain refers to the type of image represented by the learning data, such as RGB images and infrared camera images (hereinafter referred to as "TIR images"), where the types of images differ.

[0003] The learning device comprises a first learning unit, a first generation unit, and a second learning unit. The first learning unit uses a first data set to learn a first multi-layer neural network (NN). The first generation unit generates a second multi-layer NN between the first layer and the second layer following the first layer in the first multi-layer NN. The second learning unit uses a second data set with characteristics different from the first data set to learn the second multi-layer NN. The first and second data sets are each training data.

[0004] Existing technical documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-185127 Summary of the Invention

[0006] In the learning apparatus disclosed in Patent Document 1, the task corresponding to the learning data and the reasoning task performed when obtaining the output data of the neural network are sometimes different. For example, if the task corresponding to the learning data is an image recognition task, but the task during reasoning is an object recognition task, then the task corresponding to the learning data and the task during reasoning are different. Furthermore, the domain of the image represented by the learning data and the domain of the image during reasoning are sometimes different. In cases where either the task or the domain differs, there is a problem that the accuracy of reasoning may sometimes deteriorate due to the task at the time of reasoning.

[0007] This disclosure was made to solve the problems described above, and its purpose is to provide a reasoning device that can suppress the deterioration of reasoning accuracy even under more than one different conditions in the task and domain.

[0008] The inference apparatus disclosed herein includes: an image signal acquisition unit that acquires an image signal representing an inference object image as an image reflecting a detected object, when one or more of the following conditions are different: the domain of the image differs from that of a learning image, or the recognition task differs from a previously learned task; and a feature extraction unit that provides the image signal acquired by the image signal acquisition unit to a learning model that has completed learning of the learning image, and obtains inference-time features from the learning model, wherein the inference-time features are features formed by combining multiple features of the detected object reflected in the inference object image after blurring each feature. Furthermore, the inference apparatus includes an object recognition unit that recognizes a detected object reflected in the inference object image based on a representative feature of the registered features of the detected object reflected in the transformation image and the inference-time features obtained by the feature extraction unit, wherein the transformation image is an object whose domain and recognition task are the same as those of the inference object image.

[0009] According to this disclosure, the degradation of inference accuracy can be suppressed even in cases where there are more than one difference in the task and the domain. Attached Figure Description

[0010] Figure 1 This is a structural diagram showing the inference device 3 according to Embodiment 1.

[0011] Figure 2 This is a hardware structure diagram showing the hardware of the inference device 3 according to Embodiment 1.

[0012] Figure 3 This is a hardware structure diagram of a computer that implements the inference device 3 through software or firmware.

[0013] Figure 4 This is a structural diagram showing the learning device 6.

[0014] Figure 5 This is a hardware structure diagram showing the hardware of the learning device 6.

[0015] Figure 6 This is a hardware structure diagram of a computer that implements the learning device 6 through software or firmware.

[0016] Figure 7 This is a flowchart showing the processing procedure of the inference device 3 during domain transformation.

[0017] Figure 8 This is a flowchart illustrating the reasoning method of the reasoning device 3 when it is used for object recognition.

[0018] Figure 9 This is a structural diagram showing the inference device 3 according to Embodiment 2.

[0019] Figure 10 This is a hardware structure diagram showing the hardware of the inference device 3 according to Embodiment 2.

[0020] Figure 11 This is a structural diagram showing the inference device 3 according to embodiment 3.

[0021] Figure 12 This is a hardware structure diagram showing the hardware of the inference device 3 according to Embodiment 3.

[0022] Figure 13 This is a structural diagram showing the inference device 3 according to embodiment 4.

[0023] Figure 14 This is a hardware structure diagram showing the hardware of the inference device 3 according to Embodiment 4. Detailed Implementation

[0024] The following description, in order to illustrate the present disclosure in more detail, describes the manner in which the present disclosure is carried out in accordance with the accompanying drawings.

[0025] Implementation method 1.

[0026] Figure 1 This is a structural diagram showing the inference device 3 according to Embodiment 1.

[0027] Figure 2 This is a hardware structure diagram showing the hardware of the inference device 3 according to Embodiment 1.

[0028] exist Figure 1 In this model, the storage unit 1 is implemented, for example, by a hard disk or RAM (Random Access Memory).

[0029] Model storage unit 1 stores the learning model 1a.

[0030] Learning model 1a is implemented, for example, through multi-layer neural networks (Deep Neural Networks: DNNs). Among DNNs are CNNs (Convolutional Neural Networks), which are convolutional neural networks.

[0031] In learning model 1a, during learning, an image signal representing the learning image is provided as learning data, and the learning of the learning image is completed. The learning image is, for example, an image used for an image recognition task.

[0032] The image used as the domain for learning can be of any kind, such as an RGB image, a TIR image, or any image generated by a CG simulator.

[0033] exist Figure 1 In the inference device 3 shown, for ease of explanation, the learning images are assumed to be RGB images. The learning model 1a is a model that is provided with a large number of RGB images and learns from RGB images.

[0034] When the learning model 1a is provided with an image signal representing a transformed image that reflects an image of a detected object, provided by the feature extraction unit 12 (described later) in a different domain and for a different recognition task than the learning image, a feature vector is output to the feature extraction unit 12. This feature vector represents a feature formed by combining multiple feature quantities after blurring multiple feature quantities of the detected object reflected in the transformed image.

[0035] The transformation image can be any image in the image domain and at least one image different from the learning image in the recognition task. Figure 1 In the inference device 3 shown, for ease of explanation, the image used for transformation is assumed to be a TIR image.

[0036] The CNNs that implement learning model 1a are very deep CNNs. For example, a ResNet with 101 layers exists as a very deep CNN. Therefore, in tasks such as object recognition, when an image signal is provided to the input layer of learning model 1a, the feature vectors output from the output layer of learning model 1a represent high-dimensional feature quantities. High-dimensional feature quantities include feature quantities of multiple dimensions, and feature vectors representing high-dimensional feature quantities are, for example, represented using tensors.

[0037] The low-dimensional features output from the shallow layers of the multi-level hidden layers contained in the learning model 1a are, for example, features representing color, brightness, or orientation. Therefore, the low-dimensional features depend on the domain of the image represented by the image signal provided to the input layer. That is, the features represented by the feature vector output from the shallow layers of the learning model 1a when the input layer is provided with an RGB image signal are sometimes quite different from the features represented by the feature vector output from the shallow layers of the learning model 1a when the input layer is provided with a TIR image signal.

[0038] On the other hand, the high-dimensional features output from the sufficiently deep intermediate layers of learning model 1a represent the conceptual features of the detected object. Therefore, these high-dimensional features become conceptual information of the image represented by the image signal provided to the input layer, with extremely low domain dependence. Furthermore, by employing high-dimensional features from deeper layers, general information with low task dependence can be obtained. This includes features of conceptual objects, such as known "Objectness" or "Informativeness".

[0039] That is, when the input layer of the learning model 1a is provided with an image signal of an RGB image, the high-dimensional feature quantity represented by the feature vector output from the output layer of the learning model 1a is small in difference from the high-dimensional feature quantity represented by the feature vector output from the output layer of the learning model 1a when the input layer of the learning model 1a is provided with an image signal of a TIR image.

[0040] Therefore, when the learning model 1a is implemented by CNNs and the inference device 3 uses high-dimensional features represented by feature vectors output from sufficiently deep intermediate layers of CNNs, the domain dependence and recognition task dependence of the image signal provided to the input layer are reduced.

[0041] Regarding the object to be detected, this could be a product that, in addition to distinguishing between normal and abnormal, also detects the location where the abnormality occurs. Specifically, objects to be detected could include circuit boards, rolled sheets, or molded plastic products.

[0042] Camera 2 can be implemented, for example, by using an infrared camera.

[0043] Camera 2 takes a picture of the object being detected.

[0044] When the inference device 3 registers images of different domains than during learning (hereinafter referred to as "domain transformation time"), the camera 2 outputs, for example, an image signal representing a TIR image of the existing object being detected as an image signal representing a transformed image of the existing object being detected to the inference device 3.

[0045] When the inference device 3 identifies the object to be detected (hereinafter referred to as "object identification time"), the camera 2 outputs an image signal representing a TIR image of the object to be detected as an image signal representing an inference object image of the object to be detected to the inference device 3.

[0046] exist Figure 1 In this process, camera 2 outputs an image signal representing the transformed image to inference device 3. However, this is just one example; the image signal can also be output to inference device 3 from a storage unit (not shown) that stores an image signal representing the transformed image of the object being detected.

[0047] The reasoning device 3 includes an image signal acquisition unit 11, a feature extraction unit 12, a representative feature registration unit 13, a representative feature storage unit 14, and an object recognition unit 15.

[0048] exist Figure 1 In the inference device 3 shown, the model storage unit 1 is located outside the inference device 3. However, this is just an example; for instance, the model storage unit 1 may also be located inside the inference device 3, and the learning model 1a may also be built into the feature extraction unit 12.

[0049] Image signal acquisition unit 11, for example, via Figure 2 The image signal acquisition circuit 21 shown is used to achieve this.

[0050] During domain transformation, the image signal acquisition unit 11 acquires an image signal from the camera 2 representing a transformed image of the existing detected object.

[0051] Then, the image signal acquisition unit 11 outputs the image signal representing the image for transformation to the feature extraction unit 12.

[0052] During object identification, the image signal acquisition unit 11 acquires an image signal from the camera 2 representing an image of the inference object that reflects the existing object to be detected.

[0053] Then, the image signal acquisition unit 11 outputs the image signal representing the image of the reasoning object to the feature extraction unit 12.

[0054] Regarding the images used for transformation and the images of the objects of inference, the domains of the images are the same, for example, both are TIR images.

[0055] Feature extraction unit 12, for example, through Figure 2 The feature quantity extraction circuit 22 shown is used to achieve this.

[0056] During domain transformation, the feature extraction unit 12 provides the image signal obtained by the image signal acquisition unit 11 to the learning model 1a, and obtains the feature vector representing the representative feature from the learning model 1a. The representative feature is a feature formed by combining multiple feature quantities after blurring multiple feature quantities of the object being detected in the image being transformed.

[0057] The feature extraction unit 12 outputs the feature vector to the representative feature registration unit 13.

[0058] During object recognition, the feature extraction unit 12 provides the image signal obtained by the image signal acquisition unit 11 to the learning model 1a, and obtains the feature vector representing the feature quantity during inference from the learning model 1a. The feature quantity during inference is a feature quantity formed by combining multiple feature quantities after the feature quantities of the detected object that are reflected in the image of the inference object are blurred.

[0059] The feature extraction unit 12 outputs the feature vector to the object recognition unit 15.

[0060] As a process that blurs multiple features separately, the "Pooling Operation" is known.

[0061] Representative characteristic registration department 13, for example, through Figure 2 The representative feature quantity registration circuit 23 shown is used to implement this.

[0062] The representative feature quantity registration unit 13 registers the representative feature quantities obtained by the feature quantity extraction unit 12.

[0063] That is, the representative feature quantity registration unit 13 obtains the feature vector representing the representative feature quantity from the feature quantity extraction unit 12 and stores the feature vector in the representative feature quantity storage unit 14.

[0064] Representative feature storage unit 14, for example, through Figure 2 The representative feature quantity storage circuit 24 shown is used to implement this.

[0065] The feature storage unit 14 stores the feature vectors representing the feature quantities.

[0066] Object recognition unit 15, for example, through Figure 2 The object recognition circuit 25 shown is used to implement this.

[0067] The object identification unit 15 obtains a feature vector representing the inference time feature quantity of the detected object that is reflected in the inference object image from the feature quantity extraction unit 12, and obtains a feature vector representing the representative feature quantity from the representative feature quantity storage unit 14.

[0068] The object identification unit 15 identifies the detected object reflected in the image of the reasoning object based on representative feature quantities and inference time feature quantities.

[0069] Specifically, the object identification unit 15 calculates the similarity between a feature vector representing a characteristic quantity and a feature vector representing a characteristic quantity during inference, and identifies the detected object projected onto the inference object image based on the similarity. More specifically, the object identification unit 15 compares the similarity with a threshold. Then, based on the comparison result of the similarity and the threshold, the object identification unit 15 identifies the detected object, for example, whether the detected object projected onto the inference object image is normal or abnormal. Furthermore, in the object identification unit 15, the detected object is classified into multiple levels, for example.

[0070] The object recognition unit 15 generates display data representing the recognition result of the detected object and outputs the display data to the display device 4.

[0071] The display device 4 displays the identification result of the detected object on a display screen (not shown) according to the display data output from the object recognition unit 15.

[0072] exist Figure 1 In this context, it is envisioned that the image signal acquisition unit 11, feature extraction unit 12, representative feature registration unit 13, representative feature storage unit 14, and object recognition unit 15, which are components of the reasoning device 3, each pass through a system such as... Figure 2 The dedicated hardware shown is used to implement this. That is, the reasoning device 3 is envisioned to be implemented through an image signal acquisition circuit 21, a feature extraction circuit 22, a representative feature registration circuit 23, a representative feature storage circuit 24, and an object recognition circuit 25.

[0073] Regarding the characteristic storage circuit 24, for example, it could be a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), disk, floppy disk, optical disk, high-density disk, mini disk, or DVD (Digital Versatile Disc).

[0074] The image signal acquisition circuit 21, the feature extraction circuit 22, the representative feature registration circuit 23, and the object recognition circuit 25 are each, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0075] The components of the inference device 3 are not limited to being implemented through dedicated hardware; the inference device 3 can also be implemented through software, firmware, or a combination of software and firmware.

[0076] Software or firmware is stored as a program in a computer's memory. A computer means the hardware that executes programs, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor).

[0077] Figure 3 This is a hardware structure diagram of a computer that implements the inference device 3 through software or firmware.

[0078] When the reasoning device 3 is implemented by software or firmware, a representative feature storage unit 14 is configured on the computer's memory 31. Programs for executing the various processing steps in the image signal acquisition unit 11, the feature extraction unit 12, the representative feature registration unit 13, and the object recognition unit 15 are stored in the memory 31. Then, the computer's processor 32 executes the program stored in the memory 31.

[0079] In addition, Figure 2 The diagram illustrates an example of implementing the various components of the inference device 3 using dedicated hardware. Figure 3 The diagram shows an example of implementing the inference device 3 using software or firmware. However, this is just one example; some of the components of the inference device 3 can be implemented using dedicated hardware, while the remaining components can be implemented using software or firmware.

[0080] Figure 4 This is a structural diagram showing the learning device 6.

[0081] Figure 5 This is a hardware structure diagram showing the hardware of the learning device 6.

[0082] The learning data storage unit 5 is implemented, for example, through a hard disk or RAM.

[0083] In the learning data storage unit 5, image signals representing images used for learning are stored as learning data.

[0084] The learning device 6 includes a learning data acquisition unit 41 and a learning processing unit 42.

[0085] Learning data acquisition section 41, for example, through Figure 5 The learning data acquisition circuit 51 shown is used to achieve this.

[0086] The learning data acquisition unit 41 acquires learning data from the learning data storage unit 5.

[0087] The learning data acquisition unit 41 outputs the learning data to the learning processing unit 42.

[0088] Learning Processing Unit 42, for example, through Figure 5 The learning processing circuit 52 shown is used to implement this.

[0089] The learning processing unit 42 obtains a large amount of learning data from the learning data acquisition unit 41.

[0090] The learning processing unit 42 provides each learning data to the learning model 1a, so that the learning model 1a learns the learning image represented by the image signal contained in each learning data.

[0091] When the learned model 1a is provided with an image signal, it outputs a feature vector corresponding to the image signal during domain transformation or object recognition.

[0092] exist Figure 4 In this context, it is envisioned that the learning data acquisition unit 41 and the learning processing unit 42, which are components of the learning device 6, respectively utilize, as shown in the example... Figure 5 The learning device 6 is implemented using dedicated hardware. Specifically, the learning device 6 is envisioned to be implemented using a learning data acquisition circuit 51 and a learning processing circuit 52.

[0093] The learning data acquisition unit 41 and the learning processing unit 42 are, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0094] The components of the learning device 6 are not limited to being implemented through dedicated hardware; the learning device 6 can also be implemented through software, firmware, or a combination of software and firmware.

[0095] Figure 6 This is a hardware structure diagram of a computer that implements the learning device 6 through software or firmware.

[0096] When the learning device 6 is implemented through software or firmware, the program for causing the computer to execute each processing step in the learning data acquisition unit 41 and the learning processing unit 42 is stored in the memory 61. Then, the computer's processor 62 executes the program stored in the memory 61.

[0097] In addition, Figure 5 The diagram illustrates an example of how the various components of the learning device 6 are implemented using dedicated hardware. Figure 6 The diagram shows an example of how the learning device 6 is implemented using software or firmware. However, this is just one example; some of the components of the learning device 6 can be implemented using dedicated hardware, while the remaining components can be implemented using software or firmware.

[0098] First, let me explain Figure 4 The operation of the learning device 6 shown.

[0099] The learning data storage unit 5 stores a large amount of learning data, including image signals representing images used for learning.

[0100] The learning data acquisition unit 41 of the learning device 6 acquires a large amount of learning data from the learning data storage unit 5.

[0101] The learning data acquisition unit 41 outputs each piece of learning data to the learning processing unit 42.

[0102] The learning processing unit 42 obtains various learning data from the learning data acquisition unit 41.

[0103] The learning processing unit 42 provides each learning data to the learning model 1a, so that the learning model 1a learns the learning image represented by the image signal contained in each learning data.

[0104] For example, when the image signal representing an RGB image is provided to the input layer, the completed learning model 1a outputs a feature vector representing the high-dimensional features of the detected object reflected in the RGB image as the feature vector corresponding to the image signal.

[0105] In learning model 1a, if the learning image used for learning is, for example, an RGB image and a TIR image is not used as the learning image, even if the object to be detected in the RGB image and the object to be detected in the TIR image are both normal objects, the feature vector output from the output layer when the image signal representing the RGB image is provided to the input layer is sometimes different from the feature vector output from the output layer when the image signal representing the TIR image is provided to the input layer.

[0106] However, the CNNs that implement model 1a are very deep CNNs, and the feature vectors output from the sufficiently deep intermediate layers of model 1a are vectors representing high-dimensional features. Therefore, the differences mentioned above are minor.

[0107] Furthermore, the feature quantity represented by the feature vector output from the sufficiently deep intermediate layer of learning model 1a, as described above, is a feature quantity formed by combining the feature quantities of multiple layers of the object being detected after blurring them. Therefore, the feature quantity represented by the feature vector is a robust feature quantity representing the domain dependency of the image and the dependency of the recognition task, respectively, after excluding them.

[0108] Next, the operation of the inference device 3 during domain transformation will be explained.

[0109] Figure 7 This is a flowchart showing the processing procedure of the inference device 3 during domain transformation.

[0110] Camera 2 captures an image of the target object. The target object captured by camera 2 is a normal target object.

[0111] The object being detected captured by camera 2 can also be an abnormal object. However, in industrial manufacturing production lines, for example, the probability of an abnormal object being detected is generally extremely low, making it sometimes difficult to capture abnormal objects. Therefore, here we assume that the object being detected captured by camera 2 is a normal object.

[0112] For example, camera 2 outputs an image signal representing a TIR image of the object being detected as an image signal representing a transformed image of the object being detected to inference device 3.

[0113] Image signal acquisition unit 11 acquires an image signal from camera 2 representing a transformed image of the object being detected. Figure 7 Step ST1).

[0114] The image signal acquisition unit 11 outputs the image signal representing the image to the feature extraction unit 12.

[0115] The feature extraction unit 12 acquires the image signal representing the image for transformation from the image signal acquisition unit 11.

[0116] Feature extraction unit 12 extracts the feature quantities of the detected object that are reflected in the transformed image from the image signal. Figure 7 Step ST2).

[0117] Specifically, the feature extraction unit 12 provides the image signal to the learning model 1a and obtains a feature vector representing the representative feature from the learning model 1a. This representative feature is a feature formed by combining multiple feature values ​​of the detected object in the transformed image after blurring each feature value. Figure 7 Step ST2).

[0118] The feature extraction unit 12 outputs the feature vector to the representative feature registration unit 13.

[0119] The characteristic quantity registration unit 13 obtains the characteristic vector from the characteristic quantity extraction unit 12.

[0120] The representative characteristic quantity registration unit 13 registers the representative characteristic quantity represented by the characteristic vector. Figure 7 Step ST3).

[0121] Specifically, the representative feature registration unit 13 registers representative features by storing feature vectors in the representative feature storage unit 14.

[0122] Here, the feature vector representing the feature quantity is represented by a Tensor. Tensors can represent information of higher dimensions than vectors and are sometimes called feature maps.

[0123] Tensors can represent high-dimensional information. Therefore, when the feature vector represented by the Tensor is stored as is in the feature storage unit 14 by the feature registration unit 13, a large amount of processing time is sometimes required when the object recognition unit 15 compares the feature vectors.

[0124] In order to shorten the processing time required when comparing feature vectors in the object identification unit 15, the representative feature registration unit 13 may also transform the feature vector representing the representative feature into a one-hot vector with a dimension less than that of a Tensor, and store the one-hot vector in the representative feature storage unit 14.

[0125] Even if the feature vector registered by the feature registration unit 13 is either a Tensor or a heat vector, it is still a feature vector representing high-dimensional information of hundreds of dimensions. Therefore, even if there are some deviations among multiple normal detection objects, the feature vector will still become a feature vector that represents the characteristics of the normal detection objects in high dimensions.

[0126] Next, the operation of the reasoning device 3 during object recognition will be explained.

[0127] Figure 8 This is a flowchart illustrating the reasoning method of the reasoning device 3 when it is used for object recognition.

[0128] Camera 2 photographs the detected object. It is unclear whether the detected object photographed by camera 2 is a normal object or an abnormal object.

[0129] Camera 2 outputs the image signal representing the TIR image of the existing detected object to the inference device 3 as the image signal representing the inference object image of the existing detected object.

[0130] Image signal acquisition unit 11 acquires an image signal from camera 2 representing an image of the inference object that is being detected. Figure 8 Step ST11).

[0131] The image signal acquisition unit 11 outputs the image signal representing the image of the reasoning object to the feature extraction unit 12.

[0132] The feature extraction unit 12 acquires an image signal representing the image of the reasoning object from the image signal acquisition unit 11.

[0133] Feature extraction unit 12 extracts the features of the detected object that are reflected in the image of the inference object from the image signal. Figure 8 Step ST12).

[0134] Specifically, the feature extraction unit 12 provides the image signal to the learning model 1a and obtains the feature vector representing the inference time feature from the learning model 1a. The inference time feature is a feature formed by combining multiple feature quantities of the probe object that appears in the inference object image after blurring them.

[0135] The feature extraction unit 12 outputs the feature vector to the object recognition unit 15.

[0136] The object recognition unit 15 obtains the feature vector from the feature extraction unit 12 and obtains the feature vector representing the representative feature from the representative feature storage unit 14.

[0137] The object identification unit 15 identifies the object to be detected in the image of the inference object based on the representative feature quantity and the inference-time feature quantity represented by the feature vector output from the feature quantity extraction unit 12. Figure 8 Step ST13).

[0138] Specifically, the object recognition unit 15 calculates the similarity between the feature vector representing the representative feature quantity and the feature vector representing the feature quantity during reasoning. For example, by calculating the inner product of the feature vector representing the representative feature quantity and the feature vector representing the feature quantity during reasoning, the similarity between the feature vector representing the representative feature quantity and the feature vector representing the feature quantity during reasoning can be calculated.

[0139] The object recognition unit 15 compares similarity with a threshold, and based on the comparison results of similarity and threshold, identifies whether the detected object reflected in the inference object image is normal or abnormal.

[0140] That is, in the object recognition unit 15, if the similarity is above the threshold, the object to be detected is determined to be normal; if the similarity is below the threshold, the object to be detected is determined to be abnormal.

[0141] The threshold can be stored in the internal memory of the object recognition unit 15 or provided from the outside of the inference device 3.

[0142] exist Figure 1 The inference device shown illustrates an example where the object recognition unit 15 performs binary classification on the detected object, identifying whether the detected object is normal or abnormal. However, this is merely an example; the object recognition unit 15 can also classify the detected object into multiple levels. Embodiments 2 to 4 show examples of classifying the detected object into multiple levels.

[0143] When the learning image is, for example, an RGB image, and the transformation image and the inference object image are, for example, TIR images, the domain of the learning image is different from the domains in the transformation image and the inference object image, but the domain of the transformation image is the same as the domain of the inference object image.

[0144] Therefore, if the object being detected and reflected in the image of the object being inferred is a normal object, the representative feature quantity obtained by the feature extraction unit 12 during domain transformation will be approximately the same as the inference feature quantity obtained by the feature extraction unit 12 during object identification.

[0145] On the other hand, if the object being detected and reflected in the image of the object being inferred is an anomalous object, the representative feature quantity obtained by the feature quantity extraction unit 12 during domain transformation will be significantly different from the inference feature quantity obtained by the feature quantity extraction unit 12 during object identification.

[0146] Therefore, by comparing similarity and threshold by the object recognition unit 15, the detected object can be identified with high precision.

[0147] The object recognition unit 15 generates display data representing the recognition results of the detected object.

[0148] The object recognition unit 15 outputs display data to the display device 4.

[0149] The display device 4 displays the identification result of the detected object on a display screen (not shown) according to the display data output from the object recognition unit 15.

[0150] Therefore, inspectors can confirm whether the object being detected is normal or abnormal by observing the monitor.

[0151] In Embodiment 1 described above, the inference device 3 is configured to include: an image signal acquisition unit 11, which acquires an image signal representing an inference object image that reflects an image of a detected object, in cases where the domain of the image differs from that of the learning image and the recognition task differs from the previously learned task; and a feature extraction unit 12, which provides the image signal acquired by the image signal acquisition unit 11 to a learning model 1a that has completed learning of the learning image, and obtains inference-time features from the learning model 1a, wherein the inference-time features are features formed by combining multiple features of the detected object reflected in the inference object image after blurring each feature. Furthermore, the inference device 3 includes an object recognition unit 15, which recognizes the detected object reflected in the inference object image based on a representative feature of the registered features of the detected object reflected in the transformation image and the inference-time features obtained by the feature extraction unit 12, wherein the transformation image is an object whose domain and recognition task are the same as those of the inference object image. Therefore, the reasoning device 3 can suppress the deterioration of reasoning accuracy even under more than one different situation in the task and domain.

[0152] Furthermore, in Embodiment 1, the inference device 3 is configured as follows: the image signal acquisition unit 11 acquires an image signal representing a transformation image, and the feature extraction unit 12 provides the image signal representing the transformation image to the learning model 1a, acquiring representative features from the learning model 1a. These representative features are formed by combining multiple features of the target object projected onto the transformation image after blurring each feature. Additionally, the inference device 3 includes a representative feature registration unit 13 for registering the representative features acquired by the feature extraction unit 12. Therefore, the inference device 3 can register representative features that can be used in the identification processing of the target object.

[0153] exist Figure 1 In the inference device 3 shown, the feature extraction unit 12 provides image signals to the learning model 1a implemented by very deep CNNs, and obtains feature quantities from the learning model 1a that are formed by combining multiple feature quantities after the multiple feature quantities of the object being detected are blurred.

[0154] In the case of implementing the learning model 1a through very deep CNNs, as mentioned above, even if the domain of the inference object image represented by the image signal provided to the input layer of the learning model 1a is different from that of the learning image, and even if the detected object is different, the difference in the feature vector output from the output layer is small.

[0155] In contrast, when learning model 1a is implemented using a general neural network, if the domain of the inference object image represented by the image signal provided to the input layer of learning model 1a, or the probe object reflected in the inference object image, is different from the learning image, the difference in the feature vectors output from the output layer sometimes becomes larger.

[0156] However, the domain of the image used for transformation is the same as the domain of the image of the inference object. Therefore, even if the learning model 1a is implemented by a general neural network or the like, if the object to be detected that is mapped to the image of the inference object is a normal object, the representative feature obtained by the feature extraction unit 12 during domain transformation will be approximately the same as the inference feature obtained by the feature extraction unit 12 during object recognition.

[0157] On the other hand, if the object being detected and reflected in the image of the object being inferred is an anomalous object, the representative feature quantity obtained by the feature quantity extraction unit 12 during domain transformation will be significantly different from the inference feature quantity obtained by the feature quantity extraction unit 12 during object identification.

[0158] Therefore, even when the learning model 1a is implemented using a general neural network or the like, the object recognition unit 15 is able to identify the target object with high accuracy.

[0159] Implementation method 2.

[0160] In Embodiment 2, a reasoning device 3 is described, which includes an object identification unit 17 that identifies the type of the object to be detected.

[0161] Figure 9 This is a structural diagram showing the inference device 3 according to Embodiment 2. Figure 9 In, with Figure 1 The same symbols represent the same or equivalent parts, so the explanation is omitted.

[0162] Figure 10 This is a hardware structure diagram showing the hardware of the inference device 3 according to Embodiment 2. Figure 10 In, with Figure 2 The same symbols represent the same or equivalent parts, so the explanation is omitted.

[0163] Figure 9 The inference device 3 shown includes an image signal acquisition unit 11, a feature extraction unit 12, a representative feature registration unit 16, a representative feature storage unit 14, and an object recognition unit 17.

[0164] exist Figure 9In the inference device 3 shown, the model storage unit 1 is located outside the inference device 3. However, this is just an example; for instance, the model storage unit 1 may also be located inside the inference device 3, and the learning model 1a may also be built into the feature extraction unit 12.

[0165] exist Figure 9 The following example illustrates the inference device 3: The object being detected is identified by the object recognition unit 17 as either a passenger car, a truck, or a bus. In this case, the object is any one of a passenger car, a truck, or a bus. The task in this case is image classification.

[0166] However, this is just one example. As for the type of object identified by the object recognition unit 17, it can also identify whether the object is a passenger car with model name ○○, a passenger car with model name △△, or a passenger car with model name □□. In this case, the object is any one of the passenger car with model name ○○, the passenger car with model name △△, or the passenger car with model name □□. This task is called "Fine-grained Image Classification".

[0167] For example, even when it is possible to identify whether the object being detected is a passenger car, a truck, or a bus, the learning model 1a is provided with learning data including image signals representing the images used for learning.

[0168] Representative characteristic registration department 16, for example, through Figure 10 The representative feature quantity registration circuit 26 shown is used to implement this.

[0169] The representative feature quantity registration unit 16 registers the representative feature quantities obtained by the feature quantity extraction unit 12.

[0170] That is, the representative feature registration unit 16 obtains the feature vector of the representative feature quantity, which represents the feature quantity formed by combining multiple feature quantities of the detection object, i.e., the passenger car, which is reflected in the transformation image, after the multiple feature quantities are blurred respectively, and stores the feature vector in the representative feature quantity storage unit 14.

[0171] In addition, the representative feature registration unit 16 obtains the feature vector of the representative feature quantity, which represents the feature quantity formed by combining multiple feature quantities of the detection object, i.e., the truck, which is reflected in the transformation image after multiple feature quantities are blurred, and stores the feature vector in the representative feature quantity storage unit 14.

[0172] In addition, the representative feature registration unit 16 obtains the feature vector of the representative feature quantity, which represents the feature quantity formed by combining multiple feature quantities of the detection object, i.e., the bus, which is reflected in the transformation image after multiple feature quantities are blurred, and stores the feature vector in the representative feature quantity storage unit 14.

[0173] Object recognition unit 17, for example, through Figure 10 The object recognition circuit 27 shown is used to implement this.

[0174] The object identification unit 17 obtains the feature vector of the inference feature quantity from the feature quantity extraction unit 12, which represents the feature quantity formed by combining multiple feature quantities after blurring multiple feature quantities of the detected object in the inference object image, and obtains the feature vector of the representative feature quantity from the representative feature quantity storage unit 14.

[0175] The object identification unit 17 identifies the detected object reflected in the image of the reasoning object based on representative feature quantities and inference time feature quantities.

[0176] Specifically, the object identification unit 17 compares the representative feature quantities of multiple detection object objects of different types with the inference feature quantities extracted by the feature quantity extraction unit 12, and determines the representative feature quantity corresponding to the feature quantity extracted by the feature quantity extraction unit 12 among the representative feature quantities of the multiple detection object objects.

[0177] The object identification unit 17 identifies the type of the object being detected that is reflected in the image of the object being inferred based on the determination results of the representative feature quantities.

[0178] The object recognition unit 17 generates display data representing the recognition result of the detected object and outputs the display data to the display device 4.

[0179] exist Figure 9 In this context, it is envisioned that the image signal acquisition unit 11, feature extraction unit 12, representative feature registration unit 16, representative feature storage unit 14, and object recognition unit 17, which are components of the reasoning device 3, are respectively processed by, for example... Figure 10 The dedicated hardware shown is used to implement this. That is, the inference device 3 is envisioned to be implemented through an image signal acquisition circuit 21, a feature extraction circuit 22, a representative feature registration circuit 26, a representative feature storage circuit 24, and an object recognition circuit 27.

[0180] The image signal acquisition circuit 21, the feature extraction circuit 22, the representative feature registration circuit 26, and the object recognition circuit 27 are each, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0181] The components of the inference device 3 are not limited to being implemented through dedicated hardware; the inference device 3 can also be implemented through software, firmware, or a combination of software and firmware.

[0182] When the inference device 3 is implemented through software or firmware, etc., Figure 3 The memory 31 shown is configured with a representative feature storage unit 14. Programs for enabling the computer to execute various processing steps in the image signal acquisition unit 11, the feature extraction unit 12, the representative feature registration unit 16, and the object recognition unit 17 are stored in... Figure 3 The memory 31 shown. Then, Figure 3 The processor 32 shown executes the program stored in memory 31.

[0183] In addition, Figure 10 The diagram illustrates an example of implementing the various components of the inference device 3 using dedicated hardware. Figure 3 The diagram shows an example of implementing the inference device 3 using software or firmware. However, this is just one example; some of the components of the inference device 3 can be implemented using dedicated hardware, while the remaining components can be implemented using software or firmware.

[0184] Next, the operation of the inference device 3 during domain transformation will be explained.

[0185] Camera 2 photographs the target object. The target object photographed by camera 2 is any one of a passenger car, truck, or bus. However, the reasoning device 3 can also classify the target object into, for example, 1000 categories. Therefore, classifying the target object into three categories—passenger car, truck, or bus—is just one example.

[0186] For example, camera 2 outputs an image signal representing a TIR image of a passenger vehicle as an image signal representing a transformed image of a passenger vehicle that is the object of detection to inference device 3.

[0187] Camera 2 outputs an image signal representing a TIR image of a present truck to inference device 3 as an image signal representing a transformed image of a present truck as a detection object.

[0188] Additionally, the camera 2 outputs an image signal representing a TIR image of a present bus to the inference device 3 as an image signal representing a transformed image of a present bus as the object of detection.

[0189] The image used for transformation is not necessarily limited to TIR images, but in the case of RGB images, image recognition becomes more difficult at night, so the accuracy of object recognition sometimes deteriorates. Therefore, TIR images are used as the transformation images.

[0190] The image signal acquisition unit 11 acquires an image signal representing a transformation image of an existing passenger vehicle from the camera 2, and outputs the image signal representing the transformation image to the feature extraction unit 12.

[0191] The image signal acquisition unit 11 acquires an image signal representing a transformation image of an existing truck from the camera 2, and outputs the image signal representing the transformation image to the feature extraction unit 12.

[0192] In addition, the image signal acquisition unit 11 acquires an image signal representing a changing image of an existing bus from the camera 2, and outputs the image signal representing the changing image to the feature extraction unit 12.

[0193] The feature extraction unit 12 acquires an image signal representing a transformation image of an existing passenger vehicle from the image signal acquisition unit 11.

[0194] The feature extraction unit 12 extracts the features of the passenger vehicle that are reflected in the image of the transformation image from the image signal.

[0195] Specifically, the feature extraction unit 12 provides the image signal to the learning model 1a and obtains a feature vector Fv1 representing the multiple features of the passenger vehicle reflected in the transformation image after blurring and combining the multiple features.

[0196] The feature extraction unit 12 outputs the feature vector Fv1 to the representative feature registration unit 16.

[0197] In addition, the feature extraction unit 12 obtains an image signal representing a transformation image of an existing truck from the image signal acquisition unit 11.

[0198] The feature extraction unit 12 extracts the features of the truck that are mapped onto the image of the transformation from the image signal.

[0199] Specifically, the feature extraction unit 12 provides the image signal to the learning model 1a and obtains a feature vector Fv2 representing the feature quantity of the truck in the image of the transformation after multiple feature quantities are blurred and combined.

[0200] The feature extraction unit 12 outputs the feature vector Fv2 to the representative feature registration unit 16.

[0201] In addition, the feature extraction unit 12 obtains an image signal representing a transformation image of an existing bus from the image signal acquisition unit 11.

[0202] The feature extraction unit 12 extracts the bus features that are reflected in the image of the transformation image from the image signal.

[0203] Specifically, the feature extraction unit 12 provides the image signal to the learning model 1a and obtains a feature vector Fv3 representing the feature quantity of the bus that is mapped to the transformation image. The feature quantity is then blurred and combined with the multiple feature quantities.

[0204] The feature extraction unit 12 outputs the feature vector Fv3 to the representative feature registration unit 16.

[0205] The feature quantity registration unit 16 obtains the feature vector Fv1 from the feature quantity extraction unit 12.

[0206] The representative feature registration unit 16 registers the representative feature by storing the feature vector Fv1 into the representative feature storage unit 14.

[0207] Additionally, the feature quantity registration unit 16 obtains the feature vector Fv2 from the feature quantity extraction unit 12.

[0208] The representative feature registration unit 16 registers the representative feature by storing the feature vector Fv2 into the representative feature storage unit 14.

[0209] Additionally, the feature quantity registration unit 16 obtains the feature vector Fv3 from the feature quantity extraction unit 12.

[0210] The representative feature registration unit 16 registers the representative feature by storing the feature vector Fv3 into the representative feature storage unit 14.

[0211] Next, the operation of the reasoning device 3 during object recognition will be explained.

[0212] Camera 2 photographs the detected object. It is unclear whether the detected object photographed by camera 2 is a passenger car, a truck, or a bus.

[0213] For example, camera 2 outputs an image signal representing a TIR image of the object being detected as an image signal representing an inference object image of the object being detected to inference device 3.

[0214] In this example, the image used for inference is a TIR image. However, the domain and the image used for transformation can be the same for the image used for inference, and it is not limited to a TIR image.

[0215] The image signal acquisition unit 11 acquires an image signal from the camera 2 representing an image of the inference object that is currently being detected.

[0216] The image signal acquisition unit 11 outputs the image signal representing the image of the reasoning object to the feature extraction unit 12.

[0217] The feature extraction unit 12 acquires an image signal representing the image of the reasoning object from the image signal acquisition unit 11.

[0218] The feature extraction unit 12 extracts the feature quantities of the detected object that are reflected in the image of the inference object from the image signal.

[0219] Specifically, the feature extraction unit 12 provides the image signal to the learning model 1a and obtains the feature vector Fv from the learning model 1a, which represents the feature vector of the inference time feature vector formed by combining multiple feature vectors after blurring multiple feature vectors of the detected object in the inference object image.

[0220] The feature extraction unit 12 outputs the feature vector Fv to the object recognition unit 17.

[0221] The object recognition unit 17 obtains the feature vector Fv from the feature extraction unit 12.

[0222] In addition, the object recognition unit 17 obtains feature vectors Fv1 representing passenger car, Fv2 representing truck, and Fv3 representing bus from the representative feature quantity storage unit 14.

[0223] The object recognition unit 17 calculates the similarity Mr1, Mr2, and Mr3 between feature vectors Fv1, Fv2, and Fv3 and feature vector Fv, respectively.

[0224] The object recognition unit 17 determines the highest similarity among Mr1, Mr2, and Mr3, and determines the representative feature quantity corresponding to the highest similarity.

[0225] For example, if the highest similarity is Mr1, then the representative feature corresponding to the highest similarity is the representative feature of passenger cars. If the highest similarity is Mr2, then the representative feature corresponding to the highest similarity is the representative feature of trucks. Furthermore, if the highest similarity is Mr3, then the representative feature corresponding to the highest similarity is the representative feature of buses.

[0226] In the object identification unit 17, if the representative feature quantity with the highest similarity is the representative feature quantity of a passenger car, then the type of the detected object reflected in the inference object image is identified as a passenger car.

[0227] In the object identification unit 17, if the representative feature quantity with the highest similarity is the representative feature quantity of a truck, then the type of the detected object reflected in the inference object image is identified as a truck.

[0228] In the object identification unit 17, if the representative feature quantity with the highest similarity is the representative feature quantity of a bus, then the type of the detected object reflected in the inference object image is identified as a bus.

[0229] The object recognition unit 17 generates display data representing the recognition result of the detected object and outputs the display data to the display device 4.

[0230] The display device 4 displays the identification result of the detected object on a display screen (not shown) according to the display data output from the object recognition unit 17.

[0231] Thus, inspectors can identify the type of object being detected by observing the monitor.

[0232] In Embodiment 2 described above, the inference device 3 is configured as follows: the object identification unit 17 compares representative feature quantities of multiple detection object objects of different types with inference time feature quantities extracted by the feature extraction unit 12, determines a representative feature quantity corresponding to the inference time feature quantity extracted by the feature extraction unit 12 from the representative feature quantities of the multiple detection object objects, and identifies the type of detection object object reflected in the inference object image based on the determination result of the representative feature quantity. Therefore, even if there are more than one difference in the task and domain, the inference device 3 can suppress the deterioration of the inference accuracy regarding the identification of the type of detection object.

[0233] Implementation method 3.

[0234] In embodiment 3, a reasoning device 3 is described, which includes an object identification unit 19 that identifies the area where the target object exists. This is a task known as object detection.

[0235] Figure 11 This is a structural diagram showing the inference device 3 according to embodiment 3. Figure 11 In, with Figure 1 The same symbols represent the same or equivalent parts, so the explanation is omitted.

[0236] Figure 12 This is a hardware structure diagram showing the hardware of the inference device 3 according to Embodiment 3. Figure 12 In, with Figure 2 The same symbols represent the same or equivalent parts, so the explanation is omitted.

[0237] Figure 11 The inference device 3 shown includes an image signal acquisition unit 11, a feature extraction unit 12, a representative feature registration unit 18, a representative feature storage unit 14, and an object recognition unit 19.

[0238] exist Figure 11 In the inference device 3 shown, the model storage unit 1 is located outside the inference device 3. However, this is just an example; for instance, the model storage unit 1 may also be located inside the inference device 3, and the learning model 1a may also be built into the feature extraction unit 12.

[0239] exist Figure 11 The following example illustrates the inference device 3 shown: the area where the detected object exists, identified by the object recognition unit 19, is predicted simultaneously with multiple levels of recognition such as a passenger car, bus, or truck as the spatial location where the detected object exists.

[0240] Representative characteristic registration department 18, for example, through Figure 12 The representative characteristic quantity registration circuit 28 shown is used to implement this.

[0241] The representative feature quantity registration unit 18 registers the representative feature quantities obtained by the feature quantity extraction unit 12.

[0242] That is, the representative feature registration unit 18 obtains a feature vector representing the representative feature quantity formed by combining multiple feature quantities of the detected object in the image being transformed after blurring each feature quantity. The feature vector containing the objectness of the high-dimensional feature is stored in the representative feature storage unit 14. The stored objectness of the detected object can be registered as a Tensor as is, or only the objectness of the detected object can be extracted and compressed for registration.

[0243] Object recognition unit 19, for example, through Figure 12 The object recognition circuit 29 shown is used to implement this.

[0244] The object identification unit 19 obtains from the feature extraction unit 12 a feature vector representing the inference time feature quantity formed by combining multiple feature quantities of the detected object that are reflected in the inference object image after being blurred. It then obtains the feature vector representing the representative feature quantity from the representative feature quantity storage unit 14.

[0245] The object identification unit 19 identifies the detected object reflected in the image of the reasoning object based on representative feature quantities and inference time feature quantities.

[0246] Specifically, the object identification unit 19 obtains representative feature quantities containing the existence range (objectness) of the detected object from the representative feature quantity storage unit 14, which stores feature vectors containing the objectness of the high-dimensional features. It compares these representative feature quantities with the inference-time feature quantities extracted from the feature quantity extraction unit 12, and determines the representative feature quantity corresponding to the inference-time feature quantity extracted from the feature quantity extraction unit 12 among multiple representative feature quantities of the detected object. The level with the most similar representative feature quantity becomes the level to which the detected object belongs. Furthermore, since the inference-time feature quantity includes the existence range (objectness) of the detected object contained in the image input during inference, by transforming the description method of the high-dimensional features of the inference-time feature quantity from Tensors or the like to two-dimensional space, the spatial location of the object can be represented.

[0247] The object recognition unit 19 generates display data representing the recognition result of the detected object and outputs the display data to the display device 4. Here, when it is represented as a rectangle in two-dimensional space, it becomes an object detection task; when it is represented as a region in two-dimensional space, it becomes a semantic segmentation task.

[0248] exist Figure 11 In this context, it is envisioned that the image signal acquisition unit 11, feature extraction unit 12, representative feature registration unit 18, representative feature storage unit 14, and object recognition unit 19, which are components of the reasoning device 3, are respectively processed by, for example... Figure 12 The dedicated hardware shown is used to implement this. That is, the inference device 3 is envisioned to be implemented through an image signal acquisition circuit 21, a feature extraction circuit 22, a representative feature registration circuit 28, a representative feature storage circuit 24, and an object recognition circuit 29.

[0249] The image signal acquisition circuit 21, the feature extraction circuit 22, the representative feature registration circuit 28, and the object recognition circuit 29 are each, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0250] The components of the inference device 3 are not limited to being implemented through dedicated hardware; the inference device 3 can also be implemented through software, firmware, or a combination of software and firmware.

[0251] When the inference device 3 is implemented through software or firmware, etc., Figure 3The memory 31 shown is configured with a representative feature storage unit 14. The programs for enabling the computer to execute each processing step in the image signal acquisition unit 11, the feature extraction unit 12, the representative feature registration unit 18, and the object recognition unit 19 are stored in... Figure 3 The memory 31 shown. Then, Figure 3 The processor 32 shown executes the program stored in memory 31.

[0252] In addition, Figure 12 The diagram illustrates an example of implementing the various components of the inference device 3 using dedicated hardware. Figure 3 The diagram shows an example of implementing the inference device 3 using software or firmware. However, this is just one example; some of the components of the inference device 3 can be implemented using dedicated hardware, while the remaining components can be implemented using software or firmware.

[0253] Next, the operation of the inference device 3 during domain transformation will be explained.

[0254] Camera 2 takes a picture of the object being detected.

[0255] Camera 2 outputs the image signal representing the TIR image of the existing object to the inference device 3 as an image signal representing the transformed image of the existing object.

[0256] The image signal acquisition unit 11 acquires an image signal from the camera 2, which represents a transformed image of the existing object being detected.

[0257] The image signal acquisition unit 11 outputs image signals representing various transformation images of the level of the object to be identified to the feature extraction unit 12.

[0258] The feature extraction unit 12 acquires the image signal representing the image for each transformation from the image signal acquisition unit 11.

[0259] The feature extraction unit 12 extracts the feature quantities of the detected object that are reflected in each transformed image from each image signal.

[0260] Specifically, the feature extraction unit 12 provides the image signal to the learning model 1a and obtains a feature vector from the learning model 1a that represents the feature quantity formed by combining multiple feature quantities after blurring multiple feature quantities of the detected object.

[0261] The feature extraction unit 12 outputs the feature vector to the representative feature registration unit 18.

[0262] The characteristic quantity registration unit 18 obtains the characteristic vector from the characteristic quantity extraction unit 12.

[0263] The representative feature registration unit 18 registers representative features by storing feature vectors in the representative feature storage unit 14.

[0264] Next, the operation of the reasoning device 3 during object recognition will be explained.

[0265] Camera 2 takes a picture of the target object. The location of the target object captured by camera 2 is unknown.

[0266] Camera 2 outputs the image signal representing the TIR image of the existing detected object to the inference device 3 as the image signal representing the inference object image of the existing detected object.

[0267] The image signal acquisition unit 11 acquires an image signal from the camera 2 representing an image of the inference object that is currently being detected.

[0268] The image signal acquisition unit 11 outputs the image signal representing the image of the reasoning object to the feature extraction unit 12.

[0269] The feature extraction unit 12 acquires an image signal representing the image of the reasoning object from the image signal acquisition unit 11.

[0270] The feature extraction unit 12 extracts the feature quantities of the detected object that are reflected in the image of the inference object from the image signal.

[0271] Specifically, the feature extraction unit 12 provides the image signal to the learning model 1a and obtains from the learning model 1a a feature vector representing the inference time feature quantity formed by combining multiple feature quantities of the detected object in the image of the inference object after blurring them.

[0272] The feature extraction unit 12 outputs the feature vector to the object recognition unit 19.

[0273] The object recognition unit 19 obtains the feature vector from the feature extraction unit 12.

[0274] In addition, the object recognition unit 19 obtains multiple feature vectors representing representative feature quantities from the representative feature quantity storage unit 14.

[0275] The object identification unit 19 calculates the similarity between multiple feature vectors representing representative feature quantities and feature vectors obtained from the feature quantity extraction unit 12.

[0276] The object identification unit 19 determines the highest similarity among the registered similarity values ​​of representative feature quantities corresponding to the number of objects to be identified and the feature quantities used in reasoning, and determines the representative feature quantity corresponding to the highest similarity value. By performing this determination, the object identification unit 19 can identify which level the object belongs to.

[0277] In the object recognition unit 19, the level with the most similar representative feature quantity becomes the level to which the detected object belongs. For example, when both the representative feature quantity and the inference feature quantity use the TIR image as input, level recognition beyond the learning domain can be performed.

[0278] Furthermore, the inference-time feature quantity includes the objectness of the detected object contained in the input image during inference. Therefore, by transforming the description method of the high-dimensional features of the inference-time feature quantity from tensors to two-dimensional space, it is possible to represent the spatial location of the object. Thus, when the learning task is image recognition, it is possible to perform recognition beyond the task.

[0279] The object recognition unit 19 generates display data representing the recognition result of the detected object and outputs the display data to the display device 4.

[0280] The display device 4 displays the identification result of the detected object on a display screen (not shown) according to the display data output from the object recognition unit 19.

[0281] Therefore, inspectors can confirm the area where the detected object exists by observing the monitor.

[0282] In the above-described embodiment 3, the inference device 3 is configured as follows: the object identification unit 19 compares representative feature quantities of multiple detection object objects whose regions are different from each other with the inference-time feature quantities obtained by the feature extraction unit 12. Among the representative feature quantities of the multiple detection object objects, it determines the representative feature quantity corresponding to the inference-time feature quantity obtained by the feature extraction unit 12, and identifies the region where the detection object object exists as the detection object object reflected in the inference object image based on the determination result of the representative feature quantity. Therefore, even if there are more than one difference in the task and domain, the inference device 3 can suppress the deterioration of the inference accuracy regarding the identification of the region where the detection object object exists.

[0283] Implementation method 4.

[0284] In Embodiment 4, a reasoning device 3 is described, which includes an object identification unit 72 that identifies the type of the object to be detected and the area where it exists.

[0285] Figure 13 This is a structural diagram showing the inference device 3 according to embodiment 4. Figure 13 In, with Figure 1 The same symbols represent the same or equivalent parts, so the explanation is omitted.

[0286] Figure 14This is a hardware structure diagram showing the hardware of the inference device 3 according to Embodiment 4. Figure 14 In, with Figure 2 The same symbols represent the same or equivalent parts, so the explanation is omitted.

[0287] Figure 13 The inference device 3 shown includes an image signal acquisition unit 11, a feature extraction unit 12, a representative feature registration unit 71, a representative feature storage unit 14, and an object recognition unit 72.

[0288] exist Figure 13 In the inference device 3 shown, the model storage unit 1 is located outside the inference device 3. However, this is just an example; for instance, the model storage unit 1 may also be located inside the inference device 3, and the learning model 1a may also be built into the feature extraction unit 12.

[0289] exist Figure 13 In the inference device 3 shown, the area where the detected object exists, as identified by the object recognition unit 72, is the area of ​​the level where the detected object exists in multiple levels.

[0290] In addition, Figure 13 The following example illustrates the reasoning device 3 shown: The object being detected is identified by the object identification unit 72 as either a passenger car, a truck, or a bus. In this case, the object being detected is any one of a passenger car, a truck, or a bus.

[0291] Even when the type of the object being detected and the area where it exists can be identified separately, the learning model 1a is provided with learning data containing image signals representing the images used for learning.

[0292] Representative characteristic registration department 71, for example, through Figure 14 The representative characteristic quantity registration circuit 81 shown is used to implement this.

[0293] The representative characteristic quantity registration unit 71 registers the representative characteristic quantities obtained by the characteristic quantity extraction unit 12.

[0294] That is, the representative feature registration unit 71 obtains the feature vector of the representative feature formed by combining multiple feature quantities of a passenger vehicle existing in any region after blurring them, and stores the feature vector in the representative feature storage unit 14.

[0295] In addition, the representative feature registration unit 71 obtains a feature vector representing a truck existing in any region, which is formed by combining multiple feature quantities after blurring them, and stores the feature vector in the representative feature storage unit 14.

[0296] In addition, the representative feature registration unit 71 obtains a feature vector representing a bus existing in any region, which is formed by combining multiple feature quantities after blurring them, and stores the feature vector in the representative feature storage unit 14.

[0297] Object recognition unit 72, for example, through Figure 14 The object recognition circuit 82 shown is used to implement this.

[0298] The object identification unit 72 obtains from the feature extraction unit 12 a feature vector representing the inference time feature quantity formed by combining multiple feature quantities of the detected object that are reflected in the inference object image after being blurred. It then obtains the feature vector representing the representative feature quantity from the representative feature quantity storage unit 14.

[0299] The object identification unit 72 identifies the type of the detected object and the area where it exists in the image of the inference object based on the representative feature quantity and the inference time feature quantity.

[0300] Specifically, the object identification unit 72 obtains representative feature quantities including both the object's existence range and object type from the representative feature quantity storage unit 14, which stores feature vectors containing both high-dimensional features. It then compares these representative feature quantities with inference-time features extracted from the feature extraction unit 12, and determines the representative feature quantity corresponding to the inference-time features extracted from the feature extraction unit 12 among multiple representative feature quantities of the detected object. The level with the most similar representative feature quantity becomes the level to which the detected object belongs. Furthermore, since the inference-time features include the object's existence range contained within the image input during inference, by transforming the description method of the high-dimensional features of the inference-time features from tensors to two-dimensional space, it is possible to represent the spatial location of objects.

[0301] The object recognition unit 72 generates display data representing the recognition result of the detected object and outputs the display data to the display device 4. Here, when it is represented as a rectangle in two-dimensional space, it is called the object detection task; when it is represented as a region in two-dimensional space, it is called the sematic segmentation task.

[0302] exist Figure 13 In this context, it is envisioned that the image signal acquisition unit 11, feature extraction unit 12, representative feature registration unit 71, representative feature storage unit 14, and object recognition unit 72, which are components of the reasoning device 3, each pass through a system such as... Figure 14The dedicated hardware shown is used to implement this. That is, the inference device 3 is envisioned to be implemented through an image signal acquisition circuit 21, a feature extraction circuit 22, a representative feature registration circuit 81, a representative feature storage circuit 24, and an object recognition circuit 82.

[0303] The image signal acquisition circuit 21, the feature extraction circuit 22, the representative feature registration circuit 81, and the object recognition circuit 82 are each, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0304] The components of the inference device 3 are not limited to being implemented through dedicated hardware; the inference device 3 can also be implemented through software, firmware, or a combination of software and firmware.

[0305] When the inference device 3 is implemented through software or firmware, etc., Figure 3 The memory 31 shown is configured with a representative feature storage unit 14. Programs for enabling the computer to execute various processing steps in the image signal acquisition unit 11, the feature extraction unit 12, the representative feature registration unit 71, and the object recognition unit 72 are stored in... Figure 3 The memory 31 shown. Then, Figure 3 The processor 32 shown executes the program stored in memory 31.

[0306] In addition, Figure 14 The diagram illustrates an example of implementing the various components of the inference device 3 using dedicated hardware. Figure 3 The diagram shows an example of implementing the inference device 3 using software or firmware. However, this is just one example; some of the components of the inference device 3 can be implemented using dedicated hardware, while the remaining components can be implemented using software or firmware.

[0307] Next, the operation of the inference device 3 during domain transformation will be explained.

[0308] Camera 2 photographs the target object. The target object photographed by camera 2 is any one of a passenger car, truck, or bus. However, the reasoning device 3 can also classify the target object into, for example, 1000 categories. Therefore, classifying the target object into three categories—passenger car, truck, or bus—is just one example.

[0309] The detected object, captured by camera 2, exists in any region.

[0310] For example, camera 2 outputs an image signal representing a TIR image of a detected object as an image signal representing a transformed image of a detected object existing in any region to inference device 3.

[0311] The image signal acquisition unit 11 acquires an image signal from the camera 2 representing a transformation image of a detected object existing in an arbitrary area, and outputs the image signal representing the transformation image to the feature extraction unit 12.

[0312] That is, the image signal acquisition unit 11 acquires an image signal from the camera 2 representing a transformation image of a passenger vehicle existing in any area, and outputs the image signal representing the transformation image to the feature extraction unit 12.

[0313] The image signal acquisition unit 11 acquires an image signal from the camera 2 representing a transformation image of a truck existing in an arbitrary area, and outputs the image signal representing the transformation image to the feature extraction unit 12.

[0314] In addition, the image signal acquisition unit 11 acquires an image signal from the camera 2 representing a transformation image of a bus existing in any area, and outputs the image signal representing the transformation image to the feature extraction unit 12.

[0315] The feature extraction unit 12 acquires from the image signal acquisition unit 11 an image signal representing a transformation image of a passenger vehicle existing in any region.

[0316] The feature extraction unit 12 extracts the features of the passenger vehicle that are reflected in each transformed image from each image signal.

[0317] Specifically, the feature extraction unit 12 provides the image signal to the learning model 1a and obtains a feature vector representing the multiple features of a passenger vehicle existing in any region from the learning model 1a. These features are blurred and then combined to form a feature vector representing the multiple features.

[0318] The feature extraction unit 12 outputs the feature vector to the representative feature registration unit 71.

[0319] In addition, the feature extraction unit 12 obtains from the image signal acquisition unit 11 an image signal representing a transformed image of a truck existing in any region.

[0320] The feature extraction unit 12 extracts the features of the truck that are reflected in each transformed image from each image signal.

[0321] Specifically, the feature extraction unit 12 provides the image signal to the learning model 1a and obtains a feature vector representing the multiple features of a truck existing in any region from the learning model 1a. These features are blurred and then combined to form a feature vector representing the multiple features.

[0322] The feature extraction unit 12 outputs the feature vector to the representative feature registration unit 71.

[0323] In addition, the feature extraction unit 12 obtains from the image signal acquisition unit 11 an image signal representing a transformation image of a bus existing in any region.

[0324] The feature extraction unit 12 extracts the bus features reflected in each transformed image from each image signal.

[0325] Specifically, the feature extraction unit 12 provides the image signal to the learning model 1a and obtains a feature vector representing the feature quantity from the learning model 1a, which is formed by combining multiple feature quantities of a bus existing in any region after blurring them.

[0326] The feature extraction unit 12 outputs the feature vector to the representative feature registration unit 71.

[0327] The characteristic quantity registration unit 71 obtains each characteristic vector from the characteristic quantity extraction unit 12.

[0328] The representative feature registration unit 71 registers representative features by storing each feature vector into the representative feature storage unit 14.

[0329] Next, the operation of the reasoning device 3 during object recognition will be explained.

[0330] Camera 2 photographs the detected object. It is unclear whether the detected object photographed by camera 2 is a passenger car, truck, or bus. Furthermore, the area where the detected object exists is unknown.

[0331] Camera 2 outputs the image signal representing the TIR image of the existing detected object to the inference device 3 as the image signal representing the inference object image of the existing detected object.

[0332] The image signal acquisition unit 11 acquires an image signal from the camera 2 representing an image of the inference object that is currently being detected.

[0333] The image signal acquisition unit 11 outputs the image signal representing the image of the reasoning object to the feature extraction unit 12.

[0334] The feature extraction unit 12 acquires an image signal representing the image of the reasoning object from the image signal acquisition unit 11.

[0335] The feature extraction unit 12 extracts the feature quantities of the detected object that are reflected in the image of the inference object from the image signal.

[0336] Specifically, the feature extraction unit 12 provides the image signal to the learning model 1a and obtains from the learning model 1a a feature vector representing the inference time feature quantity formed by combining multiple feature quantities of the detected object in the image of the inference object after blurring them.

[0337] The feature extraction unit 12 outputs the feature vector to the object recognition unit 72.

[0338] The object recognition unit 72 obtains the feature vector from the feature extraction unit 12.

[0339] In addition, the object recognition unit 72 obtains multiple feature vectors representing representative feature quantities from the representative feature quantity storage unit 14.

[0340] The object identification unit 72 calculates the similarity between multiple feature vectors representing representative feature quantities and feature vectors obtained from the feature quantity extraction unit 12.

[0341] The object identification unit 72 determines the highest similarity among the registered representative feature quantities and reasoning feature quantities corresponding to the number of objects to be identified, and determines the representative feature quantity corresponding to the highest similarity. By performing this determination, the object identification unit 19 can determine which level it belongs to.

[0342] In the object recognition unit 72, the level with the most similar representative feature quantity becomes the level to which the detected object belongs. For example, when both the representative feature quantity and the inference feature quantity use the TIR image as input, level recognition beyond the learning domain can be performed.

[0343] Furthermore, the inference-time feature quantity includes the objectness of the detected object contained in the input image during inference. Therefore, by transforming the description method of the high-dimensional features of the inference-time feature quantity from tensors to two-dimensional space, it is possible to represent the location of the object in space. Thus, when the learning task is image recognition, it is possible to perform recognition beyond the task.

[0344] In the object identification unit 72, if the representative feature quantity with the highest similarity is, for example, the representative feature quantity of a passenger car, then the type of the detected object reflected in the inference object image is identified as a passenger car, and the area where the detected object exists is identified.

[0345] In the object identification unit 72, if the representative feature with the highest similarity is, for example, the representative feature of a truck, then the type of the detected object reflected in the inference object image is identified as a truck, and the area where the detected object exists is identified.

[0346] In the object identification unit 72, if the representative feature with the highest similarity is, for example, the representative feature of a bus, then the type of the detected object reflected in the inference object image is identified as a bus, and the area where the detected object exists is identified.

[0347] The object recognition unit 72 generates display data representing the recognition result of the detected object and outputs the display data to the display device 4.

[0348] The display device 4 displays the identification result of the detected object on a display screen (not shown) according to the display data output from the object recognition unit 72.

[0349] Thus, inspectors can identify the type of object being detected and the area where it exists by observing the monitor.

[0350] In the above-described embodiment 4, the inference device 3 is configured as follows: the object identification unit 72 compares representative feature quantities of multiple target objects, each with different types and regions of existence, with inference-time feature quantities obtained by the feature extraction unit 12. It then determines a representative feature quantity from among the representative feature quantities of the multiple target objects that corresponds to the inference-time feature quantity obtained by the feature extraction unit 12. Based on the determination result of the representative feature quantity, it identifies the type and region of the target object as the target object reflected in the inference object image. Therefore, even in cases where there are more than one difference in the task and domain, the inference device 3 can suppress the deterioration of inference accuracy regarding the identification of the type and region of the target object.

[0351] Furthermore, this disclosure allows for free combination of various embodiments, modification of any constituent elements of each embodiment, or omission of any constituent elements in each embodiment.

[0352] Industrial availability

[0353] This disclosure applies to inference devices, inference methods, and inference procedures.

[0354] Explanation of symbols

[0355] 1: Model storage unit; 1a: Learning model; 2: Camera; 3: Inference device; 4: Display device; 5: Learning data storage unit; 6: Learning device; 11: Image signal acquisition unit; 12: Feature extraction unit; 13: Representative feature registration unit; 14: Representative feature storage unit; 15: Object recognition unit; 16, 18: Representative feature registration units; 17, 19: Object recognition units; 21: Image signal acquisition circuit; 22: Feature extraction circuit; 23: Representative feature registration circuit 24: Represents feature quantity storage circuit; 25: Object recognition circuit; 26, 28: Represents feature quantity registration circuit; 27, 29: Object recognition circuit; 31: Memory; 32: Processor; 41: Learning data acquisition unit; 42: Learning processing unit; 51: Learning data acquisition circuit; 52: Learning processing circuit; 61: Memory; 62: Processor; 71: Represents feature quantity registration unit; 72: Object recognition unit; 81: Represents feature quantity registration circuit; 82: Object recognition circuit.

Claims

1. A reasoning device, comprising: The image signal acquisition unit acquires an image signal representing an inference object image that reflects an existing object to be detected, in cases where the domain of the image is different from the learning image or the recognition task is different from the previously learned task. The feature extraction unit provides the image signal acquired by the image signal acquisition unit to the learning model after learning the learning image, and obtains the inference-time feature values ​​from the learning model, wherein... The feature quantity during inference is a combination of multiple feature quantities of the detected object that are reflected in the image of the inference object after the multiple feature quantities are blurred respectively. as well as The object identification unit identifies the object to be detected in the inference object image based on the representative feature quantity of the registered feature quantity of the object to be detected as a detected object reflected in the transformation image and the inference time feature quantity obtained by the feature quantity extraction unit, wherein the transformation image is an object whose domain and identification task are the same as those of the inference object image.

2. The reasoning device according to claim 1, characterized in that, The image signal acquisition unit acquires an image signal representing the image used for transformation. The feature extraction unit provides the image signal representing the transformed image to the learning model, and obtains representative feature quantities from the learning model, which are feature quantities formed by combining multiple feature quantities of the detected object reflected in the transformed image after they have been blurred. The inference device includes a representative feature registration unit that registers representative features obtained by the feature extraction unit.

3. The reasoning device according to claim 1, characterized in that, The object identification unit calculates the similarity between the feature vector representing the representative feature quantity and the feature vector representing the inference feature quantity obtained by the feature quantity extraction unit, and identifies the detected object reflected in the inference object image based on the similarity.

4. The reasoning device according to claim 1, characterized in that, The object identification unit compares representative feature quantities of multiple detection object objects of different types with inference time feature quantities obtained by the feature quantity extraction unit, determines a representative feature quantity corresponding to the inference time feature quantity obtained by the feature quantity extraction unit from the representative feature quantities of the multiple detection object objects, and identifies the type of detection object object based on the determination result of the representative feature quantity as the detection object object reflected in the inference object image.

5. The reasoning device according to claim 1, characterized in that, The object identification unit compares representative feature quantities of multiple detection object objects with mutually different regions with inference time feature quantities obtained by the feature quantity extraction unit. It determines a representative feature quantity that corresponds to the inference time feature quantity obtained by the feature quantity extraction unit from the representative feature quantities of the multiple detection object objects. Based on the determination result of the representative feature quantity, it identifies the region where the detection object object exists as the detection object object reflected in the inference object image.

6. The reasoning device according to claim 1, characterized in that, The object identification unit compares representative feature quantities of multiple detection object objects, each with different types and existing regions, with inference time feature quantities obtained by the feature quantity extraction unit. It determines a representative feature quantity from the representative feature quantities of the multiple detection object objects that corresponds to the inference time feature quantity obtained by the feature quantity extraction unit. Based on the determination result of the representative feature quantity, the detection object object reflected in the inference object image is identified, and the type and existing region of the detection object object are identified respectively.

7. The reasoning device according to claim 1, characterized in that, The learning model has multi-layer neural networks (DNNs). The feature extraction unit provides the image signal obtained by the image signal acquisition unit to the DNNs, and obtains the inference time feature from the DNNs.

8. A reasoning method, wherein, The image signal acquisition unit acquires an image signal representing an inference object image that reflects an existing object to be detected, in cases where the image domain differs from the learning image or the recognition task differs from the previously learned task. The feature extraction unit provides the image signal acquired by the image signal acquisition unit to the learning model of the learning image after learning, and obtains inference-time features from the learning model. These inference-time features are formed by combining multiple features of the detected object in the inference object image after blurring them. The object identification unit identifies the object being detected in the inference object image based on the representative feature quantity of the registered feature quantity of the object being detected as a transformed image and the inference time feature quantity obtained by the feature quantity extraction unit, wherein the transformed image is an object whose domain and identification task are the same as those of the inference object image.

9. A recording medium containing a reasoning program for causing a computer to execute: In the image signal acquisition process, the image signal acquisition unit acquires an image signal representing an inference object image that reflects an existing object to be detected, in cases where the domain of the image is different from the learning image and the recognition task is different from the previously learned task. In the feature acquisition process, the feature extraction unit provides the image signal acquired by the image signal acquisition unit to the learning model that has completed learning from the learning image, and obtains the inference-time feature values ​​from the learning model, wherein... The feature quantity during inference is a combination of multiple feature quantities of the detected object that are reflected in the image of the inference object after the multiple feature quantities are blurred respectively. as well as In the object identification process, the object identification unit identifies the object being detected as reflected in the inference object image based on the representative feature quantity of the registered feature quantity of the object being detected as reflected in the transformation image and the inference time feature quantity obtained by the feature quantity extraction unit. The transformation image is an object whose domain and identification task are the same as those of the inference object image.

Images