IMAGE RECOGNITION SYSTEM, IMAGE RECOGNITION DEVICE, SENTENCE CHARACTERISTIC EXTRACTION DEVICE, PROGRAM, AND IMAGE RECOGNITION METHOD

Abstract

The image recognition system (10) includes a text feature extraction device (2000) and an image recognition device (1000). The text feature extraction device (2000) includes a text acquisition unit (2100) that acquires a phrase in a natural language input by a user, a text feature extraction unit (2200) that extracts text features from the phrase in the natural language, and a text feature transmission unit (2300) that transmits the text features. The image recognition device (1000) includes a text feature reception unit (1300) that receives text features from the text feature extraction device (2000), an image acquisition unit (1100) that acquires image data, an image analysis unit (1200) that extracts image features that are feature amounts of a candidate area and an image in the candidate area from the image data, a feature comparison unit (1400) that compares the text features with the image features corresponding to the candidate area and outputs the comparison results, and a detection result generation unit (1500) that generates a detection result of the detection target based on the comparison results.

Description

Technical Field

[0001] The present disclosure relates to an image recognition system, an image recognition device, a text feature extraction device, a program, and an image recognition method for detecting a detection target specified by a natural language phrase from image data.

Background Art

[0002] In the image recognition technology for detecting a detection target from image data, with the development of artificial intelligence, the accuracy has been significantly improved, and the realization of advanced security systems and the like is expected. The learned model used in artificial intelligence has a very large amount of computation compared to conventional image recognition technologies, and it has been difficult to implement on devices with limited computing power such as surveillance cameras.

[0003] In response to the above problems, Non-Patent Document 1 discloses a technique called Grounding DINO (self-DIstillation with NO labels: label-free self-distillation). Grounding DINO is a technique that can handle the detection of a wider range of object categories by pre-training a DNN model with 10 to 100 times as much massive image data as conventional deep neural networks (DNNs) and phrases represented in paired natural language. Also, in Grounding DINO, at the time of inference, by specifying the object category to be detected with a natural language phrase, it is possible to change or add the object category to be detected without re-training the DNN model.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

[0005] However, even with the technology disclosed in Non-Patent Document 1, the amount of calculation required for the inference process to detect a detection target specified by a phrase input in natural language from image data using a trained model is still very large, and there is a problem that it is difficult to implement when the computing power of the image recognition device is limited.

[0006] The present disclosure has been made in consideration of the above, and aims to provide an image recognition system that is capable of implementing an inference process that uses a trained model to detect a detection target specified by a phrase input in natural language from image data, even when the computational capability of the image recognition device is limited. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, the image recognition system disclosed herein is an image recognition system that detects a detection target from image data of a recognition target, and includes a sentence feature extraction device and an image recognition device. The sentence feature extraction device includes a sentence acquisition unit that acquires a phrase in a natural language for specifying the detection target, a sentence feature extraction unit that extracts sentence features, which are features comparable to image features, from the phrase in the natural language, and a sentence feature transmission unit that transmits the sentence features. The image recognition device includes a sentence feature reception unit that receives the sentence features from the sentence feature extraction device, and a sentence feature storage unit that stores the sentence features received by the sentence feature receiving unit;an image acquisition unit that acquires image data; and an image analysis unit that extracts, from the image data, a candidate area that is a candidate for an area in which a detection target exists within the image, and an image feature that is a feature amount of the image within the candidate area; The sentence feature stored in the memory The image recognition system includes a feature comparison unit that compares the sentence features with the image features corresponding to the candidate region and outputs a comparison result, and a detection result generation unit that generates a detection result for the detection target based on the comparison result. The image recognition system is characterized in that the processing of the sentence feature extraction unit is executed by a sentence feature extraction device that is a device separate from the image recognition device. Effect of the Invention

[0008] According to the present disclosure, it is possible to obtain an image recognition system that can implement an inference process that detects a detection target specified by a phrase input in natural language from image data using a trained model, even if the computational capability of the image recognition device is limited. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a configuration diagram of an image recognition system according to a first embodiment. [Diagram 2] A diagram showing an example of the configuration of a DNN model used by the image recognition system shown in Figure 1. [Diagram 3] 1 is a flowchart illustrating an example of the operation of the image recognition system shown in FIG. [Figure 4] FIG. 1 is a configuration diagram of an image recognition system according to a second embodiment. [Diagram 5] 5 is a flowchart illustrating an example of the operation of the image recognition system shown in FIG. 4. [Figure 6] FIG. 1 is a diagram showing dedicated hardware for implementing the functions of the image recognition system according to the first and second embodiments. [Figure 7] FIG. 1 is a diagram showing a configuration of a control circuit for implementing the functions of the image recognition system according to the first and second embodiments; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An image recognition system, an image recognition device, a text feature extraction device, a program, and an image recognition method according to embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0011] Embodiment 1 FIG. 1 is a configuration diagram of an image recognition system 10 according to a first embodiment. The image recognition system 10 includes an image recognition device 1000, a text feature extraction device 2000, and a result display unit 3000. The image recognition system 10 has a function of detecting a detection target specified by a phrase input in natural language from image data by performing an inference process using a trained model. The trained model may be generated using a technology such as Grounding DINO. In the image recognition system 10, the inference process is shared and executed by the image recognition device 1000 and the text feature extraction device 2000.

[0012] Specifically, in the following embodiment, the process of extracting text features, which is a high-load process among the inference processes, in other words a process with a large amount of calculation, is executed by the text feature extraction device 2000, and the other processes are executed by the image recognition device 1000. Here, the allocation of processes to be shared between the image recognition device 1000 and the text feature extraction device 2000 is determined in advance by a human being, but other methods of determination may also be used.

[0013] As an example of the present disclosure, even if the text feature extraction device 2000 is composed of multiple devices or has multiple arithmetic units, the multiple arithmetic units included therein are treated as one unit. Similarly, even if the image recognition device 1000 is composed of multiple devices or has multiple arithmetic units, the multiple arithmetic units included therein are treated as one unit. In other words, regardless of the configuration of each arithmetic unit of the image recognition device 1000 and the text feature extraction device 2000, they are treated as a unit that executes the above-mentioned inference process. As a result, even if the image recognition device 1000 is a single imaging device having one arithmetic unit and the text feature extraction device 2000 is one or more server devices having multiple arithmetic units, for example, the image recognition device 1000 and the text feature extraction device 2000 can be compared in terms of their arithmetic processing capabilities as one unit. In the example of the present disclosure, the text feature extraction device 2000 is assumed to have a higher arithmetic processing capability than the image recognition device 1000.

[0014] The image recognition device 1000 includes an image acquisition unit 1100, an image analysis unit 1200, a text feature receiving unit 1300, a feature comparison unit 1400, a detection result generation unit 1500, and a detection result transmission unit 1600. The image recognition device 1000 recognizes whether a specified object is captured in an image captured by the image acquisition unit 1100, and if so, recognizes the area of ​​the object in the image, and transmits the recognition result to a result display unit 3000. The image recognition device 1000 is, for example, a surveillance camera with a built-in microcomputer capable of executing a processing device or processing software for image recognition, a digital camera, or a computer connected to a surveillance camera. The image recognition device 1000 is not limited to the above example. In addition, for simplicity in this embodiment, the image recognition system 10 has one image recognition device 1000, but in practice, the image recognition system 10 may have multiple image recognition devices 1000.

[0015] The text feature extraction device 2000 includes a text acquisition unit 2100, a text feature extraction unit 2200, and a text feature transmission unit 2300. The text feature extraction device 2000 is a device different from the image recognition device 1000. When a user of the image recognition system 10 specifies an object category to be detected by the image recognition device 1000 using a phrase input in natural language, the text feature extraction device 2000 extracts text features from the input natural language phrase and transmits the extracted text features to the image recognition device 1000. Here, the text features are feature amounts comparable to the feature amounts of images.

[0016] The result display unit 3000 is a device for displaying the result of the object detection performed by the image recognition device 1000 from an image. The result display unit 3000 only needs to have a display function, and there is no particular limitation on the specific configuration.

[0017] Hereinafter, the details of each part of the image recognition system 10 will be described. The image acquisition unit 1100 acquires image data to be processed and outputs the acquired image data to the image analysis unit 1200. The image acquisition unit 1100 is, for example, an image sensor such as a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor), or an imaging device such as a digital camera or a surveillance camera connected to a computer.

[0018] The image analysis unit 1200 extracts a plurality of candidate regions, which are regions considered to contain an object in the image data acquired by the image acquisition unit 1100, and extracts image features, which are feature amounts included in each of the extracted plurality of candidate regions. The image analysis unit 1200 outputs the extracted plurality of candidate regions and the image features corresponding to each candidate region to the feature comparison unit 1400. The image analysis unit 1200 uses a first partial model, which is a part of a pre-trained DNN model and whose output depends only on the input of image data and not on the input of natural language, to extract candidate regions and image features.

[0019] The image analysis unit 1200 can extract the same number of image features as the number of candidate regions. For example, when the image analysis unit 1200 extracts N candidate regions, it can extract N image features.

[0020] Here, the expression format of the information indicating the candidate area is not particularly limited as long as it can express the range of the candidate area in the image. For example, the shape of the candidate area can be a geometric shape, and the position of the geometric shape can be specified to express the candidate area. The geometric shape may be, for example, a polygon such as a rectangle, or a circle. When the shape of the candidate area is a polygon, the position of the candidate area can be expressed using the coordinates of the vertices of the polygon, the length of the sides, the coordinates of the center of gravity, and the like. For example, when the shape of the candidate area is a rectangle, the candidate area may be expressed by the coordinates of the four vertices of the rectangle, the candidate area may be expressed by a combination of the coordinates of two vertices located diagonally out of the four vertices of the rectangle, the candidate area may be expressed by the coordinates of one of the four vertices of the rectangle and the numerical value of the length of each side of the rectangle, or the candidate area may be expressed by the coordinates of the center of gravity of the rectangle and the numerical value of the length of each side of the rectangle. The numerical values ​​of the vertex coordinates, the center of gravity coordinates, and the length of each side may be a combination of the image size and the actual coordinates, or may be relative values ​​to the image size.

[0021] The sentence feature receiving unit 1300 receives sentence features transmitted from the sentence feature extraction device 2000, and outputs the received sentence features to the feature comparing unit 1400. There are as many sentence features as there are specified object categories, and when there are M specified object categories, M sentence features are also received. When the sentence feature is a Q-dimensional vector, the sentence feature receiving unit 1300 receives data of M Q-dimensional vectors.

[0022] The feature comparison unit 1400 performs a similarity calculation by comparing N image features corresponding to each candidate region output by the image analysis unit 1200 with M text features output by the text feature receiving unit 1300, and calculates which of the M text features each image feature is most similar to. The feature comparison unit 1400 outputs the similarity calculation result to the detection result generation unit 1500 as a comparison result. The feature comparison unit 1400 is a part of a pre-trained DNN model, and receives as input the output of a first partial model executed by the image analysis unit 1200 and the output of a second partial model executed by the text feature extraction unit 2200 described later, and performs the above-mentioned processing by executing a third partial model that outputs the similarity as a comparison result.

[0023] The detection result generating unit 1500 identifies an object category corresponding to the text feature to which each candidate region is most similar based on the comparison result output by the feature comparing unit 1400, and outputs the identified object category as a detection result to the detection result transmitting unit 1600. Furthermore, if there is an overlap between the object category and the candidate region, the redundant result is removed from the detection result.

[0024] The detection result transmission unit 1600 outputs the detection result output by the detection result generation unit 1500 to the result display unit 3000 .

[0025] The result display unit 3000 displays the detection result to the user of the image recognition system 10 in a desired manner based on the detection result received from the detection result transmission unit 1600. There are no particular limitations on the method of displaying the detection result.

[0026] The text acquisition unit 2100 accepts input of a phrase in a natural language of an object category that the user of the image recognition system 10 wants the image recognition device 1000 to detect, and outputs the accepted phrase to the text feature extraction unit 2200. The text acquisition unit 2100 can accept input of a phrase in a natural language by, for example, character input using an input means such as a keyboard or a touch sensor, voice recognition using a microphone, or reading data of a phrase list created in advance from a storage device. In addition, there may be multiple phrases to be input, and each phrase corresponds to an object category.

[0027] The sentence feature extraction unit 2200 extracts sentence features from the natural language phrases acquired by the sentence acquisition unit 2100, and outputs the extracted sentence features to the sentence feature transmission unit 2300. When there are multiple input natural language phrases, the sentence feature extraction unit 2200 extracts sentence features for each phrase. When there are M phrases, the sentence feature extraction unit 2200 extracts M sentence features. The sentence feature extraction unit 2200 performs the above-mentioned processing by executing a second partial model, which is a part of a pre-trained DNN model and whose output does not depend on the input of image data but only on the input of natural language phrases.

[0028] The sentence feature sending unit 2300 sends the sentence features output by the sentence feature extraction unit 2200 to the sentence feature receiving unit 1300 of the image recognition device 1000. When there are multiple sentence features, the sentence feature sending unit 2300 sends all of these sentence features.

[0029] FIG. 2 is a diagram showing a configuration example of the DNN model #0 used by the image recognition system 10 shown in FIG. 1. The DNN model #0 is a trained model trained to detect an object corresponding to the natural language phrase from within an image by inputting image data and a phrase in natural language. This DNN model #0 is composed of three partial models. The DNN model #1 is a first partial model whose output depends only on the input of image data and does not depend on the input of a phrase in natural language, and can operate independently. The DNN model #1 takes image data as input and outputs candidate regions that are candidates for the region in the image where the detection target exists, and image features for each candidate region. The DNN model #1 corresponds to the processing of the image analysis unit 1200. The DNN model #2 is a second partial model whose output depends only on the input of a phrase in natural language and does not depend on the input of image data, and can operate independently. The DNN model #2 takes a phrase in natural language as input and outputs sentence features for each phrase. The natural language phrase indicates an object category to be detected by the image recognition device 1000, and in the example of FIG. 2, phrases for detecting people, such as "male", "female", "person with a cane", and "person on a bicycle", are exemplified. DNN model #2 corresponds to the processing of the text feature extraction unit 2200. DNN model #3 is the remainder of DNN model #0 excluding the DNN model #1 and DNN model #2 parts, and is a third partial model. DNN model #3 receives the output of DNN model #1 and the output of DNN model #2 as input, and calculates the similarity between the text feature and the image feature. DNN model #3 corresponds to the processing of the feature comparison unit 1400.

[0030] Before describing the basic processing flow in the image recognition system 10, the necessary initial processing will be described.

[0031] First, a DNN model #0 is prepared in advance. The DNN model #0 is trained using a sufficiently large data set. The representation format of the region in the image follows that described above in the description of the image analysis unit 1200.

[0032] Next, DNN models having the same structure as DNN model #1, DNN model #2, and DNN model #3 are defined, and the weight coefficients are copied from the trained DNN model #0 to the corresponding layers of each DNN model. Then, DNN model #1 is placed in the image analysis unit 1200, DNN model #2 is placed in the text feature extraction unit 2200, and DNN model #3 is placed in the feature comparison unit 1400. Each DNN model is defined in a form that can be calculated by the image recognition device 1000 and the text feature extraction device 2000, respectively.

[0033] The above is the content of the initial processing. Next, the processing content in the image recognition system 10 will be described. FIG. 3 is a flowchart for explaining an example of the operation of the image recognition system 10 shown in FIG. 1. The image recognition device 1000 performs the following processing every time the image acquisition unit 1100 acquires image data. First, the image acquisition unit 1100 acquires image data periodically (step S2110). The acquisition period is determined in advance according to the application of the image recognition system 10.

[0034] Next, the image analysis unit 1200 uses the DNN model #1 to extract multiple candidate regions from the acquired image data and extract image features from each candidate region (step S2120). The number of candidate regions obtained here is set to N. After the candidate regions and image features are obtained, the feature comparison unit 1400 selects one of the candidate regions if there are multiple candidate regions, starts a feature comparison process for each candidate region (step S2130), and waits for the text feature receiving unit 1300 to receive the text feature.

[0035] Meanwhile, in the sentence feature extraction device 2000, the following processes are performed in parallel with the processes from step S2110 to step S2130 described above. The sentence acquisition unit 2100 performs a sentence acquisition process to acquire a phrase in a natural language input by the user of the image recognition system 10 (step S2210), and outputs the acquired phrase in the natural language to the sentence feature extraction unit 2200. The phrase in the natural language indicates an object category to be detected by the image recognition device 1000.

[0036] The text feature extraction unit 2200 starts a text feature extraction process to extract text features from the words and phrases in the natural language output by the text acquisition unit 2100 (step S2220). The text feature extraction unit 2200 uses the DNN model #2 to extract text features from the words and phrases in the given natural language (step S2230). The text feature extraction unit 2200 judges whether all the words and phrases acquired by the text acquisition unit 2100 have been processed (step S2240). If there are words and phrases that have not yet been processed (step S2240: No), the text feature extraction unit 2200 repeats the process of step S2230 for the next word and phrase. If all the words and phrases have been processed (step S2240: Yes), the text feature extraction unit 2200 ends the text feature extraction process (step S2250). If the number of words and phrases in the natural language is M, the text feature extraction unit 2200 repeats the processes from step S2220 to step S2250 M times.

[0037] When M sentence features corresponding to each natural language phrase are obtained, the sentence feature transmitting unit 2300 transmits the sentence features to the sentence feature receiving unit 1300 of the image recognition device 1000 in association with the natural language phrase (step S2260).

[0038] When the text feature receiving unit 1300 receives the natural language phrase and text feature, the feature comparing unit 1400 selects one of the received natural language phrases and starts a feature amount comparison process for each phrase (step S2140). The feature comparing unit 1400 performs a feature amount comparison between the image feature of the selected candidate region and the text feature of the selected phrase (step S2150). Specifically, the feature comparing unit 1400 calculates the similarity between the image feature and the text feature using the DNN model #3.

[0039] The feature comparison unit 1400 judges whether or not the feature amount has been compared with all the words and phrases for the target candidate region (step S2160). If there is a word that has not yet been compared (step S2160: No), the feature comparison unit 1400 selects the next word and repeats the process from step S2150. If there is a word that has not yet been compared with all the words and phrases (step S2160: Yes), the feature comparison unit 1400 ends the feature amount comparison process for each word and phrase for the target candidate region (step S2170) and judges whether or not the feature amount has been compared with all the candidate regions (step S2180). If there is a candidate region that has not yet been compared (step S2180: No), the feature comparison unit 1400 selects the next candidate region and repeats the process from step S2140. If there is a feature amount that has not yet been compared with all the candidate regions (step S2180: Yes), the feature comparison unit 1400 ends the feature amount comparison process for each candidate region (step S2190). Here, when the calculation of similarity for all combinations of the text features of the M words and the image features of the N candidate regions is completed, the feature comparison unit 1400 stores the word with the highest similarity for each candidate region.

[0040] When the detection result generation unit 1500 obtains the comparison results from the feature comparison unit 1400, if the information for each candidate area is a relative value to the width and height of the image data, it converts the information for the candidate area to the original image coordinate reference value, and if there are any overlapping candidate areas, it compares the size of the overlap between the areas and the associated sentence features.If the overlap is sufficiently large and there are multiple areas with the same associated sentence features, it determines that they are overlapping candidate areas, keeps one of the overlapping areas, and removes unnecessary candidate areas (step S2310), generates a detection result, and outputs the generated detection result to the detection result transmission unit 1600.

[0041] The detection result transmission unit 1600 transmits the detection result output by the detection result generation unit 1500 to the result display unit 3000 (step S2320). After transmitting the detection result, the image recognition device 1000 waits for the acquisition of the next image data, and starts the process again from step S2110.

[0042] The above is the flow of the process of object detection in the image recognition system 10 according to the first embodiment. Usually, the process of the DNN model #0 is performed using an integrated computing device, but in this embodiment, the process of extracting text features that does not depend on the feature extraction result of the image data is separated and executed by another computer, thereby facilitating object detection in devices with limited computing power, for example, embedded devices such as surveillance cameras. The DNN model #0 is a trained model that receives image data and natural language phrases that specify the category of the object to be detected from the image data as input, and outputs the detection result, and is trained using a sufficiently large data set. Therefore, when a user of the image recognition system 10 wants to change the object category to be detected, the user can change the object category to be detected without re-learning the DNN model by simply changing the natural language phrases provided to the text acquisition unit 2100.

[0043] 3, the loop in which a comparison process is performed for one image feature with multiple sentence features is repeated the same number of times as the number of image features, but the order of the loops may be reversed. In other words, the same results can be obtained by repeating the loop in which a comparison process is performed for one sentence feature with multiple image features the same number of times as the number of sentence features.

[0044] In addition, the processing of the feature comparison unit 1400 uses the DNN model #3, but depending on the structure of the DNN model #0, this part does not necessarily have to be a DNN model. For example, the feature comparison unit 1400 may be a processing unit that calculates the cosine similarity between the image feature and the text feature, or a processing unit that calculates the Euclidean distance between the image feature and the text feature. However, it is assumed that the DNN model #0 has been sufficiently learned by applying the above-mentioned processing unit when learning in advance.

[0045] For the sake of explanation, the result display unit 3000 and the text feature extraction device 2000 are shown as separate entities in FIG. 1, but the text feature extraction device 2000 may also function as the result display unit 3000 .

[0046] As described above, according to the first embodiment, it is possible to provide an image recognition system 10 that includes the image recognition device 1000 and the text feature extraction device 2000 and detects a detection target from image data of a recognition target. The text feature extraction device 2000 includes a text acquisition unit 2100 that acquires a phrase in a natural language for specifying a detection target, a text feature extraction unit 2200 that extracts a text feature that is a feature amount comparable to a feature amount of an image from the phrase in the natural language, and a text feature transmission unit 2300 that transmits the text feature. The image recognition device 1000 includes a text feature reception unit 1300 that receives the text feature from the text feature extraction device 2000, an image acquisition unit 1100 that acquires image data, an image analysis unit 1200 that extracts a candidate area that is a candidate for an area in the image where the detection target exists and an image feature that is a feature amount of an image in the candidate area from the image data, a feature comparison unit 1400 that compares the text feature with the image feature corresponding to the candidate area and outputs a comparison result, and a detection result generation unit 1500 that generates a detection result of the detection target based on the comparison result. The image recognition system 10 is characterized in that the processing of the sentence feature extraction unit 2200 is executed by a sentence feature extraction device 2000, which is a device separate from the image recognition device 1000. As a result, the high-load processing of extracting sentence features from phrases in natural language can be executed by the sentence feature extraction device 2000, which is a device separate from the image recognition device 1000. Therefore, even if the computing power of the image recognition device 1000 is limited, it becomes possible to implement an inference process of detecting a detection target specified by a phrase input in natural language from image data using a trained model.

[0047] The image analysis unit 1200 is realized by executing the DNN model #1, which is a part of the DNN model #0, which is a trained model for detecting a detection target specified by a phrase in a natural language from image data, and is a first partial model whose output does not depend on the input of a natural language. The DNN model #0 is, for example, a generation AI (Artificial Intelligence). A generation AI that inputs multimodal data such as image data and phrases in a natural language requires a large amount of calculation and requires a very high computing power. Therefore, when operating it in an embedded device such as a surveillance camera, the computing power may be insufficient and implementation may be difficult. By using the technology of this embodiment, even if an embedded device such as a surveillance camera does not have enough computing power and it is difficult to process multimodal data collectively, it is possible to execute some of the processing with a high processing load in an external device.

[0048] In addition, in the image recognition system 10, the sentence feature extraction unit 2200 may be realized by executing DNN model #2, which is a part of DNN model #0, which is a trained model for detecting a detection target specified by a phrase in a natural language from image data, and is a second partial model whose output does not depend on the input of image data.

[0049] In addition, in the image recognition system 10, the feature comparison unit 1400 may be realized by executing DNN model #3, which is a third partial model that is part of DNN model #0, which is a trained model for detecting a detection target specified by a phrase in a natural language from image data, and that accepts as input the output of DNN model #1, which is a first partial model that does not depend on natural language input, and the output of DNN model #2, which is a second partial model that does not depend on image data input, and outputs a comparison result between text features and image features.

[0050] According to the first embodiment, it is also possible to provide an image recognition device 1000 that detects a detection target from image data of a recognition target. The image recognition device 1000 is characterized by including a sentence feature receiving unit 1300 that receives sentence features extracted from a phrase in a natural language for specifying a detection target, which are features comparable to image features, from outside the image recognition device 1000, an image acquiring unit 1100 that acquires image data, an image analyzing unit 1200 that extracts a candidate area that is a candidate for an area in the image where the detection target exists, and an image feature that is a feature of an image in the candidate area, from the image data, a feature comparing unit 1400 that compares the sentence feature with the image feature corresponding to the candidate area and outputs a comparison result, and a detection result generating unit 1500 that generates a detection result of the detection target based on the comparison result.

[0051] Also, in the image recognition device 1000, the image analysis unit 1200 may be realized by executing a DNN model #1, which is a part of the DNN model #0, which is a trained model for detecting a detection target specified by a phrase in natural language from image data, and is a first partial model whose output does not depend on the input of natural language. Also, in the image recognition device 1000, the feature comparison unit 1400 may be realized by executing a DNN model #3, which is a third partial model that is a part of the DNN model #0, which is a trained model, and receives as input the output of the DNN model #1, which is a first partial model, and the output of the DNN model #2, which is a part of the DNN model #0, which is a trained model, and is a second partial model that does not depend on the input of image data, and outputs a comparison result.

[0052] Moreover, according to the first embodiment, it is also possible to provide a sentence feature extraction device 2000. The sentence feature extraction device 2000 is characterized by including a sentence acquisition unit 2100 that acquires a phrase in a natural language for specifying a detection target to be detected from image data, a sentence feature extraction unit 2200 that extracts sentence features, which are features comparable to the features of an image, from the phrase in the natural language, and a sentence feature transmission unit 2300 that transmits the sentence features to the image recognition device 1000 that detects the detection target from image data.

[0053] In the sentence feature extraction device 2000, the sentence feature extraction unit 2200 is part of DNN model #0, which is a trained model for detecting a detection target specified by a phrase in a natural language from image data, and can be realized by executing DNN model #2, which is a second partial model whose output does not depend on the input of image data.

[0054] In addition, although the hardware configuration will be described later, according to the first embodiment, it is also possible to provide a program for causing a computer to function as the image recognition device 1000 to detect a detection target from image data of a recognition target. This program can cause a computer functioning as the image recognition device 1000 to execute the following steps: receiving sentence features, which are features comparable to image features and are extracted from natural language phrases for specifying the detection target in a computer different from the computer; acquiring image data; extracting from the image data a candidate area that is a candidate for an area in the image where the detection target exists, and image features that are features of the image in the candidate area; comparing the sentence features with the image features corresponding to the candidate area and outputting the comparison result; and generating a detection result of the detection target based on the comparison result.

[0055] Moreover, according to the first embodiment, it is also possible to provide a program for causing a computer to function as the sentence feature extraction device 2000. This program can cause a computer functioning as the sentence feature extraction device 2000 to execute the following steps: acquiring a phrase in a natural language for specifying a detection target to be detected from image data, extracting sentence features, which are features comparable to image features, from the phrase in the natural language, and transmitting the sentence features to a computer different from the computer functioning as the sentence feature extraction device 2000 and functioning as the image recognition device 1000 for detecting the detection target from image data.

[0056] Moreover, according to the first embodiment, an image recognition method for detecting a detection target from image data of a recognition target can be provided. This image recognition method includes a step of extracting a sentence feature, which is a feature quantity comparable to the feature quantity of an image, from a phrase in a natural language for specifying the detection target, a step of extracting a candidate area, which is a candidate for an area in the image where the detection target exists, from the image data, and an image feature, which is a feature quantity of an image in the candidate area, a step of comparing the sentence feature with the image feature corresponding to the candidate area to generate a comparison result, and a step of generating a detection result of the detection target based on the comparison result. This image recognition method is also characterized in that the process of extracting the sentence feature and the process of extracting the image feature are performed by different devices. For example, the image recognition method is performed by an image recognition system 10 including an image recognition device 1000 and a sentence feature extraction device 2000, and the sentence feature extraction device 2000 can execute a step of extracting the sentence feature, and the image recognition device 1000 can execute a step of extracting the candidate area and the image feature, a step of generating a comparison result, and a step of generating a detection result. The image recognition method may further include a step of acquiring a phrase in a natural language, a step of transmitting the extracted sentence features, and a step of receiving the sentence features. In this case, the sentence feature extraction device 2000 executes the step of acquiring a phrase in a natural language and the step of transmitting the extracted sentence features, and the image recognition device 1000 executes the step of receiving the sentence features and executes the step of generating a comparison result using the received sentence features. In the above-mentioned image recognition method, the processing of each step described does not necessarily have to be executed in the order described. The image recognition method may include multiple steps executed simultaneously in parallel. In the image recognition method, the processing of each step described does not necessarily have to be a series of processes executed consecutively, and may include multiple processes executed at different times.

[0057] Embodiment 2 4 is a configuration diagram of an image recognition system 20 according to the second embodiment. The image recognition system 20 has an image recognition device 1050, a sentence feature extraction device 2000, and a result display unit 3000. The image recognition system 20 has an image recognition device 1050 instead of the image recognition device 1000 of the image recognition system 10. Below, differences from the first embodiment will be mainly described, and detailed description of the same parts as those in the first embodiment will be omitted.

[0058] The image recognition system 20 executes the process of extracting sentence features only when the object category to be detected is changed or added.

[0059] The image recognition device 1050 includes an image acquisition unit 1100, an image analysis unit 1200, a text feature receiving unit 1300, a feature comparison unit 1400, a detection result generation unit 1500, and a detection result transmission unit 1600. The image recognition device 1050 includes a text feature storage unit 1350 in addition to the components of the image recognition device 1000.

[0060] The sentence feature storage unit 1350 stores the sentence features received by the sentence feature receiving unit 1300 from the sentence feature extraction device 2000. The sentence feature storage unit 1350 is capable of storing a plurality of sentence features and is capable of outputting the stored sentence features to the feature comparison unit 1400.

[0061] Figure 5 is a flowchart for explaining an example of the operation of the image recognition system 20 shown in Figure 4. Figure 5 is obtained by adding step S2330 to the flowchart of Figure 3. Since the steps other than step S2330 are the same as those in Figure 3, detailed explanations of each step will be omitted, and only the operations that differ from those in Figure 3 will be explained below.

[0062] In image recognition system 20, a series of processes from step S2210 to step S2250, which is the process of extracting sentence features, is not necessarily executed in parallel with the image feature extraction process, but is basically executed independently at any timing earlier than the image feature extraction process. After sentence feature sending unit 2300 sends the sentence features in step S2260, sentence feature receiving unit 1300 stores the received sentence features in sentence feature storage unit 1350 (step S2330).

[0063] After the sentence features are stored in the sentence feature storage unit 1350, the feature comparison unit 1400 executes a process of comparing the sentence features with the image features in step S2150. The subsequent process is the same as that in FIG.

[0064] Furthermore, when the sentence acquisition process of step S2210 occurs at any timing, the sentence feature extraction device 2000 executes the processes of steps S2220 to S2260 again, and the sentence feature receiving unit 1300 updates the contents of the sentence feature storage unit 1350.

[0065] As described above, in the image recognition system 20 according to the second embodiment, the image recognition device 1050 further includes the sentence feature storage unit 1350 that stores the sentence feature received by the sentence feature receiving unit 1300, and the feature comparison unit 1400 can compare the sentence feature stored in the sentence feature storage unit 1350 with the image feature. With this configuration, it is possible to extract the sentence feature and store it in the image recognition device 1050 before acquiring the image data. As a result, if there is no change in the detection target, it is not necessary to extract the sentence feature, and the feature comparison unit 1400 can execute the feature amount comparison process using the sentence feature stored in advance in the sentence feature storage unit 1350 without waiting for the reception of the sentence feature. Therefore, the throughput of the image recognition device 1050 is improved, and the calculation load of the image recognition system 20 as a whole can be reduced.

[0066] In this case, the text feature extraction device 2000 can extract text features from the new natural language phrase by the text feature extraction unit 2200 every time the text acquisition unit 2100 acquires a new natural language phrase, and transmit the extracted text features to the image recognition device 1050 by the text feature transmission unit 2300. The acquisition of a new natural language phrase does not have to be performed every time the image recognition process is executed, for example. After the detection target is initially set, image recognition is basically performed using the text features stored in the text feature storage unit 1350 until the detection target is changed, that is, an object category of the detection target is added or an object category set in the detection target is deleted. When an object category is added, the text acquisition unit 2100 acquires a new natural language phrase, and the text feature extraction unit 2200 extracts new text features. Usually, when performing inference processing using a natural language phrase and image data as input using a DNN model, it is necessary to operate the entire DNN model every time an image is acquired. In this case, unless the detection target is changed, there is actually no change in the processing contents and results of the sentence feature extraction unit 2200, resulting in poor calculation efficiency. In contrast, according to the technology of the second embodiment, unless the detection target is changed, the sentence feature extraction process is omitted, improving the throughput of the image recognition device 1050 and reducing the calculation load of the image recognition system 20 as a whole.

[0067] Here, a description will be given of a hardware configuration for realizing the functions of the image recognition systems 10 and 20. The text acquisition unit 2100, the text feature extraction unit 2200, the image acquisition unit 1100, the image analysis unit 1200, the feature comparison unit 1400, and the detection result generation unit 1500 are realized by processing circuits. These processing circuits may be realized by dedicated hardware, or may be control circuits using a CPU (Central Processing Unit).

[0068] When the above processing circuits are realized by dedicated hardware, they are realized by a processing circuit 90 shown in Fig. 6. Fig. 6 is a diagram showing dedicated hardware for realizing the functions of the image recognition systems 10 and 20 according to the first and second embodiments. The processing circuit 90 is a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of these.

[0069] When the above processing circuit is realized by a control circuit using a CPU, the control circuit is, for example, a control circuit 91 having a configuration shown in FIG. 7. FIG. 7 is a diagram showing the configuration of a control circuit for realizing the functions of the image recognition systems 10 and 20 according to the first and second embodiments. As shown in FIG. 7, the control circuit 91 includes a processor 92 and a memory 93. The processor 92 is a CPU, and is also called a processing device, an arithmetic device, a microprocessor, a microcomputer, a DSP (Digital Signal Processor), or the like. The memory 93 is, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), or an EEPROM (registered trademark) (Electrically EPROM), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disk).

[0070] When the above processing circuit is realized by the control circuit 91, it is realized by the processor 92 reading and executing a program corresponding to the processing of each component, which is stored in the memory 93. The memory 93 is also used as a temporary memory for each processing executed by the processor 92. The program executed by the processor 92 may be provided in a state stored in a storage medium, or may be provided via a communication path such as the Internet.

[0071] Also, the functions of the text feature receiving unit 1300, the detection result transmitting unit 1600, and the text feature transmitting unit 2300 can be realized using a communication device (not shown). Also, as described above, the function of the image acquisition unit 1100 can be realized using an image sensor or an imaging device. The text feature storage unit 1350 can be realized using a storage device.

[0072] The configurations shown in the above embodiments are merely examples, and it is possible to combine them with other known technologies, or to combine the embodiments with each other, and it is also possible to omit or change a part of the configuration without departing from the gist.

[0073] For example, in FIGS. 1 and 4, for simplicity, only one image recognition device 1000, 1050 and one text feature extraction device 2000 are shown, but the image recognition systems 10, 20 may include a plurality of image recognition devices 1000, 1050. The text feature extraction device 2000 can transmit text features to a plurality of image recognition devices 1000, 1050. Also, depending on the required computing power, the image recognition systems 10, 20 may include a plurality of text feature extraction devices 2000.

[0074] Also, in the above embodiment, the case where the image analysis unit 1200 of the image recognition devices 1000, 1050 extracts a plurality of candidate regions has been described, but there may be a case where only one candidate region can be extracted from the image. In this case, the loop process from step S2130 to step S2190 in FIGS. 3 and 5 is executed only once.

[0075] In the above embodiment, the image recognition devices 1000, 1050 receive the sentence features directly from the sentence feature extraction device 2000, but the present invention is not limited to this example. It is sufficient that the sentence feature extraction process is performed outside the image recognition devices 1000, 1050. For example, the sentence features extracted by the sentence feature extraction device 2000 may be stored in a computer other than the sentence feature extraction device 2000, and the image recognition devices 1000, 1050 may receive the sentence features from the computer that stores the sentence features. [Explanation of symbols]

[0076] 10,20 image recognition system, 90 processing circuit, 91 control circuit, 92 processor, 93 memory, 1000,1050 image recognition device, 1100 image acquisition unit, 1200 image analysis unit, 1300 sentence feature receiving unit, 1350 sentence feature storage unit, 1400 feature comparison unit, 1500 detection result generation unit, 1600 detection result transmission unit, 2000 sentence feature extraction device, 2100 sentence acquisition unit, 2200 sentence feature extraction unit, 2300 sentence feature transmission unit, 3000 result display unit.

Claims

1. An image recognition system that detects a target from image data to be recognized, It comprises a text feature extraction device and an image recognition device, The aforementioned text feature extraction device is A text acquisition unit that acquires natural language phrases to specify the target for detection, A text feature extraction unit extracts text features from the aforementioned natural language phrases, which are features comparable to image features. A text feature transmission unit that transmits the aforementioned text features, It has, The aforementioned image recognition device is A text feature receiving unit that receives the text features from the text feature extraction device, An image acquisition unit that acquires the aforementioned image data, An image analysis unit extracts candidate regions from the image data, which are candidate regions in which the detection target exists within the image, and image features, which are feature quantities of the image within the candidate regions. A feature comparison unit compares the aforementioned text features with the aforementioned image features corresponding to the candidate region and outputs the comparison result. A detection result generation unit generates a detection result for the target to be detected based on the comparison result, It has, The processing of the aforementioned text feature extraction unit is performed by the text feature extraction device, which is a separate device from the image recognition device. An image recognition system characterized by the following features.

2. The image analysis unit executes a first submodel, which is part of a trained model for detecting the target specified by the natural language phrase from the image data, and whose output does not depend on the natural language input. The image recognition system according to feature 1.

3. The text feature extraction unit executes a second subset model which is part of a trained model for detecting the target specified by the natural language phrases from the image data, and whose output does not depend on the image data input. The image recognition system according to feature 1.

4. The feature comparison unit executes a third subset model that is part of a trained model for detecting the target specified by the natural language phrase from the image data. This subset model receives the output of a first subset model that does not depend on the natural language input and the output of a second subset model that does not depend on the image data input as input, and outputs the comparison result. The image recognition system according to feature 1.

5. The aforementioned image recognition device is A text feature storage unit that stores the text features received by the text feature receiving unit, It further possesses, The feature comparison unit compares the text features stored in the text feature storage unit with the image features. The image recognition system according to any one of claims 1 to 4.

6. The aforementioned text feature extraction device is Each time the text acquisition unit acquires a new natural language phrase, the text feature extraction unit extracts text features from the new natural language phrase, and the text feature transmission unit transmits the extracted text features. The image recognition system according to feature 5.

7. An image recognition system that detects a target from image data to be recognized, The system comprises an image recognition device and a text feature extraction device with higher computational processing capabilities than the image recognition device, The aforementioned text feature extraction device is A text acquisition unit that acquires natural language phrases to specify the target for detection, A text feature extraction unit extracts text features from the aforementioned natural language phrases, which are features comparable to image features. A text feature transmission unit that transmits the aforementioned text features, It has, The aforementioned image recognition device is A text feature receiving unit that receives the text features from the text feature extraction device, An image acquisition unit that acquires the aforementioned image data, An image analysis unit extracts candidate regions from the image data, which are candidate regions in which the detection target exists within the image, and image features, which are feature quantities of the image within the candidate regions. A feature comparison unit compares the aforementioned text features with the aforementioned image features corresponding to the candidate region and outputs the comparison result. A detection result generation unit generates a detection result for the target to be detected based on the comparison result, It has, The processing of the aforementioned text feature extraction unit is performed by the text feature extraction device, which is a separate device from the image recognition device. An image recognition system characterized by the following features.

8. An image recognition device that detects a target from image data of the target to be recognized, A text feature receiving unit receives from outside the image recognition device text features extracted from natural language phrases that specify the detection target, which are comparable to the features of an image. An image acquisition unit that acquires the aforementioned image data, An image analysis unit extracts candidate regions from the image data, which are candidate regions in which the detection target exists within the image, and image features, which are feature quantities of the image within the candidate regions. A feature comparison unit compares the aforementioned text features with the aforementioned image features corresponding to the candidate region and outputs the comparison result. A detection result generation unit generates a detection result for the target to be detected based on the comparison result, Equipped with An image recognition device characterized by the following features.

9. The image analysis unit executes a first submodel, which is part of a trained model for detecting the target specified by the natural language phrase from the image data, and whose output does not depend on the natural language input. The image recognition device according to feature 8.

10. The feature comparison unit executes a third submodel that is part of the trained model, receiving the output of the first submodel and the output of a second submodel, which is part of the trained model and does not depend on the image data input, as inputs, and outputs the comparison result. The image recognition device according to feature 9.

11. A text feature storage unit that stores the text features received by the text feature receiving unit, Furthermore, The feature comparison unit compares the text features stored in the text feature storage unit with the image features. The image recognition device according to any one of claims 8 to 10.

12. A text acquisition unit that obtains natural language phrases to specify the detection target from image data, A text feature extraction unit extracts text features from the aforementioned natural language phrases, which are features comparable to image features. A text feature transmission unit transmits the aforementioned text features to an image recognition device that detects the target from the image data, Equipped with A text feature extraction device characterized by the following features.

13. The text feature extraction unit executes a second subset model which is part of a trained model for detecting the target specified by the natural language phrases from the image data, and whose output does not depend on the image data input. The text feature extraction device according to claim 12.

14. When a new natural language phrase is input, the text feature extraction unit extracts the text features by executing the second partial model, and the text feature transmission unit transmits the extracted text features to the image recognition device. If there is no change in the detection target and no new natural language phrases are input, the execution of the second partial model is omitted even if new image data is acquired. The text feature extraction device according to claim 13.

15. A program that causes a computer to function as an image recognition device that detects a target from image data to be recognized, The steps include receiving textual features extracted from natural language phrases used to specify the detection target on a computer different from the computer in question, which are comparable to the features of an image. The steps include acquiring the aforementioned image data, The steps include extracting candidate regions from the image data, which are candidate regions in which the object to be detected exists within the image, and image features, which are feature quantities of the image within the candidate regions. The steps include comparing the text features with the image features corresponding to the candidate region and outputting the comparison results, The steps include generating the detection result of the target to be detected based on the comparison result, This is performed by a computer that functions as the image recognition device. A program characterized by the following features.

16. A program that makes a computer function as a text feature extraction device, The steps include obtaining natural language phrases to specify the target to be detected from the image data, The steps include: extracting textual features from the aforementioned natural language phrases, which are features comparable to image features; The steps include transmitting the aforementioned text features to a computer different from the computer that functions as the text feature extraction device, which functions as an image recognition device that detects the target from the image data, This is performed by a computer that functions as the aforementioned text feature extraction device. A program characterized by the following features.

17. An image recognition method for detecting a target from image data to be recognized, The steps include: extracting textual features, which are features comparable to image features, from natural language phrases used to specify the target for detection; The steps include extracting candidate regions from the image data, which are candidate regions in which the object to be detected exists within the image, and image features, which are feature quantities of the image within the candidate regions. A step of comparing the aforementioned text features with the aforementioned image features corresponding to the candidate region to generate a comparison result, The steps include generating the detection result of the target to be detected based on the comparison result, Includes, The process of extracting the text features and the process of extracting the image features are performed on different devices. An image recognition method characterized by the following features.

18. A program for causing a computer to function as an image recognition device that detects a target from image data of a target to be recognized, The steps include receiving textual features extracted from natural language phrases used to specify the detection target on a computer different from the computer in question, which are comparable to the features of an image. The steps include: memorizing the aforementioned textual features, The steps include acquiring the aforementioned image data, The steps include extracting candidate regions from the image data, which are candidate regions in which the object to be detected exists within the image, and image features, which are feature quantities of the image within the candidate regions. The steps include comparing the stored text features with the image features corresponding to the candidate region and outputting the comparison result, The steps include generating the detection result of the target to be detected based on the comparison result, This is performed by a computer that functions as the image recognition device. A program characterized by the following features.

19. A program for causing a computer to function as a text feature extraction device, A step of obtaining a natural language phrase to specify the target to be detected from the image data each time a new natural language phrase is input, The steps include: extracting textual features from the aforementioned natural language phrases, which are features comparable to image features; The steps include transmitting the aforementioned text features to a computer different from the computer that functions as the text feature extraction device, which stores the received text features and functions as an image recognition device that uses the stored text features to detect the target from the image data, This is performed by a computer that functions as the aforementioned text feature extraction device. A program characterized by the following features.

20. An image recognition system comprising an image recognition device that executes the program described in Claim 18.

21. An image recognition system comprising a text feature extraction device that executes the program described in Claim 19.

Images