Image recognition system, image recognition device, sentence feature extraction device, program, and image recognition method

Abstract

This image recognition system (10) comprises a sentence feature extraction device (2000) and an image recognition device (1000). The sentence feature extraction device (2000) comprises: a sentence acquisition unit (2100) that acquires a phrase of a natural language input by a user; a sentence feature extraction unit (2200) that extracts a sentence feature from the phrase of the natural language; and a sentence feature transmission unit (2300) that transmits the sentence feature. The image recognition device (1000) comprises: a sentence feature reception unit (1300) that receives the sentence feature from the sentence feature extraction device (2000); an image acquisition unit (1100) that acquires image data; an image analysis unit (1200) that extracts, from the image data, a candidate region and an image feature that is a feature amount of an image in the candidate region; a feature comparison unit (1400) that compares the sentence feature with the image feature corresponding to the candidate region and outputs a comparison result; and a detection result generation unit (1500) that generates a detection result of a detection target on the basis of the comparison result.

Description

Image recognition system, image recognition device, text feature extraction device, program, and image recognition method

[0001] The present disclosure relates to an image recognition system, an image recognition device, a text feature extraction device, a program, and an image recognition method that detect a detection target specified by a phrase in natural language from image data.

[0002] Image recognition technology, which detects targets from image data, has become significantly more accurate with the development of artificial intelligence, and is expected to lead to the realization of advanced security systems, etc. The trained models used in artificial intelligence require significantly more computation than conventional image recognition technology, making them difficult to implement in devices with limited computing power, such as surveillance cameras.

[0003] To address the above-mentioned problem, Non-Patent Document 1 discloses a technology called Grounding DINO (self-DIstillation with NO labels). Grounding DINO is a technology that can detect a wider range of object categories by training a DNN model in advance using a huge amount of image data, 10 to 100 times larger than that of conventional deep neural networks (DNNs), and paired natural language phrases as input. Furthermore, with Grounding DINO, by specifying the object category to be detected using a natural language phrase during inference, it is possible to change or add the object category to be detected without retraining the DNN model.

[0004] Shilong Liu and 10 others, "Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection", [online], [Retrieved December 1, 2023], Internet <URL: https: / / arxiv.org / pdf / 2303.05499.pdf>

[0005] However, even with the technology disclosed in Non-Patent Document 1, the amount of calculation required for the inference process to detect a detection target specified by a phrase input in natural language from image data using a trained model is still very large, and there is a problem in that it is difficult to implement when the computing power of the image recognition device is limited.

[0006] The present disclosure has been made in consideration of the above, and aims to provide an image recognition system that can implement an inference process that uses a trained model to detect a detection target specified by a phrase input in natural language from image data, even when the computational power of the image recognition device is limited.

[0007] To solve the above-mentioned problems and achieve the object, the image recognition system disclosed herein is an image recognition system that detects a detection target from image data of a recognition target, and includes a text feature extraction device and an image recognition device. The text feature extraction device includes a text acquisition unit that acquires natural language phrases for specifying the detection target, a text feature extraction unit that extracts text features, which are features comparable to image features, from the natural language phrases, and a text feature transmission unit that transmits the text features. The image recognition device includes a text feature reception unit that receives the text features from the text feature extraction device, an image acquisition unit that acquires image data, an image analysis unit that extracts, from the image data, candidate areas that are candidates for areas in the image where the detection target exists and image features, which are features of the image within the candidate areas, a feature comparison unit that compares the text features with the image features corresponding to the candidate areas and outputs a comparison result, and a detection result generation unit that generates a detection result for the detection target based on the comparison result. The image recognition system is characterized in that the processing of the text feature extraction unit is performed by the text feature extraction device, which is a device separate from the image recognition device.

[0008] According to the present disclosure, even if the computational power of the image recognition device is limited, it is possible to obtain an image recognition system that can implement an inference process that uses a trained model to detect a detection target specified by a phrase input in natural language from image data.

[0009] 1 is a block diagram of an image recognition system according to a first embodiment; FIG. 2 is a block diagram of an image recognition system according to a second embodiment; FIG. 3 is a block diagram of an image recognition system according to a first embodiment; FIG. 4 is a block diagram of an image recognition system according to a second embodiment;

[0010] An image recognition system, an image recognition device, a text feature extraction device, a program, and an image recognition method according to embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0011] First Embodiment. FIG. 1 is a configuration diagram of an image recognition system 10 according to a first embodiment. The image recognition system 10 includes an image recognition device 1000, a sentence feature extraction device 2000, and a result display unit 3000. The image recognition system 10 has a function of detecting a detection target specified by a phrase input in natural language from image data by performing inference processing using a trained model. The trained model may be generated using a technology such as Grounding DINO. In the image recognition system 10, the inference processing is shared and executed by the image recognition device 1000 and the sentence feature extraction device 2000.

[0012] Specifically, in the following embodiment, the process of extracting text features, which is a high-load process among the inference processes, in other words a process with a large amount of calculation, is executed by the text feature extraction device 2000, and the other processes are executed by the image recognition device 1000. Here, the allocation of the processes to be shared between the image recognition device 1000 and the text feature extraction device 2000 is determined in advance by a human being, but other methods of determination may also be used.

[0013] In addition, as an example of the present disclosure, even if the sentence feature extraction device 2000 is composed of multiple devices or has multiple arithmetic units, the multiple arithmetic units included therein are treated as a single entity. Similarly, even if the image recognition device 1000 is composed of multiple devices or has multiple arithmetic units, the multiple arithmetic units included therein are treated as a single entity. In other words, regardless of the configuration of each arithmetic unit of the image recognition device 1000 and the sentence feature extraction device 2000, they are treated as a single entity that executes the above-described inference process. As a result, for example, even if the image recognition device 1000 is a single imaging device with a single arithmetic unit and the sentence feature extraction device 2000 is one or more server devices with multiple arithmetic units, the image recognition device 1000 and the sentence feature extraction device 2000 can be compared in terms of their respective arithmetic processing capabilities as a single entity. In the example of the present disclosure, the sentence feature extraction device 2000 is considered to have higher arithmetic processing capabilities than the image recognition device 1000.

[0014] The image recognition device 1000 includes an image acquisition unit 1100, an image analysis unit 1200, a sentence feature receiving unit 1300, a feature comparison unit 1400, a detection result generation unit 1500, and a detection result transmission unit 1600. The image recognition device 1000 determines whether a specified object is present in the image acquired by the image acquisition unit 1100, and if so, recognizes the area occupied by the object within the image, and transmits the recognition result to a result display unit 3000. The image recognition device 1000 may be, for example, a surveillance camera with a built-in microcomputer capable of executing a processing device or processing software for image recognition, a digital camera, or a computer connected to a surveillance camera. The image recognition device 1000 is not limited to the above example. For simplicity, in this embodiment, the image recognition system 10 includes one image recognition device 1000. However, in practice, the image recognition system 10 may include multiple image recognition devices 1000.

[0015] The sentence feature extraction device 2000 includes a sentence acquisition unit 2100, a sentence feature extraction unit 2200, and a sentence feature transmission unit 2300. The sentence feature extraction device 2000 is a device different from the image recognition device 1000. When a user of the image recognition system 10 specifies an object category that the image recognition device 1000 is to detect by inputting a phrase in natural language, the sentence feature extraction device 2000 extracts sentence features from the input phrase in natural language and transmits the extracted sentence features to the image recognition device 1000. Here, the sentence features are features that can be compared with the features of an image.

[0016] The result display unit 3000 is a device for displaying the results of object detection from an image performed by the image recognition device 1000. The result display unit 3000 only needs to have a display function, and there are no particular limitations on the specific configuration.

[0017] The following describes in detail each unit of the image recognition system 10. The image acquisition unit 1100 acquires image data to be processed and outputs the acquired image data to the image analysis unit 1200. The image acquisition unit 1100 is, for example, an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), or an imaging device such as a digital camera or a surveillance camera connected to a computer.

[0018] The image analysis unit 1200 extracts a plurality of candidate areas, which are areas in which an object is thought to appear, from the image data acquired by the image acquisition unit 1100, and extracts image features, which are feature quantities contained in each of the extracted candidate areas. The image analysis unit 1200 outputs the extracted plurality of candidate areas and the image features corresponding to each candidate area to the feature comparison unit 1400. The image analysis unit 1200 extracts the candidate areas and image features using a first partial model, which is part of a pre-trained DNN model and is a part whose output does not depend on natural language input but depends only on image data input.

[0019] The image analysis unit 1200 can extract the same number of image features as the number of candidate regions. For example, if the image analysis unit 1200 extracts N candidate regions, it can extract N image features.

[0020] Here, the representation format of the information indicating the candidate area is not particularly limited as long as it can represent the range of the candidate area in the image. For example, the shape of the candidate area can be a geometric shape, and the position of the geometric shape can be specified to represent the candidate area. The geometric shape may be, for example, a polygon such as a rectangle, or a circle. If the shape of the candidate area is a polygon, the position of the candidate area can be represented using the coordinates of the polygon's vertices, the lengths of its sides, the coordinates of its center of gravity, and the like. For example, if the shape of the candidate area is rectangular, the candidate area may be represented by the coordinates of the four vertices of the rectangle, or by a combination of the coordinates of two of the four vertices of the rectangle located diagonally, or by the coordinates of one of the four vertices of the rectangle and the numerical values ​​of the lengths of each side of the rectangle, or by the coordinates of the center of gravity of the rectangle and the numerical values ​​of the lengths of each side of the rectangle. The numerical values ​​of the vertex coordinates, the center of gravity coordinates, and the lengths of each side may be a combination of the image size and actual coordinates, or may be relative values ​​to the image size.

[0021] The sentence feature receiving unit 1300 receives sentence features transmitted from the sentence feature extraction device 2000 and outputs the received sentence features to the feature comparing unit 1400. There are as many sentence features as there are specified object categories, and if there are M specified object categories, M sentence features are also received. If the sentence features are Q-dimensional vectors, the sentence feature receiving unit 1300 will receive data of M Q-dimensional vectors.

[0022] The feature comparison unit 1400 performs a similarity calculation by comparing N image features corresponding to each candidate region output by the image analysis unit 1200 with M text features output by the text feature receiving unit 1300, and calculates which of the M text features each image feature is most similar to. The feature comparison unit 1400 outputs the similarity calculation result to the detection result generation unit 1500 as a comparison result. The feature comparison unit 1400 is part of a pre-trained DNN model, and performs the above-mentioned processing by executing a third partial model that receives as input the output of a first partial model executed by the image analysis unit 1200 and the output of a second partial model executed by the text feature extraction unit 2200 (described later).

[0023] The detection result generation unit 1500 identifies the object category corresponding to the sentence feature to which each candidate area is most similar based on the comparison result output by the feature comparison unit 1400, and outputs the identified object category as the detection result to the detection result transmission unit 1600. Furthermore, if there is overlap between the object category and the candidate area, the redundant result is removed from the detection result.

[0024] The detection result transmitting unit 1600 outputs the detection result output by the detection result generating unit 1500 to the result display unit 3000 .

[0025] The result display unit 3000 displays the detection results to the user of the image recognition system 10 in a desired manner based on the detection results received from the detection result transmission unit 1600. There are no particular limitations on the method for displaying the detection results.

[0026] The sentence acquisition unit 2100 accepts input of natural language phrases of object categories that the user of the image recognition system 10 wants the image recognition device 1000 to detect, and outputs the accepted phrases to the sentence feature extraction unit 2200. The sentence acquisition unit 2100 can accept input of natural language phrases by, for example, character input using an input means such as a keyboard or a touch sensor, voice recognition using a microphone, or reading data of a pre-created phrase list from a storage device. In addition, multiple phrases may be input, and each phrase corresponds to an object category.

[0027] The sentence feature extraction unit 2200 extracts sentence features from the natural language phrases acquired by the sentence acquisition unit 2100 and outputs the extracted sentence features to the sentence feature transmission unit 2300. When there are multiple input natural language phrases, the sentence feature extraction unit 2200 extracts sentence features for each phrase. When there are M phrases, the sentence feature extraction unit 2200 extracts M sentence features. The sentence feature extraction unit 2200 performs the above processing by executing a second partial model, which is part of a pre-trained DNN model and whose output does not depend on the input of image data but depends only on the input of natural language phrases.

[0028] The sentence feature sending unit 2300 sends the sentence features output by the sentence feature extraction unit 2200 to the sentence feature receiving unit 1300 of the image recognition device 1000. If there are multiple sentence features, the sentence feature sending unit 2300 sends all of these sentence features.

[0029] FIG. 2 is a diagram showing an example of the configuration of DNN model #0 used by the image recognition system 10 shown in FIG. 1 . DNN model #0 is a trained model that receives image data and natural language phrases as input and is trained to detect objects corresponding to the natural language phrases from within an image. This DNN model #0 is composed of three partial models. DNN model #1 is a first partial model whose output depends only on the input of image data, not on the input of natural language phrases, and can operate independently. DNN model #1 receives image data as input and outputs candidate regions, which are candidates for regions in the image where the detection target exists, and image features for each candidate region. DNN model #1 corresponds to the processing of the image analysis unit 1200. DNN model #2 is a second partial model whose output depends only on the input of natural language phrases, not on the input of image data, and can operate independently. DNN model #2 receives natural language phrases as input and outputs sentence features for each phrase. The natural language phrase indicates an object category to be detected by the image recognition device 1000, and in the example of FIG. 2, phrases for detecting people, such as "male," "female," "person with a cane," and "person riding a bicycle," are exemplified. DNN model #2 corresponds to the processing of the sentence feature extraction unit 2200. DNN model #3 is the remainder of DNN model #0 excluding the portions of DNN model #1 and DNN model #2, and is a third partial model. DNN model #3 receives the outputs of DNN model #1 and DNN model #2 as inputs and calculates the similarity between sentence features and image features. DNN model #3 corresponds to the processing of the feature comparison unit 1400.

[0030] Before describing the basic processing flow in the image recognition system 10, the necessary initial processing will be described.

[0031] First, a DNN model #0 is prepared in advance. The DNN model #0 is trained using a sufficiently large dataset. The representation format of regions within an image follows the description of the image analysis unit 1200 above.

[0032] Next, DNN models having the same structure as DNN model #1, DNN model #2, and DNN model #3 are defined, and the weighting coefficients from trained DNN model #0 are copied to the corresponding layers of each DNN model, and then DNN model #1 is placed in image analysis unit 1200, DNN model #2 in sentence feature extraction unit 2200, and DNN model #3 in feature comparison unit 1400. Each DNN model is defined in a form that can be calculated by image recognition device 1000 and sentence feature extraction device 2000, respectively.

[0033] The above is the content of the initial processing. Next, the processing content in the image recognition system 10 will be described. FIG. 3 is a flowchart for explaining an example of the operation of the image recognition system 10 shown in FIG. 1. The image recognition device 1000 performs the following processing each time the image acquisition unit 1100 acquires image data. First, the image acquisition unit 1100 periodically acquires image data (step S2110). The acquisition period is determined in advance depending on the application of the image recognition system 10.

[0034] Next, the image analysis unit 1200 uses DNN model #1 to extract multiple candidate regions from the acquired image data and extract image features from each candidate region (step S2120). Let N be the number of candidate regions obtained. Once the candidate regions and image features are obtained, the feature comparison unit 1400 selects one of the candidate regions (if there are multiple candidate regions) and begins feature comparison processing for each candidate region (step S2130), then waits for the text feature receiving unit 1300 to receive the text features.

[0035] Meanwhile, in the sentence feature extraction device 2000, the following processing is performed in parallel with the processing of the above-mentioned steps S2110 to S2130. The sentence acquisition unit 2100 performs a sentence acquisition process to acquire natural language phrases input by the user of the image recognition system 10 (step S2210), and outputs the acquired natural language phrases to the sentence feature extraction unit 2200. The natural language phrases indicate object categories that are desired to be detected by the image recognition device 1000.

[0036] The sentence feature extraction unit 2200 starts a sentence feature extraction process to extract sentence features from natural language phrases output by the sentence acquisition unit 2100 (step S2220). The sentence feature extraction unit 2200 extracts sentence features from the given natural language phrases using DNN model #2 (step S2230). The sentence feature extraction unit 2200 determines whether all phrases acquired by the sentence acquisition unit 2100 have been processed (step S2240). If there are phrases that have not yet been processed (step S2240: No), the sentence feature extraction unit 2200 repeats the process of step S2230 for the next phrase. If all phrases have been processed (step S2240: Yes), the sentence feature extraction unit 2200 ends the sentence feature extraction process (step S2250). If the number of natural language words and phrases is M, the sentence feature extraction unit 2200 repeats the processes of steps S2220 to S2250 M times.

[0037] Once M sentence features corresponding to each natural language phrase have been obtained, the sentence feature sending unit 2300 sends the sentence features to the sentence feature receiving unit 1300 of the image recognition device 1000 in association with the natural language phrase (step S2260).

[0038] When the sentence feature receiving unit 1300 receives the natural language phrases and sentence features, the feature comparing unit 1400 selects one of the received natural language phrases and starts a feature comparison process for each phrase (step S2140). The feature comparing unit 1400 compares the image features of the selected candidate region with the sentence features of the selected phrase (step S2150). Specifically, the feature comparing unit 1400 calculates the similarity between the image features and the sentence features using DNN model #3.

[0039] The feature comparison unit 1400 determines whether the feature amounts of the target candidate region have been compared with all terms (step S2160). If there are terms that have not yet been compared (step S2160: No), the feature comparison unit 1400 selects the next term and repeats the process from step S2150. If there are terms that have not yet been compared with all terms (step S2160: Yes), the feature comparison unit 1400 ends the feature amount comparison process for each term for the target candidate region (step S2170) and determines whether the feature amounts of all candidate regions have been compared (step S2180). If there are candidate regions that have not yet been compared (step S2180: No), the feature comparison unit 1400 selects the next candidate region and repeats the process from step S2140. If there are terms that have not yet been compared with all candidate regions (step S2180: Yes), the feature comparison unit 1400 ends the feature amount comparison process for each candidate region (step S2190). Here, when the calculation of similarity for all combinations of the text features of the M words and the image features of the N candidate areas is completed, the feature comparison unit 1400 stores the word with the highest similarity for each candidate area.

[0040] When the detection result generation unit 1500 obtains the comparison results from the feature comparison unit 1400, if the information for each candidate area is a relative value to the width and height of the image data, it converts the information for the candidate area to the original image coordinate reference value, and if there are any candidate areas that overlap, it compares the size of the overlap between the areas and the associated sentence features.If the overlap is sufficiently large and there are multiple areas with the same associated sentence features, it determines that the candidate areas are overlapping, keeps one of the overlapping areas, and removes unnecessary candidate areas (step S2310), generates a detection result, and outputs the generated detection result to the detection result transmission unit 1600.

[0041] The detection result transmission unit 1600 transmits the detection result output by the detection result generation unit 1500 to the result display unit 3000 (step S2320). After transmitting the detection result, the image recognition device 1000 waits for the acquisition of the next image data and starts the process again from step S2110.

[0042] The above is the flow of the object detection process in the image recognition system 10 according to the first embodiment. While the processing of the DNN model #0 is typically performed using an integrated computing device, in this embodiment, the text feature extraction process, which is independent of the image data feature extraction results, is separated and performed on another computer, thereby facilitating object detection in devices with limited computing power, such as embedded devices such as surveillance cameras. The DNN model #0 is a trained model that accepts as input image data and natural language phrases specifying the category of the object to be detected from the image data, and outputs a detection result. The DNN model #0 has been trained using a sufficiently large dataset. Therefore, if a user of the image recognition system 10 wishes to change the object category to be detected, they can change the object category to be detected simply by changing the natural language phrases provided to the text acquisition unit 2100, without retraining the DNN model.

[0043] 3, the loop in which a comparison process is performed for one image feature with a plurality of sentence features is repeated a number of times, but the order of the loops may be reversed. That is, the same results can be obtained by repeating the loop in which a comparison process is performed for one sentence feature with a plurality of image features a number of times.

[0044] Furthermore, although the processing of the feature comparison unit 1400 is described as using DNN model #3, this part does not necessarily have to be a DNN model depending on the structure of DNN model #0. For example, the feature comparison unit 1400 may be a processing unit that calculates the cosine similarity between image features and sentence features, or a processing unit that calculates the Euclidean distance between image features and sentence features. However, this is premised on the fact that the DNN model #0 has been sufficiently trained using the above-mentioned processing units when it was trained in advance.

[0045] For the sake of explanation, the result display unit 3000 and the text feature extraction device 2000 are shown as separate entities in FIG. 1, but the text feature extraction device 2000 may also function as the result display unit 3000 .

[0046] As described above, according to the first embodiment, it is possible to provide an image recognition system 10 including an image recognition device 1000 and a sentence feature extraction device 2000, which detects a detection target from image data of a recognition target. The sentence feature extraction device 2000 includes a sentence acquisition unit 2100 that acquires natural language phrases for specifying the detection target, a sentence feature extraction unit 2200 that extracts sentence features, which are features comparable to image features, from the natural language phrases, and a sentence feature transmission unit 2300 that transmits the sentence features. The image recognition device 1000 includes a sentence feature reception unit 1300 that receives the sentence features from the sentence feature extraction device 2000, an image acquisition unit 1100 that acquires image data, an image analysis unit 1200 that extracts, from the image data, candidate areas that are candidates for areas in the image where the detection target exists and image features, which are feature quantities of the image within the candidate areas, a feature comparison unit 1400 that compares the sentence features with the image features corresponding to the candidate areas and outputs the comparison results, and a detection result generation unit 1500 that generates a detection result for the detection target based on the comparison results. The image recognition system 10 is characterized in that the processing of the sentence feature extraction unit 2200 is executed by a sentence feature extraction device 2000, which is a device separate from the image recognition device 1000. This allows the high-load processing of extracting sentence features from phrases in natural language to be executed by the sentence feature extraction device 2000, which is a device separate from the image recognition device 1000. Therefore, even if the calculation capability of the image recognition device 1000 is limited, it becomes possible to implement an inference process that uses a trained model to detect from image data a detection target specified by a phrase input in natural language.

[0047] The image analysis unit 1200 is realized by executing a DNN model #1, which is a part of a DNN model #0, which is a trained model for detecting a detection target specified by a natural language phrase from image data, and is a first partial model whose output does not depend on natural language input. The DNN model #0 is, for example, generative AI (artificial intelligence). Generative AI that inputs multimodal data such as image data and natural language phrases requires a very large amount of calculation and requires very high computing power. Therefore, when running on an embedded device such as a surveillance camera, the computing power may be insufficient, making implementation difficult. By using the technology of this embodiment, even if an embedded device such as a surveillance camera has insufficient computing power and is difficult to process multimodal data collectively, it is possible to execute some of the processing with a high processing load on an external device.

[0048] Furthermore, in the image recognition system 10, the sentence feature extraction unit 2200 may be realized by executing DNN model #2, which is a part of DNN model #0, which is a trained model for detecting a detection target specified by a phrase in natural language from image data, and is a second partial model whose output does not depend on the input of image data.

[0049] Furthermore, in the image recognition system 10, the feature comparison unit 1400 may be realized by executing DNN model #3, which is a third partial model that is part of DNN model #0, which is a trained model for detecting a detection target specified by a phrase in natural language from image data, and that accepts as input the output of DNN model #1, which is a first partial model that does not depend on natural language input, and the output of DNN model #2, which is a second partial model that does not depend on image data input, and outputs the comparison results between sentence features and image features.

[0050] According to the first embodiment, it is also possible to provide an image recognition device 1000 that detects a detection target from image data of a recognition target. The image recognition device 1000 is characterized by including a sentence feature receiving unit 1300 that receives, from outside the image recognition device 1000, sentence features that are comparable to image features and that are extracted from natural language phrases for specifying the detection target, an image acquisition unit 1100 that acquires image data, an image analysis unit 1200 that extracts, from the image data, candidate areas that are candidates for areas in the image where the detection target exists and image features that are feature amounts of the image in the candidate areas, a feature comparison unit 1400 that compares the sentence features with the image features corresponding to the candidate areas and outputs the comparison results, and a detection result generation unit 1500 that generates a detection result for the detection target based on the comparison results.

[0051] Furthermore, in the image recognition device 1000, the image analysis unit 1200 may be realized by executing a DNN model #1, which is a first partial model that is part of a DNN model #0 that is a trained model for detecting a detection target specified by a phrase in natural language from image data and whose output does not depend on natural language input. Furthermore, in the image recognition device 1000, the feature comparison unit 1400 may be realized by executing a DNN model #3, which is a third partial model that is part of the trained model DNN model #0 and that receives as input the output of the DNN model #1, which is the first partial model, and the output of a DNN model #2, which is a second partial model that is part of the trained model DNN model #0 and does not depend on image data input, and outputs a comparison result.

[0052] Moreover, according to the first embodiment, it is also possible to provide a sentence feature extraction device 2000. The sentence feature extraction device 2000 is characterized by including a sentence acquisition unit 2100 that acquires natural language phrases for specifying a detection target to be detected from image data, a sentence feature extraction unit 2200 that extracts sentence features, which are feature amounts comparable to image feature amounts, from the natural language phrases, and a sentence feature transmission unit 2300 that transmits the sentence features to the image recognition device 1000 that detects the detection target from image data.

[0053] In the sentence feature extraction device 2000, the sentence feature extraction unit 2200 is part of DNN model #0, which is a trained model for detecting detection targets specified by natural language phrases from image data, and can be realized by executing DNN model #2, which is a second partial model whose output does not depend on the input of image data.

[0054] Furthermore, although the hardware configuration will be described later, according to the first embodiment, it is also possible to provide a program for causing a computer to function as the image recognition device 1000 to detect a detection target from image data of a recognition target. This program can cause the computer functioning as the image recognition device 1000 to execute the following steps: receiving sentence features, which are features comparable to image features, extracted from natural language phrases for specifying the detection target in a computer different from the computer; acquiring image data; extracting, from the image data, candidate areas that are candidates for areas in the image where the detection target exists and image features that are feature amounts of the image in the candidate areas; comparing the sentence features with the image features corresponding to the candidate areas and outputting the comparison results; and generating a detection result for the detection target based on the comparison results.

[0055] Furthermore, according to the first embodiment, it is also possible to provide a program for causing a computer to function as the sentence feature extraction device 2000. This program can cause the computer functioning as the sentence feature extraction device 2000 to execute the following steps: acquiring natural language phrases for specifying a detection target to be detected from image data; extracting sentence features, which are features comparable to image features, from the natural language phrases; and transmitting the sentence features to a computer that is different from the computer functioning as the sentence feature extraction device 2000 and functions as the image recognition device 1000 for detecting the detection target from image data.

[0056] Furthermore, according to the first embodiment, an image recognition method for detecting a target object from image data of a target object can be provided. This image recognition method includes the steps of: extracting sentence features, which are features comparable to image features, from natural language phrases specifying the target object; extracting candidate areas, which are candidates for areas in the image where the target object exists, from the image data, and image features, which are features of the image within the candidate areas; comparing the sentence features with the image features corresponding to the candidate areas to generate a comparison result; and generating a target object detection result based on the comparison result. This image recognition method is characterized in that the steps of extracting sentence features and extracting image features are performed by different devices. For example, the image recognition method can be executed by an image recognition system 10 including an image recognition device 1000 and a sentence feature extraction device 2000, in which the sentence feature extraction device 2000 performs the step of extracting sentence features, and the image recognition device 1000 performs the steps of extracting candidate areas and image features, generating a comparison result, and generating a detection result. The image recognition method may further include a step of acquiring natural language phrases, a step of transmitting extracted sentence features, and a step of receiving sentence features. In this case, the sentence feature extraction device 2000 executes the step of acquiring natural language phrases and the step of transmitting extracted sentence features, and the image recognition device 1000 executes the step of receiving sentence features and the step of generating a comparison result using the received sentence features. In the above-described image recognition method, the processing of each described step does not necessarily have to be performed in the order described. The image recognition method may include multiple steps performed simultaneously in parallel. In addition, in the image recognition method, the processing of each described step does not necessarily have to be a series of processes performed consecutively, and may include multiple processes performed at different times.

[0057] Second Embodiment Fig. 4 is a configuration diagram of an image recognition system 20 according to a second embodiment. The image recognition system 20 has an image recognition device 1050, a sentence feature extraction device 2000, and a result display unit 3000. The image recognition system 20 has an image recognition device 1050 instead of the image recognition device 1000 of the image recognition system 10. Below, differences from the first embodiment will be mainly described, and detailed description of the same parts as in the first embodiment will be omitted.

[0058] The image recognition system 20 executes the process of extracting sentence features only when there is a change or addition to the object category to be detected.

[0059] The image recognition device 1050 includes an image acquisition unit 1100, an image analysis unit 1200, a sentence feature receiving unit 1300, a feature comparison unit 1400, a detection result generation unit 1500, and a detection result transmission unit 1600. In addition to the components of the image recognition device 1000, the image recognition device 1050 includes a sentence feature storage unit 1350.

[0060] The sentence feature storage unit 1350 stores the sentence features that the sentence feature receiving unit 1300 receives from the sentence feature extraction device 2000. The sentence feature storage unit 1350 is capable of storing a plurality of sentence features and is capable of outputting the stored sentence features to the feature comparison unit 1400.

[0061] Fig. 5 is a flowchart for explaining an example of the operation of the image recognition system 20 shown in Fig. 4. Fig. 5 is the flowchart of Fig. 3 to which step S2330 has been added. Since the steps other than step S2330 are the same as those in Fig. 3, detailed explanation of each step will be omitted, and only the operations that differ from those in Fig. 3 will be explained below.

[0062] In the image recognition system 20, the series of steps from S2210 to S2250, which are the process of extracting sentence features, are not necessarily executed in parallel with the image feature extraction process, but are basically executed independently at any timing earlier than the image feature extraction process. After the sentence feature sending unit 2300 sends the sentence features in step S2260, the sentence feature receiving unit 1300 stores the received sentence features in the sentence feature storage unit 1350 (step S2330).

[0063] After the text features are stored in the text feature storage unit 1350, the feature comparison unit 1400 executes a process of comparing the text features with the image features in step S2150. The subsequent process is the same as that shown in FIG.

[0064] Furthermore, when the sentence acquisition process of step S2210 occurs at any timing, the sentence feature extraction device 2000 executes the processes of steps S2220 to S2260 again, and the sentence feature receiving unit 1300 updates the contents of the sentence feature storage unit 1350.

[0065] As described above, in the image recognition system 20 according to the second embodiment, the image recognition device 1050 further includes a sentence feature storage unit 1350 that stores sentence features received by the sentence feature receiving unit 1300, and the feature comparison unit 1400 can compare the sentence features stored in the sentence feature storage unit 1350 with image features. This configuration makes it possible to extract sentence features and store them in the image recognition device 1050 before acquiring image data. This eliminates the need to extract sentence features when there is no change in the detection target, and the feature comparison unit 1400 can perform feature amount comparison processing using sentence features previously stored in the sentence feature storage unit 1350 without waiting for the reception of sentence features. This improves the throughput of the image recognition device 1050 and reduces the calculation load on the image recognition system 20 as a whole.

[0066] In this case, in the sentence feature extraction device 2000, each time the sentence acquisition unit 2100 acquires a new natural language phrase, the sentence feature extraction unit 2200 extracts sentence features from the new natural language phrase, and the sentence feature transmission unit 2300 transmits the extracted sentence features to the image recognition device 1050. Acquisition of a new natural language phrase does not need to be performed, for example, each time image recognition processing is performed. After the detection target is initially set, image recognition is basically performed using the sentence features stored in the sentence feature storage unit 1350 until the detection target is changed, i.e., until an object category set for the detection target is added or an object category set for the detection target is deleted. When an object category is added, the sentence acquisition unit 2100 acquires a new natural language phrase, and the sentence feature extraction unit 2200 extracts new sentence features. Typically, when performing inference processing using a DNN model with natural language phrases and image data as input, the entire DNN model needs to be operated each time an image is acquired. In this case, unless the detection target is changed, there is actually no change in the processing content or results of the sentence feature extraction unit 2200, resulting in poor computational efficiency. In contrast, according to the technology of the second embodiment, unless the detection target is changed, the sentence feature extraction process is omitted, thereby improving the throughput of the image recognition device 1050 and reducing the computational load on the image recognition system 20 as a whole.

[0067] Here, we will explain the hardware configuration for realizing the functions of the image recognition systems 10 and 20. The text acquisition unit 2100, text feature extraction unit 2200, image acquisition unit 1100, image analysis unit 1200, feature comparison unit 1400, and detection result generation unit 1500 are realized by processing circuits. These processing circuits may be realized by dedicated hardware, or may be control circuits using a CPU (Central Processing Unit).

[0068] When the above processing circuits are realized by dedicated hardware, they are realized by a processing circuit 90 shown in Fig. 6. Fig. 6 is a diagram showing dedicated hardware for realizing the functions of the image recognition systems 10 and 20 according to the first and second embodiments. The processing circuit 90 is a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof.

[0069] When the processing circuit is realized by a control circuit using a CPU, the control circuit is, for example, a control circuit 91 having a configuration shown in FIG. 7 . FIG. 7 is a diagram showing the configuration of a control circuit for realizing the functions of the image recognition systems 10 and 20 according to the first and second embodiments. As shown in FIG. 7 , the control circuit 91 includes a processor 92 and a memory 93. The processor 92 is a CPU, and is also called a processing device, an arithmetic unit, a microprocessor, a microcomputer, a DSP (Digital Signal Processor), etc. The memory 93 is, for example, a non-volatile or volatile semiconductor memory such as a random access memory (RAM), a read-only memory (ROM), a flash memory, an erasable programmable read-only memory (EPROM), an electrically programmable programmable read-only memory (EEPROM), a magnetic disk, a flexible disk, an optical disk, a compact disk, a minidisk, or a digital versatile disk (DVD).

[0070] When the above processing circuit is realized by the control circuit 91, it is realized by the processor 92 reading and executing a program corresponding to the processing of each component, which is stored in the memory 93. The memory 93 is also used as a temporary memory for each process executed by the processor 92. The program executed by the processor 92 may be provided in a state stored in a storage medium, or may be provided via a communication path such as the Internet.

[0071] The functions of the sentence feature receiving unit 1300, the detection result transmitting unit 1600, and the sentence feature transmitting unit 2300 can be realized using a communication device (not shown). The functions of the image acquiring unit 1100 can be realized using an image sensor or an imaging device, as described above. The sentence feature storage unit 1350 can be realized using a storage device.

[0072] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention.

[0073] For example, for simplicity, Figures 1 and 4 show only one image recognition device 1000, 1050 and one sentence feature extraction device 2000, but the image recognition systems 10, 20 may include multiple image recognition devices 1000, 1050. The sentence feature extraction device 2000 can transmit sentence features to multiple image recognition devices 1000, 1050. Furthermore, the image recognition systems 10, 20 may include multiple sentence feature extraction devices 2000 depending on the required computing power.

[0074] In the above embodiment, the image analysis unit 1200 of the image recognition device 1000 or 1050 extracts multiple candidate regions. However, there may be cases where only one candidate region can be extracted from an image. In this case, the loop process from step S2130 to step S2190 in Figures 3 and 5 is executed only once.

[0075] In the above embodiment, the image recognition devices 1000 and 1050 receive the sentence features directly from the sentence feature extraction device 2000, but the present invention is not limited to this example. It is sufficient that the sentence feature extraction process is performed by a device other than the image recognition devices 1000 and 1050. For example, the sentence features extracted by the sentence feature extraction device 2000 may be stored in a computer other than the sentence feature extraction device 2000, and the image recognition devices 1000 and 1050 may receive the sentence features from the computer that stores the sentence features.

[0076] 10, 20 Image recognition system, 90 Processing circuit, 91 Control circuit, 92 Processor, 93 Memory, 1000, 1050 Image recognition device, 1100 Image acquisition unit, 1200 Image analysis unit, 1300 Text feature receiving unit, 1350 Text feature storage unit, 1400 Feature comparison unit, 1500 Detection result generation unit, 1600 Detection result transmission unit, 2000 Text feature extraction device, 2100 Text acquisition unit, 2200 Text feature extraction unit, 2300 Text feature transmission unit, 3000 Result display unit.

Claims

1. An image recognition system that detects a detection target from image data of a recognition target, comprising: a sentence feature extraction device; and an image recognition device, wherein the sentence feature extraction device has: a sentence acquisition unit that acquires natural language phrases used to specify the detection target; a sentence feature extraction unit that extracts sentence features, which are features comparable to image features, from the natural language phrases; and a sentence feature transmission unit that transmits the sentence features, wherein the image recognition device has: a sentence feature reception unit that receives the sentence features from the sentence feature extraction device; an image acquisition unit that acquires the image data; an image analysis unit that extracts, from the image data, candidate areas that are candidates for areas in the image where the detection target exists, and image features, which are features of the image within the candidate areas, a feature comparison unit that compares the sentence features with the image features corresponding to the candidate areas and outputs a comparison result, and a detection result generation unit that generates a detection result for the detection target based on the comparison result, wherein the processing of the sentence feature extraction unit is executed by the sentence feature extraction device, which is a device separate from the image recognition device.

2. The image recognition system described in claim 1, characterized in that the image analysis unit executes a first partial model that is part of a trained model for detecting the detection target specified by a phrase in the natural language from the image data, and whose output does not depend on the natural language input.

3. The image recognition system according to claim 1 or 2, characterized in that the sentence feature extraction unit executes a second partial model that is part of a trained model for detecting the detection target specified by a phrase in the natural language from the image data, and whose output does not depend on the input of image data.

4. The image recognition system of claim 1, wherein the feature comparison unit is part of a trained model for detecting the detection target specified by a phrase in the natural language from the image data, and executes a third partial model that accepts as input the output of a first partial model that does not depend on natural language input and the output of a second partial model that does not depend on image data input, and outputs the comparison result.

5. The image recognition system according to any one of claims 1 to 4, characterized in that the image recognition device further has a sentence feature storage unit that stores the sentence features received by the sentence feature receiving unit, and the feature comparison unit compares the sentence features stored in the sentence feature storage unit with the image features.

6. The image recognition system according to claim 5, characterized in that the sentence feature extraction device extracts sentence features from the new natural language phrase by the sentence feature extraction unit each time the sentence acquisition unit acquires a new natural language phrase, and transmits the extracted sentence features by the sentence feature transmission unit.

7. An image recognition system that detects a detection target from image data of a recognition target, comprising: an image recognition device; and a text feature extraction device having a higher processing power than said image recognition device, wherein said text feature extraction device has: a text acquisition unit that acquires natural language phrases for specifying said detection target, a text feature extraction unit that extracts text features, which are features comparable to image features, from said natural language phrases, and a text feature transmission unit that transmits said text features, wherein said image recognition device has: a text feature reception unit that receives the text features from said text feature extraction device, an image acquisition unit that acquires the image data, an image analysis unit that extracts, from the image data, candidate areas that are candidates for areas in the image where said detection target exists, and image features, which are features of images in said candidate areas, a feature comparison unit that compares the text features with the image features corresponding to the candidate areas and outputs a comparison result, and a detection result generation unit that generates a detection result for said detection target based on the comparison result, wherein the processing of said text feature extraction unit is executed by said text feature extraction device, which is a device separate from said image recognition device.

8. An image recognition device that detects a detection target from image data of a recognition target, comprising: a sentence feature receiving unit that receives sentence features extracted from natural language phrases that specify the detection target and are comparable to image features from outside the image recognition device; an image acquisition unit that acquires the image data; an image analysis unit that extracts candidate areas from the image data that are candidates for areas in the image where the detection target exists, and image features that are feature amounts of the image within the candidate areas; a feature comparison unit that compares the sentence features with the image features corresponding to the candidate areas and outputs a comparison result; and a detection result generation unit that generates a detection result for the detection target based on the comparison result.

9. The image recognition device according to claim 8, characterized in that the image analysis unit executes a first partial model, which is part of a trained model for detecting the detection target specified by a phrase in the natural language from the image data, and whose output does not depend on the input of the natural language.

10. The image recognition device described in claim 9, characterized in that the feature comparison unit executes a third partial model that is part of the trained model, accepts as input the output of the first partial model and the output of a second partial model that is part of the trained model and does not depend on the input of image data, and outputs the comparison result.

11. An image recognition device as described in any one of claims 8 to 10, further comprising a sentence feature storage unit that stores the sentence features received by the sentence feature receiving unit, wherein the feature comparison unit compares the sentence features stored in the sentence feature storage unit with the image features.

12. A text feature extraction device comprising: a text acquisition unit that acquires natural language phrases for specifying a detection target to be detected from image data; a text feature extraction unit that extracts text features, which are features comparable to image features, from the natural language phrases; and a text feature transmission unit that transmits the text features to an image recognition device that detects the detection target from the image data.

13. The text feature extraction device according to claim 12, characterized in that the text feature extraction unit executes a second partial model, which is part of a trained model for detecting the detection target specified by a phrase in the natural language from the image data, and whose output does not depend on the input of image data.

14. A program for causing a computer to function as an image recognition device that detects a detection target from image data of a recognition target, the program causing the computer to function as the image recognition device to execute the following steps: receiving sentence features that are comparable to image features and that are extracted from natural language phrases for specifying the detection target in a computer different from the computer; acquiring the image data; extracting from the image data candidate areas that are candidates for areas within the image where the detection target exists, and image features that are feature amounts of the image within the candidate areas; comparing the sentence features with the image features corresponding to the candidate areas and outputting the comparison results; and generating a detection result for the detection target based on the comparison results.

15. A program for causing a computer to function as a text feature extraction device, the program causing the computer functioning as the text feature extraction device to execute the following steps: acquiring natural language phrases for specifying a detection target to be detected from image data; extracting text features from the natural language phrases, which are features comparable to image features; and transmitting the text features to a computer other than the computer functioning as the text feature extraction device, which functions as an image recognition device for detecting the detection target from the image data.

16. An image recognition method for detecting a detection target from image data of a recognition target, comprising: a step of extracting sentence features, which are features comparable to image features, from natural language phrases used to specify the detection target; a step of extracting from the image data candidate areas, which are candidates for areas within the image where the detection target exists, and image features, which are features of the image within the candidate areas; a step of comparing the sentence features with the image features corresponding to the candidate areas to generate a comparison result; and a step of generating a detection result for the detection target based on the comparison result, wherein the processing of the step of extracting sentence features and the processing of the step of extracting image features are performed by different devices.

Images