Target detection method, target detection model and target detection device
By adopting a multimodal feature fusion method in open word list object detection, visual and text features are extracted from the image to be detected, the problem that plain text prompts are difficult to describe visual details is solved, and the accuracy and stability of object detection are improved. Especially in the query accuracy of uncommon object categories.
Patent Information
- Application Number
- CN202411179737.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-06
AI Technical Summary
In the existing open word list object detection method, it is difficult to fully describe visual details in plain text prompts, resulting in low category ambiguity and query accuracy. Especially for uncommon object categories or difficult to describe in short sentences or keywords, the difficulty of understanding of pre-trained models increases. .
Using the multimodal feature fusion method, the first image feature is extracted from the image to be detected, and the second image feature and text feature are obtained from the more representative image area, and they are fused through a linear layer to perform similarity matching to output. The target detection results enhance the importance of features and reduce noise, and improve query accuracy.
By fusing visual and text features, the limitations of plain text cues are solved, and the accuracy and stability of object detection are improved, especially query accuracy when describing uncommon objects.
Smart Images

Figure CN119091121A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer vision technology, and in particular to a target detection method, a target detection model, and a target detection device. Background Art
[0002] In the prior art, open vocabulary target detection usually uses plain text prompts to query potential targets in the target image. Due to the limitations of text descriptions, it is difficult to fully describe visual details. Plain text queries mainly have the following problems: First, it is difficult to fully describe visual details with text descriptions. For example, if some strange biological species appear in the image, it is difficult to describe what species they are, and writing text descriptions for a large number of categories is a tedious task; second, category ambiguity, the same text may refer to multiple objects, and there may be multiple meanings of a word; third, some query targets are difficult to accurately describe with a short sentence or keyword, and using longer query texts will increase the difficulty of understanding for the pre-trained model. Summary of the invention
[0003] The present disclosure provides a target detection method, a target detection model, a target detection device, an electronic device, and a storage medium.
[0004] According to one aspect of the present disclosure, there is provided a target detection method, comprising:
[0005] Acquire an image to be detected, and extract a first image feature from the image to be detected;
[0006] Acquire a prompt image from the image to be detected, and extract a second image feature from the prompt image;
[0007] Obtaining a prompt text, and extracting text features from the prompt text;
[0008] Fusing the text feature with the second image feature to obtain a multimodal feature;
[0009] Similarity matching is performed on the multimodal features and the first image features, and a corresponding target detection result is output according to the similarity matching result.
[0010] According to another aspect of the present disclosure, a target detection model is provided, comprising:
[0011] A first image feature extraction model is configured to obtain an image to be detected and extract a first image feature from the image to be detected;
[0012] A second image feature extraction model is configured to extract a second image feature from a prompt image; wherein the prompt image is obtained from the image to be detected;
[0013] A text feature extraction model is configured to obtain a prompt text and extract text features from the prompt text;
[0014] A linear layer configured to fuse the text feature with the second image feature to obtain a multimodal feature;
[0015] The output layer is configured to perform similarity matching on the multimodal features and the first image features, and output corresponding target detection results according to the similarity matching results.
[0016] According to a third aspect of the present disclosure, there is provided a target detection device, comprising:
[0017] A first image feature extraction module is configured to obtain an image to be detected and extract a first image feature from the image to be detected;
[0018] A second image feature extraction module is configured to obtain a prompt image from the image to be detected and extract a second image feature from the prompt image;
[0019] A text feature extraction module is configured to obtain a prompt text and extract text features from the prompt text;
[0020] A feature fusion module, configured to fuse the text feature with the second image feature to obtain a multimodal feature;
[0021] The output module is configured to perform similarity matching on the multimodal features and the first image features, and output corresponding target detection results according to the similarity matching results.
[0022] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:
[0023] at least one processor; and
[0024] a memory communicatively connected to the at least one processor; wherein,
[0025] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method in any of the above technical solutions.
[0026] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods described in the above technical solutions.
[0027] According to a sixth aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements any one of the methods described in the above technical solutions when executed by a processor.
[0028] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0030] Figure 1 is a schematic diagram of the steps of the target detection method in an embodiment of the present disclosure;
[0031] Figure 2 is a schematic diagram of the structure of the target detection model in the embodiment of the present disclosure;
[0032] Figure 3 is a principle block diagram of a target detection device in an embodiment of the present disclosure;
[0033] Figure 4 It is a block diagram of an electronic device used to implement the target detection method of the embodiment of the present disclosure. DETAILED DESCRIPTION
[0034] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0035] The current open vocabulary object detection adopts a pure text prompt solution. Specifically, taking the APE model as an example, the APE model includes a language model and an image extraction model, which can obtain the input prompt text prompt, such as "girl" and "sky", and the input image to be detected at the same time, and detect the corresponding target object from the image to be detected through the prompt text "girl" and "sky". This method can still accurately describe conventional scenes with words, but for complex target objects or some less common object categories, it is difficult to describe them with words, or it cannot be described with shorter keywords or sentences.
[0036] In view of the above technical problems, the present disclosure provides a target detection method, such as Figure 1 As shown, including:
[0037] Step S101, obtaining an image to be detected, and extracting a first image feature from the image to be detected.
[0038] Specifically, the target detection method refers to detecting the corresponding target object from the image to be detected according to the user's prompt. Figure 2 As shown, in this embodiment, the APE-VIT (Vision Transformer) model can be used to extract visual features of the original model to be detected. The VIT model divides the input image into multiple image blocks (patches), and then projects each patch into a vector of fixed length and sends it to the Transformer to obtain the first image feature. The present disclosure is not limited to the APE model, and grounding-dino, GLIP (Grounded Language-Image Pretraining, multimodal language image) models, etc. can also be used.
[0039] Step S102: obtaining a prompt image from the image to be detected, and extracting a second image feature from the prompt image.
[0040] like Figure 2 As shown, after obtaining the image to be detected, a more representative image region can be extracted from the image to be detected, and the image region can be used as visual cue information. Subsequently, visual features can be extracted from the cue image through another CLIP-VIT model, and the extracted visual features can be used as second image features. It should be noted that the weights of the VIT model used to extract the first image feature and the second image feature are different.
[0041] Step S103, obtaining a prompt text, and extracting text features from the prompt text.
[0042] like Figure 2 As shown, while obtaining visual prompt information, a language model can be used to extract text features in the prompt text.
[0043] Step S104: fusing the text feature with the second image feature to obtain a multimodal feature.
[0044] Specifically, feature fusion can be to add the text feature to the second image feature, which can strengthen important features and reduce noise.
[0045] Step S105, performing similarity matching on the multimodal features and the first image features, and outputting corresponding target detection results according to the similarity matching results.
[0046] The present disclosure adds visual prompts on the basis of pure text prompts, and integrates visual feature information into text feature information. This multimodal query method solves the limitation of using only pure text prompts. In addition, the second image feature in the present disclosure comes from the image to be detected, which is a more representative image area cut out from the image to be detected, such as the image area where the target object is located, which can mine the visual information of the target object in a targeted manner and further improve the query accuracy.
[0047] As an optional implementation, step S105, performing similarity matching on the multimodal feature and the first image feature, and outputting a corresponding target detection result according to the similarity matching result includes:
[0048] Step S105a: generating a corresponding candidate box according to the first image feature.
[0049] Since the first image feature is extracted based on the entire image to be detected, when performing similarity matching on the features, the candidate box can be used to limit the area where the target object is located, so as to facilitate subsequent similarity matching with multimodal features based on the image features and position of the area. Among them, the candidate box can be used to filter out the image area where the target object is located, which helps to narrow the scope of feature matching and improve matching efficiency. The candidate box can also clarify the approximate position of the target object in the image and locate the target object.
[0050] Step S105b, performing similarity matching on the multimodal features and the candidate boxes to obtain a similarity matching result, and outputting a corresponding target detection result based on the similarity matching result.
[0051] like Figure 2 As shown, the first image feature extraction model can be used to divide the image to be detected into multiple small image blocks, and then input into the Encoder module of the Transformer structure for encoding, and finally multiple encoding vectors, i.e., original image features, are obtained. Next, the original image features are input into the DETR (Detection Transformer, an end-to-end target detection model) model, and the candidate boxes of the target object are output, each of which corresponds to a target object in the image. These candidate boxes not only contain the location information of the target (such as coordinate information), but also contain feature information related to it (i.e., regionbox feature). The main function of the candidate box is to locate the target object in the image, match the first image feature corresponding to the target object with the multimodal feature, and thus determine the category of the target object.
[0052] As an optional implementation, step S102, obtaining a prompt image from the image to be detected includes:
[0053] The image area corresponding to each instance is cropped from the image to be detected as a prompt image.
[0054] Specifically, for example, if the image to be detected includes multiple instances such as "girl", "sky", and "branch", the image area where "girl", "sky", and "branch" are located can be cropped out respectively, and small images of the corresponding areas can be cropped out according to these instances as prompt images. Each prompt image corresponds to a specific category, and the details of the visual feature description are more accurate.
[0055] As an optional implementation, after extracting the second image feature from the prompt image in step S102, the method further includes:
[0056] The second image feature is randomly updated with a preset probability.
[0057] like Figure 2 As shown in the figure, when training the model, each instance is cropped into small images as prompt images according to the number of training categories, and features are extracted through the pre-trained CLIP-VIT model. The feature vector of each category is randomly updated with a preset probability (for example, 0.5) to avoid always using the same source small image as an instance. The specific update process can be expressed as:
[0058] f_new=(1-r)*f_old+r*f_current
[0059] Among them, f_new is the updated second image feature; f_old is the historical second image feature; f_current is the current second image feature; and r is the preset probability.
[0060] As an optional implementation, step S102, fusing the text feature with the second image feature to obtain a multimodal feature includes:
[0061] The text features and the second image features are projected into the same feature space, and then the text features and the second image features are linearly added to obtain multimodal features.
[0062] like Figure 2 As shown in the figure, the second image feature extracted by the CLIP-VIT model and the text feature extracted by the CLIP-text model are fused in the linear layer (adapter linear), that is, the two feature vectors are added to obtain a new multimodal feature. The feature addition method can strengthen important features and reduce noise, which helps to improve the stability and generalization ability of the network.
[0063] As an optional implementation, step S105, performing similarity matching on the multimodal feature and the first image feature, and outputting a corresponding target detection result according to the similarity matching result includes:
[0064] The multimodal features are concatenated with the first image features, and a similarity matrix between the multimodal features and the first image features is generated as a similarity matching result.
[0065] Output the corresponding target detection results according to the similarity matrix.
[0066] Specifically, the feature splicing is characterized by being able to retain all the information of the two feature maps. The first image feature is the full-image feature extracted from the original image to be detected, while the visual feature in the multimodal feature is the instance image of the corresponding category extracted from the image to be detected. For a specific target object, the multimodal feature can be described more accurately. The multimodal feature can be compared with the first image feature to obtain the category with the highest feature similarity as the category of the target object, and the category of the target object and its position in the image to be detected are output as the target detection result.
[0067] The present disclosure also provides a target detection model, such as Figure 2 As shown, including:
[0068] The first image feature extraction model 201 is configured to obtain an image A to be detected and extract a first image feature from the image to be detected.
[0069] Specifically, the target detection method refers to detecting the corresponding target object from the image to be detected according to the user's prompt. Figure 2 As shown, in this embodiment, the APE-VIT model can be used to extract visual features from the original model to be detected. The VIT model divides the input image into multiple image blocks (patches), and then projects each patch into a vector of fixed length and sends it to the Transformer to obtain the first image feature. The present disclosure is not limited to the APE model, and the grounding-dino, GLIP model, etc. can also be used.
[0070] The second image feature extraction model 202 is configured to extract the second image feature from the prompt image B; wherein the prompt image is obtained from the image to be detected.
[0071] like Figure 2As shown, after obtaining the image to be detected, a more representative image region can be extracted from the image to be detected, and the image region can be used as visual cue information. Subsequently, a CLIP-VIT model can be used to extract visual features from the cue image, and the extracted visual features can be used as second image features. It should be noted that the weights of the VIT model used to extract the first image feature and the second image feature are different.
[0072] The text feature extraction model 203 is configured to obtain the prompt text C and extract text features from the prompt text.
[0073] like Figure 2 As shown, while obtaining visual prompt information, a language model (CLIP-text model) can be used to extract text features in the prompt text. The second image feature extraction model 202 and the text feature extraction model 203 in this embodiment jointly construct a CLIP (Contrastive Language-Image Pre-Training, multimodal pre-training neural network) model.
[0074] The linear layer 204 is configured to fuse the text feature with the second image feature to obtain a multimodal feature.
[0075] Specifically, feature fusion can be to add the text feature to the second image feature, which can strengthen important features and reduce noise.
[0076] The output layer 205 is configured to perform similarity matching on the multimodal features and the first image features, and output corresponding target detection results according to the similarity matching results.
[0077] Based on the pure text prompt, the present disclosure further improves the multimodal query method, integrates visual feature information into the text feature information, and solves the limitation of using only pure text prompts. In addition, the second image feature in the present disclosure comes from the image to be detected, which is a more representative image area cut out from the image to be detected, such as the image area where the target object is located, which can mine the visual information of the target object in a targeted manner and further improve the query accuracy.
[0078] As an optional implementation, Figure 2 As shown, it also includes:
[0079] The target detection sub-model 206 is configured as follows: after the first image feature extraction model 201 extracts the first image feature from the image to be detected, a corresponding candidate box is generated according to the first image feature; the candidate box is provided to the output layer 205, and similarity matching is performed on the multimodal feature and the candidate box to obtain a similarity matching result, and the corresponding target detection result is output according to the similarity matching result.
[0080] Specifically, the first image feature extraction model 201 can use the APE-VIT model to extract features from the image to be detected. The target detection sub-model 206 can use the DETR model to generate corresponding candidate boxes based on the first image features extracted by the APE-VIT model. Each candidate box corresponds to a potential target in the image. These candidate boxes not only contain the location information of the target (such as coordinate information), but also contain feature information related to it (i.e., region box feature). The main function of the candidate box is to locate the target object in the image, match the first image feature corresponding to the target object with the multimodal feature, and thus determine the category of the target object.
[0081] As an optional implementation, Figure 2 As shown, the second image feature extraction model 202 obtains the prompt image obtained by cropping according to the instance through the cropping module, thereby extracting the second image feature.
[0082] As an optional implementation, Figure 2 As shown, it also includes:
[0083] The updating layer 207 is configured to randomly update the second image feature with a preset probability after the second image feature extraction model extracts the second image feature from the prompt image.
[0084] The update layer 207 uses Figure 2 As shown, when training the model, each instance is cropped into small images as prompt images according to the number of training categories, features are extracted through the pre-trained CLIP-VIT model, and the feature vector of each category is randomly updated with a preset probability (for example, 0.5) to avoid always using the same source small images as instances.
[0085] The present disclosure also provides a target detection device 300, such as Figure 3 As shown, including:
[0086] The first image feature extraction module 301 is configured to obtain an image to be detected and extract a first image feature from the image to be detected.
[0087] Specifically, Figure 2 As shown, in this embodiment, the APE-VIT model can be used to extract visual features from the original model to be detected. The VIT model divides the input image into multiple image blocks (patches), and then projects each patch into a vector of fixed length and sends it to the Transformer to obtain the first image feature. The present disclosure is not limited to the APE model, and the grounding-dino, GLIP model, etc. can also be used.
[0088] The second image feature extraction module 302 is configured to obtain a prompt image from the image to be detected, and extract the second image feature from the prompt image.
[0089] like Figure 2 As shown, after obtaining the image to be detected, a more representative image region can be extracted from the image to be detected, and the image region can be used as visual cue information. Subsequently, visual features can be extracted from the cue image through another CLIP-VIT model, and the extracted visual features can be used as second image features. It should be noted that the weights of the VIT model used to extract the first image feature and the second image feature are different.
[0090] The text feature extraction module 303 is configured to obtain the prompt text and extract text features from the prompt text.
[0091] like Figure 2 As shown, while obtaining visual prompt information, a language model can be used to extract text features in the prompt text.
[0092] The feature fusion module 304 is configured to fuse the text feature with the second image feature to obtain a multimodal feature.
[0093] Specifically, feature fusion can be to add the text feature to the second image feature, which can strengthen important features and reduce noise.
[0094] The output module 305 is configured to perform similarity matching on the multimodal feature and the first image feature, and output a corresponding target detection result according to the similarity matching result.
[0095] Based on the pure text prompt, the present disclosure further improves the multimodal query method, integrates the visual feature information into the text feature information, and solves the limitation of using only pure text prompts. In addition, the second image feature in the present disclosure comes from the image to be detected, which is a more representative image area cut out from the image to be detected, and can mine the visual information of the target object in a targeted manner, further improving the query accuracy.
[0096] As an optional implementation, the output module 305 includes:
[0097] The candidate box generating unit is configured to generate a corresponding candidate box according to the first image feature.
[0098] Since the first image feature is extracted based on the entire image to be detected, when performing similarity matching on the features, the candidate box can be used to limit the area where the target object is located, so as to facilitate subsequent similarity matching with multimodal features based on the image features and position of the area. Among them, the candidate box can be used to filter out the image area where the target object is located, which helps to narrow the scope of feature matching and improve matching efficiency. The candidate box can also clarify the approximate position of the target object in the image and locate the target object.
[0099] The matching unit is configured to perform similarity matching on the multimodal features and the candidate boxes to obtain a similarity matching result, and output a corresponding target detection result according to the similarity matching result.
[0100] Specifically, the first image feature extraction module in this embodiment adopts an APE model structure, including an APE-VIT model and an APE-DETR model, wherein the first image feature extraction unit can adopt the APE-VIT model, and the candidate frame generation unit can adopt the APE-DETR model. Figure 2 As shown, the first image feature extraction model can be used to divide the image to be detected into multiple small image blocks, and then input into the Encoder module of the Transformer structure for encoding, and finally multiple encoding vectors, i.e., original image features, are obtained. Next, the original image features are input into the DETR (Detection Transformer, an end-to-end target detection model) model, and the candidate boxes of the target objects are output, each of which corresponds to a target object in the image. These candidate boxes not only contain the location information of the target (such as coordinate information), but also contain feature information related to it (i.e., region box feature). The main function of the candidate box is to locate the target object in the image, match the first image feature corresponding to the target object with the multimodal feature, and thus determine the category of the target object.
[0101] As an optional implementation, Figure 2 As shown, the target detection device also includes:
[0102] The cropping module is configured to crop the image region corresponding to each instance from the image to be detected as the prompt image before the second image feature extraction module 302 extracts the second image feature from the prompt image.
[0103] Specifically, for example, if the image to be detected includes multiple instances such as "girl", "sky", "branch", etc., the image area where the "girl", the image area where the "sky", and the image area where the "branch" are located can be cropped out respectively, and small images of the corresponding areas can be cropped out according to these instances as prompt images. Each prompt image corresponds to a specific category, and the details of the visual feature description are more accurate. After the cropping module crops the prompt images corresponding to each instance, such as the "girl" image, the "sky" image, and the "branch" image, the second image feature extraction module 302 extracts the visual features of the prompt images of each category to obtain the second image features corresponding to each category.
[0104] As an optional implementation, the target detection device further includes:
[0105] The updating module is configured to randomly update the second image feature with a preset probability after the second image feature extraction module extracts the second image feature from the prompt image.
[0106] like Figure 2 As shown in the figure, when training the model, each instance is cropped into small images as prompt images according to the number of training categories, and features are extracted through the pre-trained CLIP-VIT model. The feature vector of each category is randomly updated with a preset probability (for example, 0.5) to avoid always using the same source small image as an instance. The specific update process can be expressed as:
[0107] f_new=(1-r)*f_old+r*f_current
[0108] Among them, f_new is the updated second image feature; f_old is the historical second image feature; f_current is the current second image feature; and r is the preset probability.
[0109] As an optional implementation, the feature fusion module 304 fuses the text feature with the second image feature to obtain a multimodal feature including:
[0110] The text features and the second image features are projected into the same feature space, and the text features and the second image features are linearly added to obtain multimodal features.
[0111] like Figure 2 As shown in the figure, the second image features extracted by the CLIP-VIT model (second image feature extraction model) and the text features extracted by the CLIP-text model (text feature extraction model) are fused in the linear layer (adapter linear), that is, the two feature vectors are added to obtain a new multimodal feature. The feature addition method can strengthen important features and reduce noise, which helps to improve the stability and generalization ability of the network.
[0112] As an optional implementation, the output module 305 outputs the corresponding target detection result according to the multimodal feature and the first image feature, including:
[0113] The multimodal features are concatenated with the first image features, and a similarity matrix between the multimodal features and the first image features is generated as a similarity matching result.
[0114] Output the corresponding target detection results according to the similarity matrix.
[0115] Specifically, the feature splicing is characterized by being able to retain all the information of the two feature maps. The first image feature is the full-image feature extracted from the original image to be detected, while the visual feature in the multimodal feature is the instance image of the corresponding category extracted from the image to be detected. For a specific target object, the multimodal feature can be described more accurately. The multimodal feature can be compared with the first image feature to obtain the category with the highest similarity as the target detection result.
[0116] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0117] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0118] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0119] like Figure 4As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0120] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0121] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as the target detection method. For example, in some embodiments, the target detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the target detection method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the target detection method in any other appropriate manner (e.g., by means of firmware).
[0122] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0123] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0124] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0126] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0127] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0128] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0129] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A target detection method, comprising: Acquire an image to be detected, and extract a first image feature from the image to be detected; Acquire a prompt image from the image to be detected, and extract a second image feature from the prompt image; Obtaining a prompt text, and extracting text features from the prompt text; Fusing the text feature with the second image feature to obtain a multimodal feature; Similarity matching is performed on the multimodal features and the first image features, and a corresponding target detection result is output according to the similarity matching result.
2. The method according to claim 1, wherein: Performing similarity matching on the multimodal feature and the first image feature, and outputting a corresponding target detection result according to the similarity matching result includes: generating a corresponding candidate frame according to the first image feature; Performing similarity matching on the multimodal feature and the candidate box to obtain the similarity matching result, and outputting the corresponding target detection result according to the similarity matching result.
3. The method according to claim 1, wherein: The acquiring of a prompt image from the image to be detected comprises: The image region corresponding to each instance is respectively cropped from the image to be detected as the prompt image.
4. The method according to claim 1 or 3, wherein: After extracting the second image feature from the prompt image, the method further includes: The second image feature is randomly updated with a preset probability.
5. The method according to claim 1, wherein: The step of fusing the text feature with the second image feature to obtain a multimodal feature comprises: The text feature and the second image feature are projected into the same feature space, and then the text feature and the second image feature are linearly added to obtain the multimodal feature.
6. The method according to any one of claims 1 to 5, wherein: Performing similarity matching on the multimodal feature and the first image feature, and outputting a corresponding target detection result according to the similarity matching result includes: splicing the multimodal feature with the first image feature, and generating a similarity matrix between the multimodal feature and the first image feature as the similarity matching result; The corresponding target detection result is output according to the similarity matrix.
7. A target detection model, comprising: A first image feature extraction model is configured to obtain an image to be detected and extract a first image feature from the image to be detected; A second image feature extraction model is configured to extract a second image feature from a prompt image; wherein the prompt image is obtained from the image to be detected; A text feature extraction model is configured to obtain a prompt text and extract text features from the prompt text; A linear layer configured to fuse the text feature with the second image feature to obtain a multimodal feature; The output layer is configured to perform similarity matching on the multimodal features and the first image features, and output corresponding target detection results according to the similarity matching results.
8. The target detection model according to claim 7, wherein: Also includes: The object detection sub-model is configured to generate a corresponding candidate box according to the first image feature; The candidate box is used to provide to the output layer, perform similarity matching on the multimodal feature and the candidate box to obtain the similarity matching result, and output the corresponding target detection result according to the similarity matching result.
9. The target detection model according to claim 7, wherein: Also includes: The update layer is configured to randomly update the second image feature with a preset probability after the second image feature extraction model extracts the second image feature from the prompt image.
10. A target detection device, comprising: A first image feature extraction module is configured to obtain an image to be detected and extract a first image feature from the image to be detected; A second image feature extraction module is configured to obtain a prompt image from the image to be detected and extract a second image feature from the prompt image; A text feature extraction module is configured to obtain a prompt text and extract text features from the prompt text; A feature fusion module, configured to fuse the text feature with the second image feature to obtain a multimodal feature; The output module is configured to perform similarity matching on the multimodal features and the first image features, and output corresponding target detection results according to the similarity matching results.
11. The device according to claim 10, wherein: The output module comprises: a candidate frame generating unit, configured to generate a corresponding candidate frame according to the first image feature; The matching unit is configured to perform similarity matching on the multimodal feature and the candidate box to obtain the similarity matching result, and output the corresponding target detection result according to the similarity matching result.
12. The device according to claim 10, wherein: Also includes: The cropping module is configured to crop the image area corresponding to each instance from the image to be detected as the prompt image before the second image feature extraction module extracts the second image feature from the prompt image.
13. The device according to claim 10 or 12, wherein: Also includes: The updating module is configured to randomly update the second image feature with a preset probability after the second image feature extraction module extracts the second image feature from the prompt image.
14. The device according to claim 10, wherein: The feature fusion module fuses the text feature with the second image feature to obtain a multimodal feature, including: The text feature and the second image feature are projected into the same feature space, and the text feature and the second image feature are linearly added to obtain the multimodal feature.
15. The device according to any one of claims 10 to 14, wherein: The output module outputs the corresponding target detection result according to the multimodal feature and the first image feature, including: splicing the multimodal feature with the first image feature, and generating a similarity matrix between the multimodal feature and the first image feature as the similarity matching result; The corresponding target detection result is output according to the similarity matrix.
16. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.
17. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.
18. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Image target detection method and device, equipment and storage medium
CN113936256A
Target detection method and device
CN117392379A
Image retrieval method and apparatus, electronic device, and storage medium
US20240168992A1
Method and apparatus for extracting multi-modal POI feature
WO2022198854A1
Cited By
Defect identification method for power system
CN120634988A
