Target detection method and device, model training method and device, electronic equipment and medium

Abstract

The invention relates to the technical field of data processing, and provides a target detection method and device, a model training method and device, electronic equipment and a medium. The target detection method comprises the steps that a to-be-recognized image and a query text are acquired, and the query text is used for querying a target object corresponding to the query text in the to-be-recognized image; performing image recognition on the to-be-recognized image to obtain image description features and region detection visual features; performing regional multi-modal fusion processing on the image description features and the regional detection visual features to obtain regional multi-modal fusion features; performing feature fusion processing on text features obtained based on the query text and the regional multi-modal fusion features to obtain text regional fusion features corresponding to the query text; and a target detection result is obtained based on the text features and the text region fusion features, so that the fusion degree of text semantics and image region features is improved, and the accuracy and robustness of target detection in a complex scene are improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a target detection method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Existing open target detection technologies typically achieve target localization based on a shallow fusion of visual features and textual information. Conventionally, they involve extracting visual features from images and semantic features from text independently and then performing simple splicing or linear transformations. This approach fails to fully explore the deep correlations between multimodal information, resulting in insufficient synergy between semantic understanding and visual localization.

[0003] It is evident that the relevant technologies suffer from insufficient multimodal information fusion, resulting in inadequate target localization accuracy and poor adaptability to complex scenarios in open target detection tasks. Summary of the Invention

[0004] In view of this, the present disclosure provides a target detection method, apparatus, electronic device and readable storage medium to solve the problems in the prior art where insufficient multimodal information fusion leads to insufficient target positioning accuracy and poor adaptability to complex scenes in open target detection tasks.

[0005] A first aspect of this disclosure provides a target detection method, comprising: acquiring an image to be identified and query text, wherein the query text is used to query a target object in the image to be identified corresponding to the query text; performing image recognition on the image to be identified to obtain image descriptive features and region detection visual features; performing region multimodal fusion processing on the image descriptive features and region detection visual features to obtain region multimodal fusion features; performing feature fusion processing on the text features obtained based on the query text and the region multimodal fusion features to obtain text region fusion features corresponding to the query text; and obtaining a target detection result based on the text features and the text region fusion features, wherein the target detection result represents the position of the target object in the image to be identified and the category of the target object.

[0006] In some embodiments, obtaining target detection results based on text features and text region fusion features includes: performing hierarchical fusion processing on the text region fusion features to obtain target fusion features corresponding to the query text; and performing multimodal decoding processing on the text features and target fusion features to obtain target detection results.

[0007] In some embodiments, performing region multimodal fusion processing on image description features and region detection visual features to obtain region multimodal fusion features includes: performing similarity calculation processing on image description features and region detection visual features to obtain an association similarity value; processing the image description features based on the association similarity value to obtain visual enhancement features; and performing feature fusion processing on the visual enhancement features and region detection visual features to obtain region multimodal fusion features.

[0008] In some embodiments, the text region fusion features are subjected to hierarchical fusion processing to obtain the target fusion features corresponding to the query text, including: downsampling the text region fusion features to obtain a coarse-grained feature map, and obtaining a fine-grained feature map based on the text region fusion features; performing global attention processing on the coarse-grained feature map to obtain global coarse-grained features; performing local attention processing on the fine-grained feature map to obtain fine-grained features; calculating attention weights on the global coarse-grained features and fine-grained features to obtain hierarchical attention weight data, wherein the hierarchical attention weight data represents the correlation and / or similarity between the global coarse-grained features and fine-grained features; weighting the fine-grained features based on the hierarchical attention weight data to obtain fine-grained weighted features; and fusing the fine-grained weighted features and global coarse-grained features to obtain the target fusion features.

[0009] In some embodiments, multimodal decoding processing is performed on text features and target fusion features to obtain target detection results, including: obtaining a learnable query vector; concatenating the learnable query vector and text features to obtain concatenated features corresponding to the text features; and performing cross-attention processing on the concatenated features and target fusion features to obtain target detection results.

[0010] In some embodiments, global attention processing is performed on the coarse-grained feature map to obtain global coarse-grained features, including: performing feature serialization processing on the coarse-grained feature map to obtain serialized features; performing multi-head attention processing on the serialized features to obtain attention features; and performing residual connection processing on the attention features to obtain global coarse-grained features.

[0011] A second aspect of this disclosure provides a method for training an object detection model, comprising: acquiring training samples, wherein the training samples include image samples to be identified and query text samples, wherein the query text samples are used to query for a target object in the image samples to be identified that corresponds to the query text sample; inputting the training samples into the object detection model to be trained for the following training to obtain a trained object detection model: performing image recognition on the image samples to be identified to obtain training image descriptive features and training region detection visual features; performing region multimodal fusion processing on the training image descriptive features and training region detection visual features to obtain training region multimodal fusion features; performing feature fusion processing on the training text features and training region multimodal fusion features obtained based on the query text samples to obtain training text region fusion features corresponding to the query text samples; obtaining an object detection prediction result based on the training text features and training text region fusion features, wherein the object detection prediction result represents the position of the target object corresponding to the query text sample in the image samples to be identified and the category of the target object; obtaining a total loss value based on the error between the object detection prediction result and the true label; and adjusting the model parameters of the object detection model based on the total loss value.

[0012] A third aspect of this disclosure provides a target detection apparatus, comprising: an acquisition module for acquiring an image to be identified and query text, wherein the query text is used to query a target object in the image to be identified corresponding to the query text; a first processing module for performing image recognition on the image to be identified to obtain image description features and region detection visual features; a second processing module for performing region multimodal fusion processing on the image description features and region detection visual features to obtain region multimodal fusion features; a third processing module for performing feature fusion processing on text features obtained based on the query text and region multimodal fusion features to obtain text region fusion features corresponding to the query text; and a fourth processing module for obtaining a target detection result based on the text features and text region fusion features, wherein the target detection result represents the position of the target object in the image to be identified and the category of the target object.

[0013] A fourth aspect of this disclosure provides a training apparatus for an object detection model, comprising: a first training processing module for performing image recognition on the image sample to be identified to obtain training image descriptive features and training region detection visual features; a second training processing module for performing region multimodal fusion processing on the training image descriptive features and the training region detection visual features to obtain training region multimodal fusion features; a third training processing module for performing feature fusion processing on training text features obtained based on the query text sample and the training region multimodal fusion features to obtain training text region fusion features corresponding to the query text sample; a fourth training processing module for obtaining an object detection prediction result based on the training text features and the training text region fusion features, wherein the object detection prediction result represents the position of the sample target object corresponding to the query text sample in the image sample to be identified and the category of the sample target object; a fifth training processing module for obtaining a total loss value based on the error between the object detection prediction result and the true label; and a sixth training processing module for adjusting the model parameters of the object detection model based on the total loss value.

[0014] A fifth aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0015] A sixth aspect of this disclosure provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0016] The beneficial effects of this disclosed embodiment compared with the prior art are as follows: By acquiring the image to be identified and the query text, the image to be identified can be identified to extract image description features and region detection visual features; the above two features can be processed by region multimodal fusion to obtain region multimodal fusion features; the query text can be converted into corresponding text features and fused with the region multimodal fusion features to obtain text region fusion features; finally, based on the text features and text region fusion features, the target detection result of the target object in the image to be identified is obtained. In this way, by deeply fusing multimodal information and based on multi-level feature processing, targets of different scales are taken into account, improving the fusion degree of text semantics and image region features, and improving the accuracy and robustness of target detection in complex scenes. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure;

[0019] Figure 2 This is a schematic flowchart of a target detection method provided in an embodiment of this disclosure;

[0020] Figure 3 This is a flowchart illustrating another target detection method provided in this embodiment of the disclosure;

[0021] Figure 4 This is a flowchart illustrating a training method for an object detection model provided in an embodiment of this disclosure;

[0022] Figure 5 This is a schematic diagram of the structure of a target detection device provided in an embodiment of this disclosure;

[0023] Figure 6 This is a schematic diagram of the structure of a target detection training device provided in an embodiment of this disclosure;

[0024] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0026] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0027] A target detection method and apparatus according to embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings.

[0028] Figure 1This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure. The application scenario may include terminal devices 1, 2, and 3, server 4, and network 5.

[0029] Terminal devices 1, 2, and 3 can be hardware or software. When terminal devices 1, 2, and 3 are hardware, they can be various electronic devices with displays and supporting communication with server 4, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 1, 2, and 3 are software, they can be installed in the aforementioned electronic devices. Terminal devices 1, 2, and 3 can be implemented as multiple software programs or software modules, or as a single software program or software module; this disclosure does not limit this. Furthermore, various applications can be installed on terminal devices 1, 2, and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0030] Server 4 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 4 can be a single server, a server cluster consisting of several servers, or a cloud computing service center. This disclosure embodiment does not limit this.

[0031] It should be noted that server 4 can be either hardware or software. When server 4 is hardware, it can be various electronic devices that provide various services to terminal devices 1, 2, and 3. When server 4 is software, it can be multiple software programs or software modules that provide various services to terminal devices 1, 2, and 3, or it can be a single software program or software module that provides various services to terminal devices 1, 2, and 3. This disclosure does not limit the scope of the embodiments.

[0032] Network 5 can be a wired network using coaxial cable, twisted pair, and fiber optic connection, or it can be a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), and Infrared. This disclosure does not limit the scope of the network.

[0033] Users can establish a communication connection with server 4 via network 5 through terminal devices 1, 2, and 3 to receive or send information. Specifically, server 4 can acquire the image to be recognized and the query text through terminal devices 1, 2, and 3. The image to be recognized can be recognized to extract image description features and region detection visual features. The above two features can be processed through region multimodal fusion to obtain region multimodal fusion features. The query text can be converted into corresponding text features and fused with the region multimodal fusion features to obtain text region fusion features. Finally, based on the text features and text region fusion features, the target object in the image to be recognized is detected.

[0034] It should be noted that the specific types, quantities, and combinations of terminal devices 1, 2, and 3, server 4, and network 5 can be adjusted according to the actual needs of the application scenario, and this disclosure embodiment does not impose any restrictions on this.

[0035] Figure 2 This is a schematic flowchart of a target detection method provided in an embodiment of this disclosure. Figure 2 Target detection methods can be derived from Figure 1 The server executes the command. For example... Figure 2 As shown, the target detection method includes:

[0036] S201, Obtain the image to be recognized and the query text, wherein the query text is used to query the target object in the image to be recognized that corresponds to the query text.

[0037] Specifically, images to be identified can be acquired in real time through cameras, by reading locally stored image files and / or by internet image resources, etc. There are no limitations on this method. Query text can be acquired through methods such as recognizing input text, speech recognition conversion and / or text generation, etc. There are no limitations on this method either. This enhances the targeting of object detection, improves the efficiency of object detection, and increases the accuracy of object detection.

[0038] The image to be identified can be image data for target detection. This image can be obtained through methods such as real-time scene images captured by a camera, image files stored on a local device, and / or images downloaded from internet resources. No specific method is specified here.

[0039] The query text can be textual information that represents the requirement to find the target object in the image to be recognized. This query text can be obtained by recognizing input text and / or converting voice commands input via speech recognition; no specific limitation is made here.

[0040] The target object can be used to represent a specific object or region in the image to be identified that corresponds to the content described by the query text. This target object can be used in various application scenarios, including but not limited to image classification, object recognition, and / or scene understanding.

[0041] For example, in an intelligent security monitoring system, the image to be identified can be the image data of the monitored area collected in real time by the cameras installed in the shopping mall. This image data can be transmitted to the target detection model as the image to be identified. The query text "black backpack" can be entered to specify the target object to be found in the image to be identified. The target object can be a black backpack that actually exists in the image to be identified.

[0042] S202, perform image recognition on the image to be recognized to obtain image description features and region detection visual features.

[0043] Specifically, the image to be recognized can be input into a visual large language model, which includes, but is not limited to, a multimodal pre-trained model (Bootstrapping Language-Image Pre-training, BLIP) or a general visual language model (InstructBLIP). The visual large language model can parse the global semantic information of the image through a multi-layer encoder-decoder structure (Transformer) to generate image description feature vectors containing object categories, attributes, and relationships. The image to be recognized can be input into a Transformer-based object detection encoder (DetectionTransformer, DETR) to obtain region detection visual features that can be used to represent the spatial location and detailed information of the region. The image description features can be represented in the form of global semantic vectors, and the region detection visual features can be represented in the form of multi-dimensional feature maps. This enhances the synergy between semantic understanding and spatial localization, improves the representation ability of multimodal features, and enhances the semantic association accuracy and region discrimination robustness of object detection in complex scenes.

[0044] Among them, the image description feature can be the feature obtained by processing the image to be recognized through the visual big language model. The image description feature can be used to represent the high-level semantic information of the image to be recognized, including but not limited to object category, attributes and relationships. The image description feature can be obtained by parsing the global semantic information of the image to be recognized through the multi-layer Transformer structure of the visual big language model. The image description feature can also be used for tasks such as image classification and image retrieval, which are not limited here.

[0045] Region detection visual features can be vectors represented in the form of multi-dimensional feature maps. These region detection visual features can be used to represent the spatial location and detailed information of a region, etc., without limitation here. These region detection visual features can be obtained by inputting the image to be recognized into a DETR encoder based on the Transformer architecture.

[0046] For example, in an e-commerce platform, the image to be searched can be input into a visual big language model. The global semantic information of the image to be searched can be parsed through the Transformer structure of the visual big language model to generate an image description feature vector. The image to be searched can then be input into a DETR encoder based on the Transformer architecture for processing to obtain region detection visual features.

[0047] S203, perform multimodal fusion processing on image description features and region detection visual features to obtain multimodal fused features.

[0048] Specifically, the correlation between image descriptive features and region detection visual features can be calculated using cosine similarity. The similarity calculation result can be input into a multi-layer perceptron (MLP) for nonlinear transformation. Then, the weight information output by the MLP can be multiplied element-wise with the image descriptive features to enhance the visual feature components that match the semantic description. The multiplication result and the region detection visual features can be superimposed through residual connection, preserving the original feature information while fusing multimodal semantic associations to obtain multimodal fusion features of the region. This enhances the complementarity and robustness of cross-modal feature expression, improves the accuracy of image region semantic alignment, and enhances the generalization ability of target detection in complex scenes.

[0049] Among them, the region multimodal fusion feature can be a feature obtained by fusing image description features and region detection visual features. This region multimodal fusion feature can be obtained through steps and structures such as calculating the feature correlation degree by cosine similarity, performing nonlinear transformation by MLP, enhancing the matching feature components by element-wise multiplication, and fusing features by residual connection.

[0050] For example, in a security inspection system, for candidate areas of suspicious items, the similarity between the visual features of the region detection and the image description features related to the non-compliant items can be calculated using cosine similarity. The related features can be metal material and / or sharp shape, etc., which are not limited here.

[0051] The similarity calculation results can then be input into an MLP for nonlinear transformation. An MLP can process the input data through multiple hidden layers to obtain weight information. The weight information output by the MLP can be multiplied element-wise with the image description features. If a candidate region has a high degree of matching with the semantic description of non-compliant items in the image description features, then element-wise multiplication can amplify the visual feature components corresponding to that candidate region, highlighting information related to non-compliant items. Furthermore, the multiplication result can be superimposed with the region detection visual features through residual connections to obtain a multimodal fusion feature of the region.

[0052] S204, Perform feature fusion processing on the text features and region multimodal fusion features obtained based on the query text to obtain the text region fusion features corresponding to the query text.

[0053] Specifically, the text features obtained by the Contrastive Language–Image Pre-training (CLIP) can be used as the query vector Q. The multimodal fusion features of the regions can be processed through residual connections to obtain the key vector K and the value vector V. Then, the association weights between the text features and the multimodal fusion features of the regions can be obtained through a multi-head cross-attention mechanism. Each attention head can perform vector dimension mapping and similarity matrix calculation on Q and K. The values ​​can be normalized using a normalized exponential function (softmax) and then weighted and aggregated into a value vector V. The outputs of each attention head can be concatenated and processed through a linear projection layer to obtain text region fusion features containing multimodal semantic associations. This enhances the ability to express regional features under text guidance, improves the alignment accuracy between text and image regions, and enhances the semantic matching ability of object detection in open scenes.

[0054] The text feature can be a vector representation obtained by encoding the query text using the CLIP text encoder. This text feature can be used to characterize the semantic information of the query text. It can be a digital abstraction of the query text content, containing information about the semantics and syntax of the query text, and is not limited here.

[0055] The text region fusion feature corresponding to the query text can be a feature vector obtained by fusing the text features of the query text with the region multimodal fusion feature. This text region fusion feature can be used to represent information from both the text and image regions.

[0056] For example, in the user search system of an e-commerce platform, the CLIP text encoder can process the input query text "red dress, long, lace trim" to obtain a fixed-dimensional feature vector. This feature vector is the text feature corresponding to the query text. Then, the product images on the e-commerce platform can be segmented into multiple regions (such as the neckline, cuffs, and hem of the dress), and processed to obtain region multimodal fusion features. These region multimodal fusion features can be processed using residual connections to obtain key vector K and value vector V. The text features obtained from the CLIP text encoder can be used as the query vector Q, and vector dimension mapping and similarity matrix calculation can be performed with the key vector K. The similarity matrix is ​​normalized using the softmax function to obtain the association weights between the text features and the region multimodal fusion features. The value vector V can then be aggregated based on the association weights. The outputs of each attention head can be concatenated and processed through a linear projection layer to obtain the text region fusion features corresponding to the query text "red dress, long, lace trim".

[0057] S205, the target detection result is obtained based on text features and text region fusion features, where the target detection result represents the position of the target object in the image to be identified and the category of the target object.

[0058] Specifically, a learnable query vector can be obtained from the parameters. Then, the text features can be concatenated with the learnable query vector to obtain an enhanced feature vector. This enhanced feature vector can be cross-attention calculated with image features fused hierarchically. The attention weights are dynamically focused on the image region related to the text description. The attention calculation result can then be input into a prediction head containing fully connected layers and activation functions to obtain the target detection result. In this way, the attention mechanism enhances feature expression and interaction, strengthens the spatial correlation between text semantics and image features, improves the robustness of target location prediction in complex scenes, and enhances the accuracy of text-driven open target detection.

[0059] The object detection result can be used to characterize the bounding box information and category of the target object. The bounding box information of the target object can include the coordinate information or orientation information of the bounding box, etc., which is not limited here. The category of the target object includes, but is not limited to, people, animals, or vehicles. The object detection result can be used in text-driven object detection scenarios, such as locating and recognizing the corresponding target object in the image based on the text description input by the user.

[0060] For example, in an intelligent photo album management system, a user can input "boys playing football in the park" as the query text and search within the intelligent album. Learnable query vectors can be obtained from the parameters. These learnable query vectors, corresponding to the text features of "boys playing football in the park," are concatenated to obtain an enhanced feature vector. This vector can then be used to perform hierarchical fusion processing on the photos, resulting in hierarchically fused image features. Cross-attention calculations can be performed on the enhanced feature vectors and the hierarchically fused image features, dynamically focusing on image regions related to the query text "boys playing football in the park" through attention weights. These regions might include people, footballs, park scenes, etc. The results of the cross-attention calculations can then be input into a prediction head containing fully connected layers and activation functions. The fully connected layers can perform linear transformations on the input features, while the activation functions can introduce non-linear factors. The prediction head can then obtain target detection results based on the input features, including the boy's position in the photo (represented by bounding box coordinates) and category ("boy").

[0061] According to the technical solution provided in this disclosure, a visual large language model is used to perform semantic parsing on the image to be recognized, obtaining image description features containing object attributes and spatial relationships. Furthermore, a DETR encoder can extract the region detection visual features of the image to be recognized. A CLIP text encoder can map the query text to the semantic vector space to obtain text features. Then, a correlation matrix between the image description features and the region visual features can be constructed through cross-modal similarity calculation. Nonlinear transformation can be performed using MLP to obtain region multimodal features. Text features can be used as attention queries and cross-attention calculations can be performed with the region multimodal features. Text region fusion features are generated through hierarchical attention architecture and cross-level attention weight fusion. Finally, text features can be concatenated with learnable query vectors and cross-attention calculations can be performed with the text region fusion features to obtain the target detection result. This enhances the semantic alignment capability of multimodal features, improves the robustness of text-image association in complex scenarios, and enhances the accuracy and generalization performance of cross-modal target detection.

[0062] In some embodiments, obtaining target detection results based on text features and text region fusion features includes: performing hierarchical fusion processing on the text region fusion features to obtain target fusion features corresponding to the query text; and performing multimodal decoding processing on the text features and target fusion features to obtain target detection results.

[0063] Specifically, text region fusion features can be hierarchically divided. Coarse-grained features containing global semantics are obtained through downsampling as coarse-grained layers, while text region fusion features retaining the original resolution are used as fine-grained features. Each fine-grained feature constitutes a fine-grained layer. Then, a multi-head self-attention mechanism can be used in the coarse-grained layer to obtain global correlation information across regions. In the fine-grained layer, detailed features can be extracted through local window attention. Furthermore, a cross-level attention mechanism can be used to obtain the mapping relationship between features at different levels. The coarse-grained features can be upsampled and adaptively weighted and fused with the fine-grained features to obtain the target fusion features. Then, through multimodal decoding processing, the text features are used as query vectors, and the target fusion features are used as key-value pairs input to the decoder. Through multiple rounds of iterative calculation, a feature matching weight matrix is ​​generated. Finally, the target bounding box coordinates and class confidence are output through a learnable detection head, thus obtaining the target detection result.

[0064] The target fusion feature corresponding to the query text can be a feature representation that includes information from different levels of text region fusion features. This target fusion feature can be used to characterize the global semantic information and local region detail features extracted from the text region fusion features, and the association between features at different levels can be established through a cross-level attention mechanism.

[0065] For example, in an intelligent medical image diagnosis assistance system, a query text such as "potential tumors in the liver, approximately 2 cm in diameter, with an irregular shape" can be input to assist in the analysis of CT images. The text region fusion features extracted from medical images can be hierarchically divided. By downsampling, the feature map is reduced to obtain coarse-grained features containing global semantics. These coarse-grained features can be used to characterize the general situation of the entire liver region, such as the overall shape and size of the liver, as a coarse-grained layer. The text region fusion features that retain the original resolution can be used as fine-grained features. These fine-grained features can be used to characterize detailed information of local areas of the liver, such as blood vessel distribution and tissue texture. The various fine-grained features constitute a fine-grained layer.

[0066] In the coarse-grained layer, multi-head self-attention mechanisms can be used to process coarse-grained features. In the fine-grained layer, local window attention mechanisms can be used to extract features from local regions of each fine-grained feature. Furthermore, a cross-layer attention mechanism can be used to establish a mapping relationship between the coarse-grained and fine-grained layers. For example, based on global correlation information, more important local detail features for determining tumor location and category can be identified. Then, the coarse-grained features can be upsampled to achieve the same resolution as the fine-grained features, and adaptive weighted fusion can be performed to obtain the target fused feature.

[0067] Using the feature "potential tumors in the liver, approximately 2 cm in diameter, with irregular shape" as the query vector, and the target fusion feature as the key-value pair input decoder, a feature matching weight matrix is ​​obtained. The location of the tumor in the CT image and the probability of whether the tumor is benign or malignant can be output through a learnable detection head, thus obtaining the target detection result.

[0068] According to the technical solution provided in this disclosure, the text region fusion features are hierarchically divided. Coarse-grained features containing global semantics are obtained through downsampling as coarse-grained layers, while text region fusion features retaining the original resolution are used as fine-grained features. Each fine-grained feature is a fine-grained layer. Then, global correlation information across regions can be obtained through a multi-head self-attention mechanism in the coarse-grained layer. Detail features can be extracted through local window attention in the fine-grained layer. Then, the mapping relationship between features at different levels can be obtained through a cross-level attention mechanism. The coarse-grained features can be upsampled and adaptively weighted and fused with the fine-grained features to obtain the target fusion features. Then, the text features are used as query vectors and the target fusion features are used as key-value pairs input to the decoder through multi-modal decoding processing. The feature matching weight matrix is ​​generated through multiple rounds of iterative calculation. Finally, the target bounding box coordinates and category confidence are output through a learnable detection head, thus obtaining the target detection result. In this way, the accuracy of target detection and cross-modal feature matching capability in complex scenes are enhanced through multi-level feature fusion and cross-modal interaction mechanisms.

[0069] In some embodiments, performing region multimodal fusion processing on image description features and region detection visual features to obtain region multimodal fusion features includes: performing similarity calculation processing on image description features and region detection visual features to obtain an association similarity value; processing the image description features based on the association similarity value to obtain visual enhancement features; and performing feature fusion processing on the visual enhancement features and region detection visual features to obtain region multimodal fusion features.

[0070] Specifically, cosine similarity can be used to calculate the correlation between image descriptive features and region detection visual features for each region, resulting in a corresponding similarity matrix. This similarity matrix can then undergo nonlinear transformation and feature enhancement using an MLP (Multi-Level Processing) to apply attention weighting to the image descriptive features, yielding visual enhancement features. Furthermore, residual connections can be used to weightedly superimpose the visual enhancement features and region detection visual features to obtain multimodal fusion features for the region.

[0071] The association similarity value is a numerical measure of the degree of association between image descriptive features and region detection visual features. This value characterizes the semantic or spatial similarity between the two features. A higher value indicates a stronger association between the two features, meaning the content expressed by the image descriptive feature matches the visual information detected in the region more closely. Visual enhancement features are obtained by applying attention-weighted processing based on the association similarity value to the image descriptive features.

[0072] For example, in an intelligent advertising system, an advertisement image displays a watch, and the query text is "new fashionable watch, metal strap, round dial". Visual features for region detection can be extracted from the advertisement image. For instance, object detection algorithms can be used to detect various regions in the advertisement image and extract the visual features of each region. The query text "new fashionable watch, metal strap, round dial" can be processed to obtain image description features. The correlation between the image description features and the region detection visual features of each region can be calculated using cosine similarity. The similarity matrix can be nonlinearly transformed and feature-enhanced using MLP. Attention weights can be applied to the image description features based on similarity information. Finally, the visual enhancement features and the region detection visual features can be weighted and superimposed using residual connections to obtain multimodal fusion features for the region.

[0073] According to the technical solution provided in this disclosure, the correlation similarity value between image descriptive features and region detection visual features is calculated using a cosine similarity algorithm; the correlation similarity value is input into an MLP for nonlinear transformation to obtain dynamic weight coefficients; these dynamic weight coefficients can be multiplied element-wise with the image descriptive features to obtain visual enhancement features; and then, the visual enhancement features can be superimposed on the region detection visual features through residual connections to obtain region multimodal fusion features. In this way, the interaction capability between semantic features and visual features is enhanced through dynamic weight allocation and residual connection mechanisms, thereby improving the representation accuracy of multimodal fusion features.

[0074] In some embodiments, the text region fusion features are subjected to hierarchical fusion processing to obtain the target fusion features corresponding to the query text, including: downsampling the text region fusion features to obtain a coarse-grained feature map, and obtaining a fine-grained feature map based on the text region fusion features; performing global attention processing on the coarse-grained feature map to obtain global coarse-grained features; performing local attention processing on the fine-grained feature map to obtain fine-grained features; calculating attention weights on the global coarse-grained features and fine-grained features to obtain hierarchical attention weight data, wherein the hierarchical attention weight data represents the correlation and / or similarity between the global coarse-grained features and fine-grained features; weighting the fine-grained features based on the hierarchical attention weight data to obtain fine-grained weighted features; and fusing the fine-grained weighted features and global coarse-grained features to obtain the target fusion features.

[0075] Specifically, coarse-grained feature maps can be extracted from text region fusion features through downsampling, while retaining fine-grained feature maps at the original resolution. Global attention mechanisms can be applied to the coarse-grained feature maps to obtain global coarse-grained features, and local window-based attention calculations can be performed on the fine-grained feature maps to obtain fine-grained features. Cross-level feature similarity measurement matrices can be established and normalized to generate inter-level attention weight data. Based on this weight data, channel weighting can be applied to the fine-grained features to obtain fine-grained weighted features. Then, the global coarse-grained features and fine-grained weighted features can be integrated through feature overlay or concatenation, without limitation, to obtain the target fusion feature.

[0076] The coarse-grained feature map can be a feature map obtained by downsampling the fused features of text regions. This coarse-grained feature map can be used to represent the global semantic information of the fused features of text regions. The fine-grained feature map can be a text region fusion feature map that retains the original resolution. This fine-grained feature map can be used to represent the local detail information of the text region, including but not limited to the strokes of characters and the shape of characters. The global coarse-grained feature can be a feature obtained by performing global attention processing on the coarse-grained feature map. The fine-grained feature can be a feature obtained by performing attention calculation based on a local window on the fine-grained feature map. This fine-grained feature highlights the important local information in the fine-grained feature map. The inter-level attention weight data can be used to represent the correlation and / or similarity between the global coarse-grained feature and the fine-grained feature. This inter-level attention weight data can be represented in the form of a weight matrix and can be used to measure the importance and correlation between features at different levels. The fine-grained weighted feature can be a feature obtained by weighting the fine-grained feature based on the inter-level attention weight data.

[0077] For example, in an intelligent document processing system, text region fusion features can be extracted based on document images. By downsampling the text region fusion features, a coarse-grained feature map can be extracted. This coarse-grained feature map retains the overall layout and general semantic information of the contract document, while a fine-grained feature map retains the original resolution. This fine-grained feature map contains local details such as the strokes and character shapes of the text in the contract document.

[0078] Global attention mechanisms can be applied to coarse-grained feature maps to obtain global coarse-grained features, such as the main content and structure of a contract. Fine-grained feature maps can be processed based on local window attention calculations to obtain fine-grained features, such as the shape and position of specific characters.

[0079] A cross-level feature similarity metric matrix can be built and normalized to generate inter-level attention weight data. This weight data can be used to measure the correlation and similarity between global coarse-grained features and fine-grained features, for example, to determine which local details are more relevant to the overall semantics of the contract.

[0080] Furthermore, based on the attention weight data between layers, fine-grained features can be channel-weighted to obtain fine-grained weighted features.

[0081] Fine-grained weighted features and global coarse-grained features can be integrated through feature concatenation to obtain the target fusion feature.

[0082] According to the technical solution provided in this disclosure, a coarse-grained feature map is extracted from the text region fusion features by downsampling, while retaining the fine-grained feature map of the original resolution. A global attention mechanism can be applied to the coarse-grained feature map to obtain global coarse-grained features, and a local window-based attention calculation can be performed on the fine-grained feature map to obtain fine-grained features. A cross-level feature similarity measurement matrix can be established and normalized to generate inter-level attention weight data. Based on this weight data, channel weighting is applied to the fine-grained features to obtain fine-grained weighted features. Then, the global coarse-grained features and fine-grained weighted features can be integrated through feature superposition or splicing, without limitation, to obtain target fusion features. This enhances the collaborative expression capability of cross-level features and improves the detection accuracy of multi-scale targets in complex scenes.

[0083] In some embodiments, multimodal decoding processing is performed on text features and target fusion features to obtain target detection results, including: obtaining a learnable query vector; concatenating the learnable query vector and text features to obtain concatenated features corresponding to the text features; and performing cross-attention processing on the concatenated features and target fusion features to obtain target detection results.

[0084] Specifically, learnable query vectors can be obtained through parameters. These learnable query vectors are then concatenated with text features according to their feature dimensions to obtain concatenated features corresponding to the text features. These concatenated features can be used as query vectors for the cross-attention mechanism, and target fusion features can be used as key and value vectors. Cross-modal feature alignment is performed by calculating the attention weight matrix to obtain the target detection results.

[0085] Among them, the learnable query vector can be a vector whose parameters can be automatically adjusted during the model training process. The learnable query vector can be used to guide the model to focus on information related to the target task in multimodal decoding processing; the concatenated feature corresponding to the text feature can be the feature obtained by concatenating the learnable query vector and the text feature according to the feature dimension.

[0086] For example, in an intelligent security monitoring system, the query text is "A person has fallen in the monitored area," and the image to be identified is an image captured by a camera. A learnable query vector is obtained through a parameter initialization mechanism. This learnable query vector and the text feature "A person has fallen in the monitored area" are then concatenated according to their feature dimensions to obtain the concatenated features corresponding to the text features. These concatenated features can then be used as the query vector for a cross-attention mechanism, with the target fusion features serving as the key and value vectors. By calculating the attention weight matrix, cross-modal feature alignment can be performed on the target fusion features. This means that, guided by the text information and the learnable query vector, the region in the image most relevant to the "person has fallen" behavior can be determined. Based on the feature alignment result, the target detection result can be obtained, thus detecting the location of the person who has fallen in the monitored image.

[0087] According to the technical solution provided in this disclosure, a learnable query vector is obtained through parameters, and the learnable query vector and text features are concatenated according to the feature dimension to obtain the concatenated features corresponding to the text features. The concatenated features can be used as the query vector of the cross-attention mechanism, and the target fusion features are used as the key vector and value vector. Cross-modal feature alignment is performed by calculating the attention weight matrix to obtain the target detection result. This improves the matching accuracy of text semantics and visual features in the target detection process and enhances the target localization accuracy in complex scenes.

[0088] In some embodiments, global attention processing is performed on the coarse-grained feature map to obtain global coarse-grained features, including: performing feature serialization processing on the coarse-grained feature map to obtain serialized features; performing multi-head attention processing on the serialized features to obtain attention features; and performing residual connection processing on the attention features to obtain global coarse-grained features.

[0089] Specifically, feature serialization can be used to unfold a two-dimensional coarse-grained feature map into a one-dimensional feature sequence based on the spatial dimension, thereby encoding the spatial information of the features and obtaining serialized features. Then, a multi-head attention mechanism can be used to perform parallel attention computation on the serialized features, and attention features can be obtained through multiple sets of independent query-key-value matrices. Finally, residual connections can be used to superimpose the coarse-grained feature map and the attention features to obtain global coarse-grained features. In this process, feature serialization can achieve feature dimension transformation by flattening the spatial dimension, multi-head attention can construct attention computation units by using layer normalization and feedforward neural network structures, and residual connection can achieve feature fusion by adding elements one by one.

[0090] In this context, serialized features can be obtained by expanding a two-dimensional coarse-grained feature map into a one-dimensional feature sequence based on spatial dimensions. This serialized feature can transform the original coarse-grained feature map with spatial structure into a linear sequence, allowing the features to be processed sequentially by attention mechanisms and other processing modules.

[0091] Attention features can be obtained by applying multi-head attention to serialized features. The multi-head attention mechanism calculates attention weights in parallel using multiple independent query-key-value matrices to obtain attention features. These attention features can be used to highlight key information in the serialized features that is relevant to the global semantics.

[0092] For example, in an intelligent traffic monitoring system, coarse-grained feature maps can be extracted from traffic monitoring images. These coarse-grained feature maps can be used to characterize the approximate shape and location information of objects such as vehicles in the monitoring images. The coarse-grained feature maps are then processed into feature serialization to obtain serialized features. Multi-head attention processing can be applied to the serialized features, and attention weights can be calculated in parallel using multiple independent query-key-value matrices to highlight the features corresponding to the areas in the monitoring images where large vehicles may appear, thus obtaining attention features. Finally, the coarse-grained feature maps and attention features can be superimposed using residual connections to obtain global coarse-grained features.

[0093] According to the technical solution provided in this disclosure, feature serialization expands a two-dimensional coarse-grained feature map into a one-dimensional feature sequence based on the spatial dimension, thereby encoding the spatial information of the features and obtaining serialized features. Then, the serialized features can be parallelized by a multi-head attention mechanism to obtain attention features through multiple sets of independent query-key-value matrices. The coarse-grained feature map and attention features can be superimposed by residual connection to obtain global coarse-grained features. In this way, the combination of multi-dimensional attention calculation and feature preservation mechanism enhances the ability to capture global semantic information of the image and improves the detection accuracy of large-scale targets.

[0094] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0095] Figure 3 This is a schematic flowchart of another target detection method provided in an embodiment of this disclosure. Figure 3 As shown, the target detection method includes:

[0096] 1. Feature Extraction

[0097] (1) Image description feature extraction

[0098] The input image (the image to be recognized) is fed into a visual large language model, such as BLIP or InstructBLIP. During processing, the visual large language model can recognize, understand, and associate various elements of the image, ultimately outputting image description features T. des This feature, presented in vector form, contains semantic information about objects in the image, such as their categories, attributes, and relationships, providing an important semantic foundation for subsequent multimodal fusion.

[0099] (2) Image detection and visual feature extraction

[0100] The image is input to the DETR encoder module. The DETR encoder extracts features from the image through convolutional layers and Transformer layers to obtain the region detection visual features V. box This feature can be used to characterize visual information in different regions of an image, helping the object detection model to locate target objects in the image.

[0101] (3) Text feature extraction

[0102] The input text (corresponding to the query text of this disclosure, i.e., the identifier of the detection category object, such as "dog") is input into the CLIP text encoder. The CLIP text encoder converts the text into a vector representation, obtaining the text feature T. clip This feature contains semantic information about the input text.

[0103] 2. Regional multimodal fusion

[0104] Similarity calculation

[0105] Calculate image descriptive features T des Visual features V for region detection box The similarity between features is determined using cosine similarity. This method allows us to understand the correlation between semantic information and visual information from different regions of an image, providing a basis for subsequent feature fusion.

[0106] (2) MLP processing

[0107] The similarity calculation results are input into a multilayer perceptron (MLP) module. The MLP contains multiple fully connected layers and activation functions (such as ReLU) to perform non-linear transformations on the similarity information. This enhances the object detection model's ability to express the relationships between features.

[0108] (3) Feature fusion

[0109] The output of the MLP is compared with the image description features T. des Multiplication enhances the feature information associated with the image description, thereby strengthening visual features that better match the semantic description.

[0110] By using residual connections, the region detection visual features are added to the above multiplication results to obtain the region multimodal fusion feature V. multi The role of residual connections is to avoid information loss during feature fusion, allowing the object detection model to better preserve the original visual features of the region detection. The final result is a region multimodal fusion feature, which integrates the semantic information of the image and the visual information of the region.

[0111] 3. Text region fusion

[0112] Text features T clip As the query (Q), the regional multimodal fusion feature T des Using the key (K) and value (V) as keys, cross-attention is calculated to obtain the text region fusion feature V. cos .

[0113] Cross-attention mechanisms allow models to focus on text-related parts of multimodal region fusion features based on textual information, thereby achieving effective fusion of text and image region features.

[0114] 4. Feature-level fusion

[0115] (1) Hierarchical division

[0116] Coarse-grained layer generation: Using average pooling to fuse text region features V cos Downsampling is performed to obtain a coarse-grained feature map. This coarse-grained feature map preserves the overall semantic information of the image while removing some detailed information, helping the model understand the image from a macroscopic level.

[0117] Fine-grained layer preservation: Text region fusion features are directly used as fine-grained features. Fine-grained feature maps preserve detailed image information, providing the model with richer local features for accurate object detection.

[0118] (2) Intra-level attention mechanism

[0119] Coarse-grained layer global attention: Coarse-grained features are reshaped into a form suitable for input to a multi-head attention mechanism. A multi-head attention machine is used for computation, and then residual connections are made to the coarse-grained features to enhance the feature representation, resulting in the updated coarse-grained feature F. coarse .

[0120] The fine-grained layer local attention uses 1×1 convolutions to project fine-grained features onto the query, key, and value spaces, respectively. The projected features are then divided into multiple non-overlapping local regions. Attention is computed within each local region, and the fine-grained features are preserved by connecting them via residuals, resulting in the updated fine-grained feature F. fine .

[0121] (3) Inter-level attention fusion

[0122] Upsampling operation: For coarse-grained features F coarse Upsampling is performed to make its size consistent with the fine-grained feature F. fine The same applies. Upsampling can use methods such as bilinear interpolation to restore the resolution of the coarse-grained feature map so that it can be fused with the fine-grained feature map.

[0123] Calculate attention weights: Calculate the similarity matrix between coarse-grained and fine-grained features, then normalize the similarity matrix using the softmax function to obtain the attention weights. The attention weights represent the degree of correlation between coarse-grained and fine-grained features.

[0124] Feature fusion: Fine-grained features are weighted and summed using attention weights, and then fused with coarse-grained features. This fully combines coarse-grained global semantic information with fine-grained local detail information to obtain the final fused feature F. com (Text region fusion features)

[0125] 5. Target Detection and Fusion

[0126] (1) Feature fusion

[0127] By concatenating text features into a learnable query vector (learnable query vector), the query vector continuously learns target features during training, which can help the model better focus on the target object.

[0128] The spliced ​​features are then combined with the final fused feature F. com Perform cross-attention calculation to obtain the final input feature F. final (Target Fusion Features). In this way, textual information and image features are further fused to obtain the final input features, which integrate textual, image semantics, and visual information, providing richer information for target detection.

[0129] (2) Target prediction and post-processing

[0130] F final The input to the DETR model's prediction head, which contains a series of fully connected layers and activation functions, outputs predicted class probabilities and bounding box coordinates (object detection results) based on the input features. The class probability represents the likelihood of the target class corresponding to each detection box, while the bounding box coordinates are used to locate the target object's position in the image.

[0131] 6. Training loss

[0132] The loss L between the category of the sample target object and the true classification label is calculated using the cross-entropy loss function. cls The cross-entropy loss function measures the difference between the category of a sample target object and the true classification label. The closer the category of the sample target object is to the true classification label, the smaller the loss value. By minimizing the classification loss, the accuracy of the object detection model in classifying the target category is improved.

[0133] Furthermore, the loss L between the position of the target object in the image sample to be identified and the coordinates of the true bounding box can be calculated by combining the mean absolute error and the generalized intersection-union loss. box Mean absolute error can be used to measure the absolute error between the position of the target object in the image sample to be identified and the coordinates of the true bounding box, while generalized intersection-union loss can be used to characterize the degree of overlap and relative positional relationship between two boxes.

[0134] The loss L between the object detection prediction result and the training text features can be calculated using similarity comparison loss. contrast This improves the matching accuracy between text and image features.

[0135] Total loss: L = α1L cls +α2L box +α3L contrast , where α1, α2, and α3 can be weighting coefficients for different losses.

[0136] Figure 4 This is a flowchart illustrating a training method for an object detection model provided in an embodiment of this disclosure. Figure 4 As shown, the training method for this object detection model includes:

[0137] S401, perform image recognition on the image sample to be recognized to obtain training image description features and training region detection visual features.

[0138] Specifically, the image to be recognized can be input into a visual big language model. The global semantic information of the image can be parsed through the Transformer structure of the visual big language model to generate training image description features containing object categories, attributes, and relationships. The image to be recognized can be input into a DETR encoder based on the Transformer architecture to obtain training region detection visual features that can be used to represent the spatial location and detailed information of the region. The training image description features can be represented in the form of global semantic vectors, and the training region detection visual features can be represented in the form of multi-dimensional feature maps. This enhances the synergy between semantic understanding and spatial localization, improves the representation ability of multimodal features, and enhances the semantic association accuracy and region discrimination robustness of target detection in complex scenes.

[0139] Among them, the training image description features can be features obtained by processing the image to be recognized through the visual big language model. The training image description features can be used to represent the high-level semantic information of the image to be recognized, including but not limited to object categories, attributes and relationships. The training image description features can be obtained by parsing the global semantic information of the image to be recognized through the multi-layer Transformer structure of the visual big language model. The training image description features can also be used for tasks such as image classification and image retrieval, which are not limited here.

[0140] The visual features for training region detection can be vectors represented in the form of multi-dimensional feature maps. These visual features can be used to represent the spatial location and detailed information of the region, etc., without limitation here. These visual features for training region detection can be obtained by inputting the image to be recognized into a DETR encoder based on the Transformer architecture.

[0141] S402, perform multimodal fusion processing on the training image description features and the training region detection visual features to obtain the multimodal fusion features of the training region.

[0142] Specifically, cosine similarity can be used to calculate the correlation between the training image descriptive features and the training region detection visual features. The similarity calculation result can be input into an MLP for nonlinear transformation. Then, the weight information output by the MLP can be multiplied element-wise with the training image descriptive features to enhance the visual feature components that match the semantic description. The multiplication result and the training region detection visual features can be superimposed through residual connection to retain the original feature information while fusing multimodal semantic associations, thereby obtaining multimodal fusion features of the training region. This enhances the complementarity and robustness of cross-modal feature expression, improves the accuracy of image region semantic alignment, and enhances the generalization ability of target detection in complex scenes.

[0143] The multimodal fusion feature of the training region can be obtained by fusing the descriptive features of the training image and the visual features detected in the training region. This multimodal fusion feature of the training region can be obtained through steps and structures such as calculating the feature correlation degree by cosine similarity, performing nonlinear transformation by MLP, enhancing the matching feature components by element-wise multiplication, and fusing features by residual connection.

[0144] S403, perform feature fusion processing on the training text features and training region multimodal fusion features obtained based on the query text samples to obtain the training text region fusion features corresponding to the query text samples.

[0145] Specifically, the training text features obtained through CLIP processing can be used as the query vector Q. The multimodal fusion features of the training region can be processed through residual connections to obtain the key vector K and value vector V. Then, the association weights between the training text features and the multimodal fusion features of the training region can be obtained through a multi-head cross-attention mechanism. Each attention head can perform vector dimension mapping and similarity matrix calculation on Q and K, which can be normalized by softmax and weighted to aggregate the value vector V. The outputs of each attention head can be concatenated and processed through a linear projection layer to obtain the training text region fusion features containing multimodal semantic associations. This enhances the ability to express regional features under text guidance, improves the alignment accuracy between text and image regions, and enhances the semantic matching ability of object detection in open scenes.

[0146] The training text features can be vector representations obtained by encoding the query text samples using the CLIP text encoder. These training text features can be used to characterize the semantic information of the query text samples. They can be digital forms of the query text sample content, containing information on the semantics and syntax of the query text samples, and are not limited here.

[0147] The training text region fusion feature corresponding to the query text sample can be a feature vector obtained by fusing the training text features of the query text sample with the training region multimodal fusion feature. This training text region fusion feature can be used to represent information from both the text and image regions.

[0148] S404. The target detection prediction result is obtained based on the training text features and the training text region fusion features. The target detection prediction result represents the position of the target object corresponding to the query text sample in the image sample to be identified and the category of the target object.

[0149] Specifically, a learnable query vector can be obtained from the parameters. Then, the training text features can be concatenated with the learnable query vector to obtain a training enhanced feature vector. This training enhanced feature vector can be cross-attention calculated with image features fused hierarchically. The attention weights are used to dynamically focus on image regions related to the text description. The attention calculation results can then be input into a prediction head containing fully connected layers and activation functions to obtain object detection prediction results. In this way, the spatial correlation between text semantics and image features is enhanced, the robustness of target location prediction in complex scenes is improved, and the accuracy of text-driven open object detection is increased.

[0150] The object detection prediction result can be used to characterize the bounding box information and category of the sample target object. The bounding box information of the sample target object may include the coordinate information or orientation information of the bounding box, etc., which is not limited here. The category of the sample target object includes, but is not limited to, people, animals, or vehicles. The object detection prediction result can be used in text-driven object detection scenarios, such as locating and recognizing the corresponding sample target object in the image based on the text description input by the user.

[0151] S405, based on the error between the object detection prediction result and the true label, the total loss value is obtained.

[0152] Specifically, the target detection prediction results and the true labels can be processed by at least one loss function to obtain the total loss value of the target detection model. The loss function includes, but is not limited to, the cross-entropy loss function or the similarity comparison loss function.

[0153] Among them, the true label can be a marker containing accurate information about the target object in the image sample to be identified. The true label can be used to characterize the category of the target object and its specific location in the image.

[0154] S406, Adjust the model parameters of the target detection model based on the total loss value.

[0155] Specifically, the gradient of the total loss value to each layer parameter in the object detection model can be calculated using the chain rule. The calculated gradient can then be used to update the object detection model parameters based on the preset learning rate and gradient information. This enhances the object detection model's ability to learn different target features, improves the convergence efficiency of the object detection model during training, and enhances the accuracy and robustness of the object detection model in practical applications.

[0156] For example, in an intelligent security monitoring system, the target detection model can process received training samples, where the image samples to be identified can be surveillance video frames collected from different monitoring scenes (such as shopping malls, stations, etc.), which contain various scenes of people and objects, and the query text samples can be text describing the target object, such as "suspicious person who took other people's property," to obtain a target detection prediction result. This result can represent the position of the target object "suspicious person who took other people's property" in the image samples to be identified (e.g., represented by bounding box coordinates) and the category of the target object (e.g., "suspicious person"). Then, based on the error between the target detection prediction result and the manually labeled real labels (the real labels accurately mark the actual position and category of "suspicious person who took other people's property" in the image), a loss function is used to calculate the total loss value. Finally, based on the total loss value, the model parameters of the target detection model are adjusted using the backpropagation algorithm and optimizer. After multiple iterations of training, a well-trained target detection model is finally obtained.

[0157] According to the technical solution provided in this disclosure, image recognition is performed on the image samples to be identified in the obtained training samples to obtain training image descriptive features and training region detection visual features; multimodal fusion processing is performed on the training image descriptive features and training region detection visual features to obtain training region multimodal fusion features; feature fusion processing is performed on the training text features and training region multimodal fusion features obtained based on query text samples to obtain training text region fusion features corresponding to the query text samples; target detection prediction results are obtained based on the training text features and training text region fusion features; the total loss value is obtained based on the error between the target detection prediction result and the true label; and the model parameters of the target detection model are adjusted based on the total loss value. This enhances the target detection model's understanding and fusion ability of multimodal information through joint optimization using a text-image fusion mechanism and multiple loss functions, thereby improving the accuracy and specificity of target detection and enhancing the generalization ability of the target detection model.

[0158] In some embodiments, the ground truth labels include ground truth classification labels and ground truth bounding box coordinates. The total loss value is obtained based on the error between the object detection prediction result and the ground truth labels, including: calculating cross-entropy loss for the category of the sample target object and the ground truth classification label to obtain the classification loss value corresponding to the object detection model; calculating regression loss for the position of the sample target object in the image sample to be identified and the ground truth bounding box coordinates to obtain the bounding box regression loss value corresponding to the object detection model; calculating contrast loss for the object detection prediction result and the training text features to obtain the contrast loss value corresponding to the object detection model; and obtaining the total loss value based on the classification loss value, the bounding box regression loss value, and the contrast loss value.

[0159] Specifically, the loss L between the category of the sample target object and the true classification label can be calculated using the cross-entropy loss function. cls The cross-entropy loss function measures the difference between the category of a sample target object and the true classification label. The closer the category of the sample target object is to the true classification label, the smaller the loss value. By minimizing the classification loss, the accuracy of the object detection model in classifying the target category is improved.

[0160] Furthermore, the loss L between the position of the target object in the image sample to be identified and the coordinates of the true bounding box can be calculated by combining the mean absolute error and the generalized intersection-union loss. box Mean absolute error can be used to measure the absolute error between the position of the target object in the image sample to be identified and the coordinates of the true bounding box, while generalized intersection-union loss can be used to characterize the degree of overlap and relative positional relationship between two boxes.

[0161] The loss L between the object detection prediction result and the training text features can be calculated using similarity comparison loss. contrast This improves the matching accuracy between text and image features.

[0162] Total loss: L = α1L cls +α2L box +α3L contrast , where α1, α2, and α3 can be weighting coefficients for different losses.

[0163] According to the technical solution provided in this disclosure, the loss L between the category of the sample target object and the true classification label is calculated using the cross-entropy loss function. cls Furthermore, the loss L between the position of the target object in the image sample to be identified and the coordinates of the true bounding box can be calculated by combining the mean absolute error and the generalized intersection-union loss. box The loss L between the target detection prediction result and the training text features can be calculated using similarity comparison loss. contrast Total loss: L = α1L cls +α2L box +α3L contrast α1, α2, and α3 can be weight coefficients for different losses, thereby improving the classification accuracy of the target detection model for target categories, improving the matching degree of text and image features, and optimizing the performance of the target detection model.

[0164] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0165] Figure 5This is a schematic diagram of a target detection device provided in an embodiment of this disclosure. Figure 5 As shown, the target detection device includes:

[0166] The acquisition module 501 is used to acquire the image to be recognized and the query text, wherein the query text is used to query the target object in the image to be recognized that corresponds to the query text;

[0167] The first processing module 502 is used to perform image recognition on the image to be recognized, and obtain image description features and region detection visual features;

[0168] The second processing module 503 is used to perform region multimodal fusion processing on image description features and region detection visual features to obtain region multimodal fusion features;

[0169] The third processing module 504 is used to perform feature fusion processing on the text features and region multimodal fusion features obtained based on the query text to obtain the text region fusion features corresponding to the query text.

[0170] The fourth processing module 505 is used to obtain target detection results based on text features and text region fusion features, wherein the target detection results represent the position of the target object in the image to be identified and the category of the target object.

[0171] According to the technical solution provided in this disclosure, a visual large language model is used to perform semantic parsing on the image to be recognized, obtaining image description features containing object attributes and spatial relationships. Furthermore, a DETR encoder can extract the region detection visual features of the image to be recognized. A CLIP text encoder can map the query text to the semantic vector space to obtain text features. Then, a correlation matrix between the image description features and the region visual features can be constructed through cross-modal similarity calculation. Nonlinear transformation can be performed using MLP to obtain region multimodal features. Text features can be used as attention queries and cross-attention calculations can be performed with the region multimodal features. Text region fusion features are generated through hierarchical attention architecture and cross-level attention weight fusion. Finally, text features can be concatenated with learnable query vectors and cross-attention calculations can be performed with the text region fusion features to obtain the target detection result. This enhances the semantic alignment capability of multimodal features, improves the robustness of text-image association in complex scenarios, and enhances the accuracy and generalization performance of cross-modal target detection.

[0172] In some embodiments, the fourth processing module 505 is specifically used to perform hierarchical fusion processing on the text region fusion features to obtain the target fusion features corresponding to the query text; and to perform multimodal decoding processing on the text features and the target fusion features to obtain the target detection result.

[0173] In some embodiments, the second processing module 503 is specifically used to: perform similarity calculation processing on image description features and region detection visual features to obtain an association similarity value; process the image description features based on the association similarity value to obtain visual enhancement features; and perform feature fusion processing on the visual enhancement features and region detection visual features to obtain region multimodal fusion features.

[0174] In some embodiments, the hierarchical fusion processing of text region fusion features to obtain target fusion features corresponding to the query text specifically involves: downsampling the text region fusion features to obtain a coarse-grained feature map, and obtaining a fine-grained feature map based on the text region fusion features; performing global attention processing on the coarse-grained feature map to obtain global coarse-grained features; performing local attention processing on the fine-grained feature map to obtain fine-grained features; calculating attention weights on the global coarse-grained features and fine-grained features to obtain hierarchical attention weight data, wherein the hierarchical attention weight data represents the correlation and / or similarity between the global coarse-grained features and fine-grained features; weighting the fine-grained features based on the hierarchical attention weight data to obtain fine-grained weighted features; and fusing the fine-grained weighted features and global coarse-grained features to obtain the target fusion feature.

[0175] In some embodiments, multimodal decoding of text features and target fusion features to obtain target detection results is specifically used to: obtain a learnable query vector; concatenate the learnable query vector and text features to obtain concatenated features corresponding to the text features; and perform cross-attention processing on the concatenated features and target fusion features to obtain target detection results.

[0176] In some embodiments, the global attention processing of the coarse-grained feature map to obtain global coarse-grained features is specifically performed by: performing feature serialization processing on the coarse-grained feature map to obtain serialized features; performing multi-head attention processing on the serialized features to obtain attention features; and performing residual connection processing on the attention features to obtain global coarse-grained features.

[0177] Figure 6 This is a schematic diagram of a target detection training device provided in an embodiment of this disclosure. Figure 6 As shown, the target detection training device includes:

[0178] The first training processing module 601 is used to perform image recognition on the image samples to be recognized, and obtain training image description features and training region detection visual features.

[0179] The second training processing module 602 is used to perform multimodal fusion processing on the training image description features and the training region detection visual features to obtain multimodal fusion features of the training region.

[0180] The third training processing module 603 is used to perform feature fusion processing on the training text features and training region multimodal fusion features obtained based on the query text samples to obtain the training text region fusion features corresponding to the query text samples.

[0181] The fourth training processing module 604 is used to obtain the target detection prediction result based on the training text features and the training text region fusion features. The target detection prediction result represents the position of the target object corresponding to the query text sample in the image sample to be identified and the category of the target object.

[0182] The fifth training processing module 605 is used to obtain the total loss value based on the error between the object detection prediction result and the true label;

[0183] The sixth training processing module 606 is used to adjust the model parameters of the object detection model based on the total loss value.

[0184] In some embodiments, the total loss value is obtained based on the error between the object detection prediction result and the true label. Specifically, this involves: calculating the cross-entropy loss between the category of the sample target object and the true classification label to obtain the classification loss value corresponding to the object detection model; calculating the regression loss between the position of the sample target object in the image sample to be identified and the true bounding box coordinates to obtain the bounding box regression loss value corresponding to the object detection model; calculating the similarity comparison loss between the object detection prediction result and the training text features to obtain the comparison loss value corresponding to the object detection model; and obtaining the total loss value based on the classification loss value, the bounding box regression loss value, and the comparison loss value.

[0185] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0186] Figure 7 This is a schematic diagram of the electronic device 7 provided in an embodiment of this disclosure. Figure 7 As shown, the electronic device 7 of this embodiment includes a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 703, it implements the steps in the various method embodiments described above. Alternatively, when the processor 701 executes the computer program 703, it implements the functions of each module / unit in the various device embodiments described above.

[0187] Electronic device 7 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 7 may include, but is not limited to, processor 701 and memory 702. Those skilled in the art will understand that... Figure 7 This is merely an example of electronic device 7 and does not constitute a limitation on electronic device 7. It may include more or fewer components than shown, or different components.

[0188] The processor 701 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0189] The memory 702 can be an internal storage unit of the electronic device 7, such as a hard disk or RAM of the electronic device 7. The memory 702 can also be an external storage device of the electronic device 7, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 7. The memory 702 can also include both internal and external storage units of the electronic device 7. The memory 702 is used to store computer programs and other programs and data required by the electronic device.

[0190] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0191] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0192] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.

Claims

1. A target detection method, characterized in that, include: Obtain an image to be identified and a query text, wherein the query text is used to query the target object in the image to be identified that corresponds to the query text; Image recognition is performed on the image to be identified to obtain image description features and region detection visual features; The image description features and the region detection visual features are subjected to region multimodal fusion processing to obtain region multimodal fusion features; The text features obtained based on the query text and the region multimodal fusion features are subjected to feature fusion processing to obtain the text region fusion features corresponding to the query text; The target detection result is obtained based on the text features and the text region fusion features, wherein the target detection result represents the position of the target object in the image to be identified and the category of the target object.

2. The target detection method according to claim 1, characterized in that, The method of obtaining the target detection result based on the text features and the text region fusion features includes: The text region fusion features are subjected to hierarchical fusion processing to obtain the target fusion features corresponding to the query text; The text features and the target fusion features are subjected to multimodal decoding processing to obtain the target detection result.

3. The target detection method according to claim 1, characterized in that, The step of performing region multimodal fusion processing on the image description features and the region detection visual features to obtain region multimodal fused features includes: The image description features and the region detection visual features are processed to calculate the similarity, and an association similarity value is obtained. The image description features are processed based on the association similarity value to obtain visual enhancement features; The visual enhancement features and the region detection visual features are fused together to obtain the region multimodal fusion features.

4. The target detection method according to claim 2, characterized in that, The step of performing hierarchical fusion processing on the text region fusion features to obtain the target fusion features corresponding to the query text includes: The text region fusion features are downsampled to obtain a coarse-grained feature map, and a fine-grained feature map is obtained based on the text region fusion features. Global attention processing is applied to the coarse-grained feature map to obtain global coarse-grained features; Local attention processing is applied to the fine-grained feature map to obtain fine-grained features; Attention weights are calculated on the global coarse-grained features and the fine-grained features to obtain inter-level attention weight data, wherein the inter-level attention weight data represents the correlation and / or similarity between the global coarse-grained features and the fine-grained features; The fine-grained features are weighted based on the inter-layer attention weight data to obtain fine-grained weighted features; The fine-grained weighted features and the global coarse-grained features are fused to obtain the target fused features.

5. The target detection method according to claim 2, characterized in that, The step of performing multimodal decoding on the text features and the target fusion features to obtain the target detection result includes: Obtain learnable query vectors; The learnable query vector and the text feature are concatenated to obtain the concatenated feature corresponding to the text feature. The spliced ​​features and the target fusion features are subjected to cross-attention processing to obtain the target detection result.

6. The target detection method according to claim 4, characterized in that, The step of performing global attention processing on the coarse-grained feature map to obtain global coarse-grained features includes: The coarse-grained feature map is subjected to feature serialization processing to obtain serialized features; The serialized features are subjected to multi-head attention processing to obtain attention features; The attention features are subjected to residual connection processing to obtain the global coarse-grained features.

7. A training method for an object detection model, characterized in that, include: Obtain training samples, wherein the training samples include image samples to be identified and query text samples, wherein the query text samples are used to query the target object in the image samples to be identified that corresponds to the query text samples; The training samples are input into the target detection model to be trained for the following training steps to obtain a trained target detection model: Image recognition is performed on the image samples to be identified to obtain training image description features and training region detection visual features; The training image description features and the training region detection visual features are subjected to region multimodal fusion processing to obtain the training region multimodal fusion features; The training text features obtained based on the query text sample and the training region multimodal fusion features are subjected to feature fusion processing to obtain the training text region fusion features corresponding to the query text sample; The target detection prediction result is obtained based on the training text features and the training text region fusion features, wherein the target detection prediction result represents the position of the target object corresponding to the query text sample in the image sample to be identified and the category of the target object; Based on the error between the target detection prediction result and the true label, the total loss value is obtained; Based on the total loss value, the model parameters of the target detection model are adjusted.

8. The training method for the target detection model according to claim 7, characterized in that, The real labels include real category labels and real bounding box coordinates; The total loss value is obtained based on the error between the target detection prediction result and the true label, including: The cross-entropy loss is calculated by performing cross-entropy loss on the category of the target object in the sample and the true classification label to obtain the classification loss value corresponding to the target detection model. The regression loss is calculated by performing regression loss processing on the position of the target object in the image sample to be identified and the coordinates of the true bounding box to obtain the bounding box regression loss value corresponding to the target detection model; The object detection prediction result and the training text features are compared using a similarity comparison loss calculation to obtain the comparison loss value corresponding to the object detection model. The total loss value is obtained based on the classification loss value, the bounding box regression loss value, and the contrast loss value.

9. A target detection device, characterized in that, include: The acquisition module is used to acquire an image to be identified and a query text, wherein the query text is used to query the target object in the image to be identified that corresponds to the query text; The first processing module is used to perform image recognition on the image to be recognized, and obtain image description features and region detection visual features; The second processing module is used to perform region multimodal fusion processing on the image description features and the region detection visual features to obtain region multimodal fusion features; The third processing module is used to perform feature fusion processing on the text features obtained based on the query text and the region multimodal fusion features to obtain the text region fusion features corresponding to the query text. The fourth processing module is used to obtain a target detection result based on the text features and the text region fusion features, wherein the target detection result represents the position of the target object in the image to be identified and the category of the target object.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images