Object detection method and device, computer equipment and storage medium

Abstract

The invention relates to an object detection method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps of obtaining a to-be-detected target image and text information used for describing a target detection object; extracting visual features of the target image and text features of the text information; performing feature enhancement processing on the other one of the visual feature and the text feature based on the visual feature and the text feature to obtain a text enhanced visual feature of the target image and a visual enhanced text feature of the text information; fusing the visual enhanced text features and the text enhanced visual features to obtain fusion features of the target image corresponding to the target detection object; and determining a detection result of the target image corresponding to the target detection object based on the fusion feature. By adopting the method, the closed set limitation of current object recognition can be broken through, and the recognition generalization ability of object detection in an open environment is improved.

Description

Technical Field

[0001] This application relates to the field of cross-modal object detection technology, and in particular to an object detection method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] Object detection is one of the core research directions in the field of computer vision, and it has broad application prospects in autonomous driving, intelligent monitoring, industrial quality inspection, robot navigation, and other fields. Traditional methods mainly rely on machine learning algorithms based on hand-designed features (such as HOG and SIFT), but their feature representation capabilities are limited, and their robustness to complex scenes such as target deformation, scale changes, and background clutter is poor.

[0003] In recent years, with breakthroughs in deep learning technology, models based on convolutional neural networks and recurrent neural networks have become the mainstream solutions for object detection. While these methods can improve recognition accuracy, they are essentially "closed-set" recognition paradigms. Their recognition ability is strictly limited to objects seen during the training phase and cannot generalize to new, unlearned objects, meaning their system scalability is poor. Furthermore, these methods are highly sensitive to the quality of visual input, and in practical deployments, their performance can easily degrade drastically due to complex factors such as occlusion in videos, drastic changes in lighting, cluttered backgrounds, and camera shake. Therefore, how to overcome the current closed-set limitations of object recognition and improve the generalization ability of object detection in open environments has become an urgent problem to be solved. Summary of the Invention

[0004] Therefore, it is necessary to provide an object detection method, apparatus, computer device, and computer-readable storage medium that can overcome the limitations of closed sets in current object recognition and improve the generalization ability of object detection in open environments, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides an object detection method, which includes:

[0006] Acquire the target image to be detected, and the text information used to describe the target object;

[0007] Extract visual features from the target image and textual features from the text information;

[0008] By performing feature enhancement processing on one of the visual features and the text features, we can obtain the text-enhanced visual features of the target image and the visual-enhanced text features of the text information.

[0009] By fusing visually enhanced text features and text-enhanced visual features, a fused feature is obtained that corresponds the target image to the target detection object.

[0010] Based on the fusion features, the detection result of the target image corresponding to the target detection object is determined.

[0011] Secondly, this application also provides an object detection device, the device comprising:

[0012] The information acquisition module is used to acquire the target image to be detected, as well as text information describing the target object to be detected;

[0013] The feature extraction module is used to extract the visual features of the target image and the text features of the text information.

[0014] The cross-modal interaction module is used to perform feature enhancement processing on one of the visual features and the text features, to obtain the text-enhanced visual features of the target image and the visual-enhanced text features of the text information.

[0015] The feature fusion module is used to fuse visually enhanced text features and text-enhanced visual features to obtain the fused features of the target image corresponding to the target detection object.

[0016] The object detection module is used to determine the detection result of the target image corresponding to the target object based on the fused features.

[0017] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0018] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0019] The aforementioned object detection methods, apparatuses, computer devices, storage media, and computer program products, when needing to detect target objects in an image to be detected, first acquire the target image to be detected and text information describing the target object. They then extract visual features from the target image and text features from the text information. Subsequently, based on one of the visual features and text features, they perform feature enhancement processing on the other, obtaining text-enhanced visual features of the target image and visually enhanced text features of the text information. These visually enhanced text features and text-enhanced visual features are then fused to obtain a fused feature corresponding to the target object in the target image. Based on this fused feature, the detection result corresponding to the target object in the target image is determined. This object detection method, by enhancing one of the visual features and text features on the other, can achieve the effects of text-enhanced images and image-enhanced text. Finally, object detection is performed based on the fused feature combining visually enhanced text features and text-enhanced visual features. This not only improves detection accuracy but also allows the introduction of natural language descriptions as query conditions. It enables flexible detection of new objects not seen during training without retraining the model, breaking through the limitations of closed sets and improving the generalization ability of object detection in open environments. Attached Figure Description

[0020] Figure 1 This is a diagram illustrating the application environment of an object detection method in one embodiment;

[0021] Figure 2 This is a flowchart illustrating an object detection method in one embodiment;

[0022] Figure 3 This is a flowchart illustrating the process of enhancing visual features with text features to obtain text-enhanced visual features of a target image in one embodiment.

[0023] Figure 4 This is a flowchart illustrating the process of enhancing text features using visual features to obtain visually enhanced text features in one embodiment.

[0024] Figure 5 This is a schematic diagram of the feature fusion model in one embodiment;

[0025] Figure 6 This is a flowchart illustrating the process of determining the detection result of a target image corresponding to a target detection object based on a task detection strategy that matches the detection task type and fusion features in one embodiment.

[0026] Figure 7 This is a schematic diagram of the overall process of a personnel behavior detection method in one embodiment;

[0027] Figure 8This is a flowchart illustrating the object detection method in another embodiment;

[0028] Figure 9 This is a structural block diagram of an object detection device in one embodiment;

[0029] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0031] The object detection method provided in this application embodiment can be applied to, for example, Figure 1 The application environment shown.

[0032] The object detection system 102 communicates with the server 104 via a network. The data storage system can store the data that the object detection system 102 needs to process.

[0033] The data storage system can be integrated into the object detection system 102, or it can be placed in the cloud or on other network servers.

[0034] The object detection system 102 can obtain the target image to be detected and the text information used to describe the target object from the server 104. It extracts the visual features of the target image and the text features of the text information. Based on one of the visual features and the text features, it performs feature enhancement processing on the other to obtain the text-enhanced visual features of the target image and the visual-enhanced text features of the text information. It fuses the visual-enhanced text features and the text-enhanced visual features to obtain the fused features of the target image corresponding to the target object. Based on the fused features, it determines the detection result of the target image corresponding to the target object.

[0035] The object detection system 102 can be integrated into the user terminal or into the server 104.

[0036] Understandably, different application scenarios for object detection methods require different user terminals. For example, in industrial production scenarios, the user terminal could be a management terminal used by production managers. These managers can use the integrated object detection system 102 on the management terminal to detect the behavior of production personnel, such as whether they are wearing the required work clothes. As another example, in autonomous driving scenarios, the user terminal could be an in-vehicle terminal integrated within the vehicle. Users can use the integrated object detection system 102 on the in-vehicle terminal to receive key information about surrounding vehicles, pedestrians, traffic signs, and other information.

[0037] Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers. Similarly, the server 104 can vary depending on the application scenario. For example, in an industrial production scenario, server 104 can be a production management server; in an autonomous driving scenario, server 104 can be a cloud simulation server.

[0038] In one embodiment, such as Figure 2 As shown, an object detection method is provided, which is applied to... Figure 1 Taking the object detection system 102 in the example, the following steps are included:

[0039] S202, acquire the target image to be detected and the text information used to describe the target object to be detected.

[0040] In this context, the target image serves as the visual foundation and processing object in the entire object detection task, and it can be obtained by acquiring images of the area to be detected. For example, when it is necessary to detect the behavior of production workers in an industrial production area, images of the industrial production area can be acquired to obtain the target image to be detected.

[0041] Understandably, the target image can be a single image acquired specifically for the target area, or it can be a frame from a video sequence acquired from video captures of the area to be detected. The target image may contain one or more scene objects and rich scene information.

[0042] In this context, the target detection object is the specific object that is expected to be detected in the target image. Understandably, the target detection object can be a category object, such as a general category like "person," "cat," or "car"; or it can be a behavioral object, such as a specific behavioral object with attributes or spatial relationships, like "a person wearing red clothes" or "a person not wearing a helmet."

[0043] Textual information used to describe the target detection object refers to information obtained by describing the target detection object through natural language or structured tags. For example, natural language query statements used to describe the target detection object. The textual information, in the form of text, clearly defines the specific target of the object detection task.

[0044] Taking the target object as an example, the text information can be a natural language query statement used to describe global behavior, such as "The person in the picture is not wearing a safety helmet", or it can be a natural language query statement used to describe the behavior of a specific person, such as "The person wearing blue clothes is making and receiving phone calls while working".

[0045] Specifically, the object detection system can acquire the target image to be detected, as well as text information describing the target object.

[0046] In one embodiment, both the target image and text information can be uploaded to the server by the user of the object detection system according to the actual detection needs. The object detection system can obtain the target image to be detected and the text information used to describe the target object by communicating with the server.

[0047] In one embodiment, the object detection system can communicate with an image acquisition device located in the area to be detected. When it receives an object detection instruction for the area to be detected, it acquires a target detection image of the area to be detected in real time from the image acquisition device, and at the same time obtains text information describing the target detection object through the server.

[0048] S204, extract the visual features of the target image and the text features of the text information.

[0049] The visual features of the target image are feature data extracted from the target image that represent its visual information. They encode information such as edges, textures, shapes, colors, and more complex object components and overall spatial structures. Understandably, visual features can be represented as feature vectors or matrices.

[0050] Among them, textual features of textual information are digital representations extracted from the description of textual information that can characterize its semantic connotation, encoding the meaning of words, grammatical structure, and the overall semantics of the entire phrase or sentence.

[0051] Specifically, after obtaining the target image and text information, the object detection system can perform feature extraction operations on the target image and text information respectively to obtain the visual features of the target image and the text features of the text information.

[0052] In one embodiment, after obtaining the target image, the object detection system can first perform image preprocessing operations on the target image to adjust its size so that it conforms to the input size required by the model. Then, the preprocessed target image is input into a pre-trained image feature encoding model, which encodes the target image based on the image feature encoding model to obtain the visual features of the target image.

[0053] In one embodiment, the image preprocessing operation may be to perform operations such as standardization and scaling on the target image, and the input size required by the image feature coding model may be 224×224 or 384×384.

[0054] In one embodiment, the image feature encoding model can be obtained using a deep convolutional neural network (CNN) or a vision transformer (ViT) as the backbone network. The deep convolutional neural network can be a residual network (ResNet), an augmented residual network (ResNeXt), etc. The vision transformer can be a sliding window transformer (Swin Transformer).

[0055] In one embodiment, the visual features of the target image can be a spatial feature map F_v ∈ R^(H×W×C). This feature map preserves the spatial structure information of the image, where each feature vector (HxW in total) corresponds to a local region (receptive field) in the original image, containing the visual information of that region.

[0056] In one embodiment, after obtaining text information describing the target object, the object detection system can first perform tokenization on the text information and add necessary special tags to form a word sequence. For example, special tags include classification tags [CLS] and separator tags [SEP]. The classification tag is always inserted at the absolute beginning of the sequence, gathering the semantic information of the entire sequence to represent the overall meaning of the whole sentence or sentence pair. The separator tag is placed at the end of the entire sequence to mark the boundaries of the input data, clarify the internal structure of the sequence, and help the model understand the boundaries and relationships between different segments.

[0057] After obtaining the word sequence, the object detection system inputs the word sequence into a pre-trained text feature encoding model. The text feature encoding model uses a self-attention mechanism to fully understand the contextual semantics of the text information, encodes the word sequence, and obtains the text features of the text information.

[0058] In one embodiment, the text features can be a sequence of text features formed by the context embedding vector of each word, F_t ∈ R^(M×D_t), where D_t is the dimension of the text features.

[0059] In one embodiment, the text feature encoding model can be a pre-trained model from the BERT series.

[0060] S206, based on one of the visual features and the text features, perform feature enhancement processing on the other to obtain the text-enhanced visual features of the target image and the visual-enhanced text features of the text information.

[0061] Feature enhancement processing utilizes feature information from one modality to enhance the features of another, enriching them with more contextual information. This feature enhancement, based on either visual or textual features, is not a simple concatenation process, but rather a bidirectional, interactive information injection process. Its core objective is to enable visual features to learn to heed textual instructions, while simultaneously allowing textual features to understand the context of an image.

[0062] Text-enhanced visual features of a target image refer to the enhanced visual features formed after the original visual features are guided and filtered by the semantic information of the text features, resulting in more discriminative visual features. Compared to the visual features of the target image, text-enhanced visual features retain only the parts of the image most relevant to the text description, making the target clearer and reducing noise.

[0063] Visually enhanced text features refer to the more precise enhanced text features formed by incorporating specific contextual information provided by visual features into the original text features. Compared to the text features of textual information, visually enhanced text features have adjusted their meaning based on the specific visual image, making the description more specific and less ambiguous.

[0064] Specifically, the object detection system can perform feature enhancement processing on one of the visual features and the text features to obtain the text-enhanced visual features of the target image and the visual-enhanced text features of the text information.

[0065] In one embodiment, the object detection system can first use visual features to enhance text features to obtain visually enhanced text features of the text information, and then use the visually enhanced text features to enhance visual features to obtain text-enhanced visual features of the target image.

[0066] In one embodiment, the object detection system may first use text features to enhance visual features to obtain text-enhanced visual features of the target image, and then use the text-enhanced visual features to enhance text features to obtain visually enhanced text features of the text information.

[0067] S208, fuse visually enhanced text features and text-enhanced visual features to obtain the fused features of the target image corresponding to the target detection object.

[0068] The fusion of visually enhanced text features and text-enhanced visual features refers to the process of integrating these features to form a targeted joint feature. The fusion feature of the target detection object is the final output of the entire fusion process. The fusion feature is a spatial-semantic joint feature that has been precisely tuned by the text semantics and fully visualized by the visual context.

[0069] Specifically, after obtaining visually enhanced text features and text-enhanced visual features, the object detection system can fuse these features to obtain fused features of the target image corresponding to the target object.

[0070] In one embodiment, the object detection system has a pre-set feature fusion model. After obtaining visually enhanced text features and text-enhanced visual features, the visually enhanced text features and text-enhanced visual features can be input into the feature fusion model to obtain the fused features of the target image corresponding to the target detection object.

[0071] S210, Based on the fusion features, determine the detection result of the target image corresponding to the target detection object.

[0072] Among them, the detection result of the target object is the object localization result, which can reflect whether the target object exists in the target image, or the specific location of the target object in the target image.

[0073] Specifically, the object detection system can perform object detection operations on the target image based on fused features to determine the detection result of the target image corresponding to the target object.

[0074] In one embodiment, the detection result of the target object is related to the specific detection task of the target object. It can be understood that different detection tasks of the target object will result in different output detection results.

[0075] In the object detection method described above, when target object detection is required on an image to be detected, the target image to be detected and text information describing the target object are first acquired. Visual features of the target image and text features of the text information are extracted. Then, feature enhancement processing is performed on one of the visual features and text features to obtain text-enhanced visual features of the target image and visually enhanced text features of the text information. These visually enhanced text features and text-enhanced visual features are fused to obtain the fused features corresponding to the target object in the target image. Based on the fused features, the detection result corresponding to the target object in the target image is determined. This object detection method, by enhancing one of the visual features and text features, can achieve the effects of text-enhanced images and image-enhanced text. Finally, object detection is performed based on the fused features that combine visually enhanced text features and text-enhanced visual features. This not only improves detection accuracy but also allows the introduction of natural language descriptions as query conditions. It enables flexible detection of new objects not seen during training without retraining the model, breaking through the limitations of closed sets and improving the generalization ability of object detection in open environments.

[0076] In one embodiment, S206, feature enhancement processing is performed on one of the visual features and the text features to obtain text-enhanced visual features of the target image and visually enhanced text features of the text information. This includes: performing feature enhancement processing on the text features using the visual features to obtain visually enhanced text features of the text information; and performing feature enhancement processing on the visual features using the text features to obtain text-enhanced visual features of the target image.

[0077] Specifically, after obtaining the visual features of the target image and the text features of the text information, the object detection system can perform feature enhancement processing on the text features using the visual features to obtain visually enhanced text features of the text information, and simultaneously perform feature enhancement processing on the visual features using the text features to obtain text-enhanced visual features of the target image. It can be understood that the feature enhancement processing for text features and visual features can be considered a parallel feature enhancement process.

[0078] In the above embodiments, visually enhanced text features are obtained by performing feature enhancement processing on text features through visual features, and text-enhanced visual features are obtained by performing feature enhancement processing on visual features through text features. This allows the information of the two modalities to no longer be independent, but to perform semantic alignment at the pixel level and word level, thereby realizing cross-modal bidirectional interactive information injection, and providing a data foundation for accurately locating the target object described in the text in the target image.

[0079] In one embodiment, such as Figure 3 As shown, text-enhanced visual features of the target image are obtained by performing feature enhancement processing on visual features using text features, including:

[0080] S302, Based on visual features, determine the first query matrix for the target image.

[0081] Among them, visual features include visual sub-features of each spatial region in the target image, and each visual sub-feature can reflect the visual information of the corresponding spatial region.

[0082] In one embodiment, the visual sub-feature can be a spatial feature vector.

[0083] The first query matrix, derived from visual features, is used to proactively initiate queries within the attention mechanism. In this mechanism, the query is proactive, and the first query matrix can be viewed as an intentional representation of the visual features of all image regions.

[0084] Specifically, the object detection system can first flatten the visual features into a sequence, and then perform a linear transformation on the flattened sequence of visual features through a learnable first weight matrix to obtain a first query matrix for the target image.

[0085] S304, Based on text features, determine the first key matrix and the first value matrix for the target image.

[0086] Among them, text features include the context features of each word in the text information, and the context features can be context embedding vectors.

[0087] The first key matrix and the first value matrix for the target image are derived from text features and are used to provide the queried information in the attention mechanism. The key matrix acts as an index to the information, indicating what type of information is stored at each position in the query matrix. The value matrix represents the actual information content; once a relevant key is found, the corresponding value is the content that is actually extracted and transmitted.

[0088] Specifically, the object detection system can linearly transform text features using a learnable second weight matrix to obtain the first key matrix of the target image. Then, it can linearly transform the text features using a learnable third weight matrix to obtain the first value matrix for the target image.

[0089] S306, by performing weight calculations using the first query matrix and the first key matrix, the first attention weight matrix corresponding to the target image and the target detection object is obtained.

[0090] The first attention weight matrix is ​​a probability distribution matrix that intuitively quantifies the association strength between each spatial location in the image and each word in the text. Therefore, the first attention weight matrix can be used to characterize the degree of attention given to each word for each spatial region.

[0091] The weight calculation is essentially a scaling dot product attention operation, which quantifies the correlation between visual and text elements through a query-key matching mechanism.

[0092] Specifically, the object detection system can transpose and multiply the first query matrix and the first key matrix to obtain an original similarity matrix. Then, after dividing the original similarity matrix by a scaling factor, it performs normalization processing to obtain the first attention weight matrix corresponding to the target image and the target detection object.

[0093] In one embodiment, the scaling factor can be the square root of the key vector dimension.

[0094] S308: By aggregating the first value matrix through the first attention weight matrix, the text-enhanced visual features of the target image are obtained.

[0095] The aggregation of the first value matrix through the first attention weight matrix is ​​essentially a process of using attention weights as guidance to extract the most relevant semantic content from the text information and assign it to the corresponding image region. This is equivalent to a weighted summation of all word information in the value matrix, with the weights provided by the first attention weight matrix.

[0096] Specifically, the object detection system can aggregate the first value matrix using the first attention weight matrix to obtain the text-enhanced visual features of the target image.

[0097] In one embodiment, the object detection system can multiply a first attention weight matrix with a first value matrix to obtain text-enhanced visual features of the target image.

[0098] In the above embodiments, by using a collaborative attention mechanism, relevant semantic content is extracted from the text information and accurately assigned to the corresponding spatial region in the image. This generates an enhanced visual feature that is enriched and guided by the text semantics, providing a data foundation for the subsequent generation of fused features.

[0099] In one embodiment, such as Figure 4 As shown, text features are enhanced through visual features to obtain visually enhanced text features, including:

[0100] S402, Based on text features, determine the second query matrix for the text information.

[0101] Among them, text features include the contextual features of each word in the text information.

[0102] The second query matrix, derived from text features, is used to proactively initiate queries within the attention mechanism. It can be viewed as a visual query representation of all lexical semantics.

[0103] Specifically, the object detection system can first perform a linear transformation on the text features through a learnable first weight matrix to obtain a second query matrix for the text information.

[0104] S404, based on visual features, determines the second key matrix and the second value matrix for text information.

[0105] Among them, visual features include the visual sub-features of each spatial region in the target image.

[0106] Among them, the second key matrix and the second value matrix for text information are transformed from visual features and are used to provide two matrices for the queried information in the attention mechanism.

[0107] Specifically, the object detection system can first reshape visual features into a sequence, then perform a linear transformation on the reshaped sequence of visual features using a learnable second weight matrix to obtain a second key matrix for the text information. Simultaneously, the reshaped sequence of visual features is linearly transformed using a learnable third weight matrix to obtain a second value matrix for the text information.

[0108] S406, by performing weight calculations using the second query matrix and the second key matrix, the second attention weight matrix corresponding to the target detection object for the text information is obtained.

[0109] The second attention weight matrix is ​​a probability distribution matrix that runs in the opposite direction to the first attention weight matrix. It quantifies the degree of association between each word in the text and each spatial region in the image. Therefore, the second attention weight matrix can be used to characterize the degree of attention paid to each word corresponding to each spatial region. For example, for the word "helmet," attention will be paid to the area of ​​the person's head in the image; for "blue clothes," attention will be paid to the area of ​​the person's upper garment.

[0110] Specifically, the object detection system can transpose and multiply the second query matrix and the second key matrix to obtain an original similarity matrix. Then, after dividing the original similarity matrix by a scaling factor, it performs normalization processing to obtain the second attention weight matrix corresponding to the target object of the text information.

[0111] S408, by aggregating the second value matrix through the second attention weight matrix, obtains the visually enhanced text features of the text information.

[0112] The process of aggregating the second value matrix through the second attention weight matrix is ​​a process of extracting relevant visual evidence from image information and assigning it to corresponding text words, using the text-to-visual attention weights as guidance.

[0113] Specifically, the object detection system can aggregate the second value matrix through the second attention weight matrix to obtain visually enhanced text features of the text information.

[0114] In one embodiment, the object detection system can multiply the second attention weight matrix with the second value matrix to obtain visually enhanced text features of the text information.

[0115] In the above embodiments, by using a collaborative attention mechanism, relevant visual evidence is extracted from image information and precisely injected into the corresponding lexical features in the text. This generates an enhanced text feature that is confirmed and enriched by the visual context, providing a data foundation for the subsequent generation of fused features.

[0116] The above embodiments illustrate the generation process of visually enhanced text features and text-enhanced visual features, as well as the fusion process between the two. Since text and image regions are often interconnected, in one embodiment, the generation and fusion process of the enhanced features can be implemented using a dual-coordinated attention feature fusion model. This feature fusion model, mounted on an object detection system, can capture the multimodal feature interaction relationships between images and text. Its structure is as follows... Figure 5 As shown, the feature fusion model is obtained by parallel connection of collaborative attention blocks, which are a variant of the standard multi-head self-attention (MSA) block, and can capture the global dependencies of all positions in the sequence.

[0117] The MSA block is a composite unit containing two basic components: a multi-head self-attention function and a fully connected feedforward network. Both parts use residual connections and are then normalized. The input to the MSA is first used to compute (d×1) dimensional queries, keys, and values, which are then fed into matrices Q, K, and V, respectively. The similarity of the dot products between Q and K determines the attention distribution on V. A multi-head attention function with m heads has m parallel self-attention functions. For the i-th head, the input is transformed by Q, K, and V, as shown in the following formula:

[0118] .

[0119] in, is the projection matrix of the i-th head, and is the dimension of the output features of each head.

[0120] The calculation process of the multi-head self-attention function is shown in the following formula:

[0121] .

[0122] in, , It represents the connection between vectors.

[0123] A fully connected feedforward network consists of two linear transformations with a ReLU activation function embedded in between, as shown in the following formula:

[0124] .

[0125] Wherein, the input and output dimensions are The inner dimension is .

[0126] The Co-Attn (Co-Attn) block is an extension of the MSA (Multi-Head Attention) block. In a Co-Attn block, the query comes from one modality, while the key and value come from another. Specifically, the query matrix is ​​used as a residual term after the multi-head attention sublayer. The rest of the architecture is the same as MSA. The Co-Attn block takes features from one modality as input and produces attention-pooled features from another modality. Similarly, if Q comes from text and K and V comes from accompanying images, the attention values ​​computed using Q and K can be used as a measure of text-image similarity, which is then weighted towards the images. This captures the interdependencies between features from different modalities.

[0127] The CA layer is obtained by connecting two Co-Attn blocks in parallel. Given different properties for the two Co-Attn blocks, the CA layer computes the query, key, and value for each Co-Attn block, just as in the MSA block. The key and value of one Co-Attn block are then passed as input to the other Co-Attn block. The CA layer simulates the dense interactions between the input modes by exchanging information between them. The outputs of the two Co-Attn blocks are concatenated and then fed back into a fully connected layer to obtain a fused representation, as shown in the following formula:

[0128] .

[0129] in, This represents the multimodal feature representation of different modal features after fusion. For mapping functions, These are the training parameters for feature fusion.

[0130] In one embodiment, S210, determining the detection result of the target image corresponding to the target detection object based on the fusion features includes: obtaining the detection task type of the target detection object, and determining the detection result of the target image corresponding to the target detection object based on the task detection strategy matching the detection task type and the fusion features.

[0131] The detection task type is a parameter used to classify object detection tasks. Different task types correspond to different output formats and evaluation criteria. Understandably, detection task types can include image-text matching tasks, language-based visual localization tasks, panoptic segmentation tasks, etc.

[0132] In this context, the task detection strategy refers to the task processing flow and output processing method customized for a specific task type. For example, task detection strategies may include bounding box detection strategies, segmentation mask strategies, and directional detection strategies. The detection result is the final output format customized according to the task type.

[0133] Specifically, when determining the detection result of the target object corresponding to the target image based on fusion features, the object detection system can first obtain the detection task type of the target object, then determine the task detection strategy that matches the detection task type based on the detection task type, and use the task detection strategy and fusion features to perform object detection on the target image to obtain the detection result of the target object corresponding to the target image.

[0134] In one embodiment, the object detection system can determine the detection task type of the target object based on the object identifier of the target object, and then find the task detection strategy that matches the detection task type based on the preset correspondence between the task type and the detection strategy.

[0135] In the above embodiments, by designing corresponding task detection strategies for each specific detection task type, the matching degree between the output detection results and the user's actual detection needs can be improved, while the versatility of the fusion features can be improved. That is, a fusion feature can serve multiple downstream tasks, and only the final task detection strategy needs to be changed.

[0136] In one embodiment, the fusion features include global fusion features. Based on the task detection strategy matching the detection task type and the fusion features, the detection result of the target image corresponding to the target detection object is determined, including: when the detection task type is an image-text matching task, the global fusion features are input into a preset classifier to obtain the confidence scores of the target image and text information.

[0137] Among them, the global fusion feature refers to the single vector identifier obtained after aggregating the entire fusion feature, which represents the overall correlation between the image and the text.

[0138] In one embodiment, the object detection system takes the fused feature corresponding to the [CLS] tag as the global joint representation F_fused_global, i.e., the global fused feature.

[0139] The Image-Text Matching (ITM) task is a binary classification task that determines whether an image and text describe each other and are semantically related. The preset classifier is a lightweight classification module specifically trained to determine the image-text matching degree, and its role is to map high-dimensional fused features to matching probabilities.

[0140] The confidence score is a scalar value between 0 and 1, used to characterize the degree of matching between the target image and the text information. It can quantitatively identify the semantic consistency between the image and the text. A confidence score of 0 indicates that the target image and the text information do not match at all, while a confidence score of 1 indicates that the target image and the text information match perfectly.

[0141] Specifically, when the object detection system determines that the detection task is an image-text matching task, it can input the global fusion features into a preset classifier. The preset classifier converts the global fusion features into a global vector, then learns the decision boundary for image-text matching based on the global vector, and then uses Sigmoid activation to ensure that the output confidence score is within the probability range, finally obtaining the confidence score of the target image and text information.

[0142] In the above embodiments, when the detection task type is image-text matching task, the complex spatial relationship can be simplified into an overall semantic relationship by using global fusion features. Then, an objective matching score is given by a preset classifier, thereby quantitatively judging whether the image and text are related, which effectively improves the processing efficiency and accuracy of image-text matching task.

[0143] In another embodiment, the fusion feature includes spatial fusion features. For example... Figure 6 As shown, based on a task detection strategy that matches the detection task type and fusion features, the detection results of the target image corresponding to the target detection object are determined, including:

[0144] S602, when the detection task type is visual localization task, input the spatial fusion features into the preset heatmap generation model to generate an attention heatmap of the target image relative to the text information.

[0145] Among them, spatial fusion features refer to fusion features that retain complete spatial location information. When generating fusion features, the object detection system retains the spatial fusion features F_fused_spatial ∈R^(H×W×C') for the localization task.

[0146] Visual localization, also known as language-based grounding, involves accurately locating the target object in an image based on a textual description. A heatmap generation model is a lightweight network that transforms spatial fusion features into a probability distribution; its role is to learn the mapping from fusion features to spatial probability distributions.

[0147] The attention heatmap is a two-dimensional probability map that visually displays the relevance of each location in the image to the text description. It takes the form of a single-channel matrix, with high-value areas (highlighted areas) representing target regions highly matched to the text description, and low-value areas (dark areas) representing background or regions unrelated to the text description.

[0148] Specifically, when the object detection system determines that the detection task type is a visual localization task, it can input spatial fusion features into a pre-trained heatmap generation model, and generate an attention heatmap of the target image relative to the text information based on the heatmap generation model.

[0149] S604, perform binarization processing on the attention heatmap according to the preset pixel threshold to obtain a binary map.

[0150] The preset pixel threshold is a pre-set pixel threshold used to distinguish between the target object and the background. The preset pixel threshold ranges from 0 to 1 and can be flexibly adjusted according to specific application requirements.

[0151] Binarization is a threshold-based decision-making process that transforms continuous probability estimation into deterministic binary classification. A binary image is a black-and-white image containing only 0 and 1 values. By binarizing the attention heatmap according to a preset pixel threshold, target and background regions can be clearly distinguished. For example, a value of 1 indicates the target region, while a value of 0 indicates the background region. The binary image is a two-dimensional matrix with the same size as the heatmap.

[0152] Specifically, the object detection system can compare each pixel in the attention heatmap with a preset pixel threshold. If the pixel value in the heatmap is greater than the preset pixel threshold, the value of the corresponding pixel in the binary image is set to 1. If the pixel value in the heatmap is less than or equal to the preset pixel threshold, the value of the corresponding pixel in the binary image is set to 0, thereby obtaining the binary image corresponding to the attention heatmap.

[0153] S606: Find connected components in a binary graph and set a rectangle for each connected component.

[0154] In a binary graph, a connected region is a continuous area consisting of all interconnected white pixels. Each connected region can be understood as representing an independent candidate region for a target. The bounding box for each connected region is the smallest bounding rectangle that completely encloses the corresponding region. The rectangle can be flush with the boundary of the connected region and completely contain all white pixels within that region.

[0155] Specifically, the object detection system searches for connected components in a binary graph and sets a bounding box for each connected component.

[0156] In one embodiment, a label matrix of the same size as the binary image is created with an initial value of 0. The current label number is set to 1. Then, each pixel of the binary image is scanned. When a white pixel is encountered that is not labeled, the pixel is used as a seed point for connected component labeling. Starting from the seed point, adjacent pixels in a preset direction are checked. If an adjacent pixel is white and not labeled, it is added to the current region until no new white pixels can be added to the current region. At this point, all pixels in the current connected component are labeled with the current label number. The object detection system can increment the label number to label the next connected component, and so on.

[0157] S608: For each rectangle, determine the confidence level of the rectangle based on the heat values ​​of each pixel in the corresponding connected region.

[0158] The heat value of each pixel refers to the probability value corresponding to each white pixel position in the connected region on the attention heatmap. The heat value represents the original confidence that the pixel belongs to the target object.

[0159] The confidence level of the bounding box is a scalar value between 0 and 1, which comprehensively evaluates the reliability of the bounding box in containing the real target object, and its function is to quantify the quality of the detection results.

[0160] Specifically, for each bounding box, the object detection system can determine the confidence level of the bounding box based on the heat values ​​of each pixel in the connected region corresponding to the bounding box.

[0161] Understandably, determining the confidence level of a bounding box is a statistical aggregation process.

[0162] In one embodiment, the object detection system can use the maximum heatmap value of all pixels within a connected region as the confidence score of the corresponding bounding box. This method yields a confidence score that reflects the strength of the most significant evidence, is insensitive to region size, and is simple and fast to calculate.

[0163] In one embodiment, the object detection system can calculate the arithmetic mean of the heatmap values ​​of all pixels within a connected region and use this as the confidence score for the corresponding bounding box. The confidence score obtained in this way reflects the overall strength of evidence, is insensitive to outliers, and exhibits good stability.

[0164] S610 outputs the highest confidence score among all confidence scores, and the target bounding box corresponding to the highest confidence score.

[0165] Specifically, after obtaining each confidence level, the object detection system can determine the maximum confidence level from among the confidence levels and output the maximum confidence level and the target location box corresponding to the maximum confidence level.

[0166] In the above embodiments, when the detection task type is a visual localization task, a visualized attention distribution map is generated by using fusion features that retain spatial information. Then, based on the attention distribution map, the target localization box and the corresponding confidence score for locating the target object are accurately output, which can improve the localization efficiency and accuracy of the target detection object.

[0167] In one embodiment, the model training process is a supervised end-to-end process, and the model loss function can be configured according to the task. For example, in the case of an image-text matching task, binary cross-entropy (BCE) loss can be used. In the case of a localization task, regression loss (such as GIoU Loss + L1 Loss) can be used to measure the difference between the predicted bounding box and the ground truth bounding box, or cross-entropy loss can be used to supervise heatmap generation.

[0168] In one embodiment, when the task is image-text matching, an image-text contrast loss (ITC Loss) can be introduced to learn matching and non-matching pairs simultaneously in a batch, thereby enhancing the model's discrimination and alignment capabilities.

[0169] In one embodiment, an object detection method is provided. Taking the application of this method in a human behavior detection scenario as an example, the overall process is as follows: Figure 7As shown, after the camera captures a frame of the scene image in real time, the scene image can be input into the image feature encoding model, i.e., ResNet50, and the text information "a person not wearing a safety helmet" can be input into the text feature encoding model, i.e., BERT. Visual features and text features are extracted based on the image feature encoding model and the text feature encoding model, respectively, and the visual features and text features are adjusted to the same dimension for feature fusion. Then, the collaborative attention block in the feature fusion model is used to perform cross-attention calculation on the visual features and text features to obtain text-enhanced visual features and visual-enhanced text features. Fine-grained feature fusion is performed on the text-enhanced visual features and visual-enhanced text features to obtain fused features. The fused features are input into the localization task module to perform behavior detection and determine whether the person is wearing a safety helmet. If the person is not wearing a safety helmet, an alarm is triggered; if the person is wearing a safety helmet, no alarm is triggered.

[0170] like Figure 8 As shown, the object detection method specifically includes the following steps:

[0171] S801, acquire the target image to be detected, and text information used to describe the behavior of the target person.

[0172] S802, extract the visual features of the target image and the text features of the text information.

[0173] S803, Based on the visual features, determine a first query matrix for the target image.

[0174] S804, Based on the text features, determine the first key matrix and the first value matrix for the target image.

[0175] S805, by performing weight calculations using the first query matrix and the first key matrix, a first attention weight matrix corresponding to the target person's behavior in the target image is obtained.

[0176] S806, the first value matrix is ​​aggregated through the first attention weight matrix to obtain the text-enhanced visual features of the target image.

[0177] S807, Based on the text features, determine a second query matrix for the text information.

[0178] S808, based on the visual features, determine the second key matrix and the second value matrix for the text information.

[0179] S809, by performing weight calculations using the second query matrix and the second key matrix, a second attention weight matrix is ​​obtained that corresponds to the text information in relation to the target person's behavior.

[0180] S810, the second value matrix is ​​aggregated through the second attention weight matrix to obtain the visually enhanced text features of the text information.

[0181] S811, the visually enhanced text features and the text-enhanced visual features are fused to obtain the fused features of the target image corresponding to the target person's behavior.

[0182] S812, the spatial fusion features are input into a preset heatmap generation model to generate an attention heatmap of the target image relative to the text information.

[0183] S813, the attention heatmap is binarized according to a preset pixel threshold to obtain a binary image.

[0184] S814, find connected components in the binary graph and set a rectangle for each connected component.

[0185] S815, for each of the rectangular frames, the confidence level of the rectangular frame is determined based on the thermal values ​​of each pixel in the corresponding connected region of the rectangular frame.

[0186] S816, output the maximum confidence score among the given confidence scores, and the target bounding box corresponding to the maximum confidence score.

[0187] The above embodiments aim to address the problems of traditional single-vision models in behavior recognition, such as closed categories, poor generalization ability, and insufficient adaptability to complex scenes. By creatively integrating and aligning natural language semantics with visual spatiotemporal features, it achieves dynamic and accurate querying and locating of behavior instances in videos using text commands. Ultimately, this improves the system's recognition generalization ability in open environments, the flexibility in defining unknown behaviors, and the robustness of detection in complex scenes.

[0188] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0189] Based on the same inventive concept, this application also provides an object detection apparatus for implementing the object detection method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more object detection apparatus embodiments provided below can be found in the limitations of the object detection method described above, and will not be repeated here.

[0190] In one embodiment, such as Figure 9 As shown, an object detection device 900 is provided, including: an information acquisition module 901, a feature extraction module 902, a cross-modal interaction module 903, a feature fusion module 904, and an object detection module 905, wherein:

[0191] The information acquisition module 901 is used to acquire the target image to be detected and the text information describing the target detection object.

[0192] The feature extraction module 902 is used to extract the visual features of the target image and the text features of the text information.

[0193] The cross-modal interaction module 903 is used to perform feature enhancement processing on one of the visual features and the text features, to obtain the text-enhanced visual features of the target image and the visual-enhanced text features of the text information.

[0194] The feature fusion module 904 is used to fuse visually enhanced text features and text-enhanced visual features to obtain the fused features of the target image corresponding to the target detection object.

[0195] The object detection module 905 is used to determine the detection result of the target image corresponding to the target detection object based on the fused features.

[0196] In one embodiment, the cross-modal interaction module 903 is used to: perform feature enhancement processing on text features through visual features to obtain visually enhanced text features of text information; and perform feature enhancement processing on visual features through text features to obtain text-enhanced visual features of the target image.

[0197] In one embodiment, the cross-modal interaction module 903 is used to: determine a first query matrix for a target image based on visual features; the visual features include visual sub-features of each spatial region in the target image; determine a first key matrix and a first value matrix for the target image based on text features; the text features include context features of multiple words in the text information; calculate weights using the first query matrix and the first key matrix to obtain a first attention weight matrix for the target image corresponding to the target detection object; the first attention weight matrix is ​​used to characterize the degree of attention of each spatial region corresponding to each word; and aggregate the first value matrix using the first attention weight matrix to obtain the text-enhanced visual features of the target image.

[0198] In one embodiment, the cross-modal interaction module 903 is used to: determine a second query matrix for text information based on text features; the text features include the context features of each word in the text information; determine a second key matrix and a second value matrix for the text information based on visual features; the visual features include the visual sub-features of each spatial region in the target image; calculate weights through the second query matrix and the second key matrix to obtain a second attention weight matrix for the text information corresponding to the target detection object; the second attention weight matrix is ​​used to characterize the degree of attention of each word corresponding to each spatial region; and aggregate the second value matrix through the second attention weight matrix to obtain the visually enhanced text features of the text information.

[0199] In one embodiment, the object detection module 905 is used to: obtain the detection task type of the target object; and determine the detection result of the target image corresponding to the target object based on the task detection strategy matching the detection task type and the fusion features.

[0200] In one embodiment, the fused features include global fused features. The object detection module 905 is used to: when the detection task type is an image-text matching task, input the global fused features into a preset classifier to obtain a confidence score between the target image and the text information; the confidence score is used to characterize the degree of matching between the target image and the text information.

[0201] In one embodiment, the fusion features include spatial fusion features. The object detection module 905 is configured to: when the detection task type is a visual localization task, input the spatial fusion features into a preset heatmap generation model to generate an attention heatmap of the target image relative to the text information; binarize the attention heatmap according to a preset pixel threshold to obtain a binary image; find connected components in the binary image and set a bounding box for each connected component; for each bounding box, determine the confidence level of the bounding box based on the heatmap values ​​of each pixel in the corresponding connected region; and output the maximum confidence level and the target localization box corresponding to the maximum confidence level.

[0202] Each module in the aforementioned object detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0203] In one embodiment, a computer device is provided, which may be a server integrating an object detection system, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to the object detection method. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an object detection method.

[0204] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0205] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the specific implementation steps of the object detection method described above.

[0206] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the specific implementation steps of the object detection method described above.

[0207] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the specific implementation steps of the object detection method described above.

[0208] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the acquisition, storage, processing, and transmission of the data all comply with relevant laws and regulations.

[0209] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0210] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0211] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An object detection method, characterized in that, The method includes: Acquire the target image to be detected, and the text information used to describe the target object; Extract the visual features of the target image and the text features of the text information; Based on one of the visual features and the text features, feature enhancement processing is performed on the other to obtain the text-enhanced visual features of the target image and the visually enhanced text features of the text information; By fusing the visually enhanced text features and the text-enhanced visual features, a fused feature of the target image corresponding to the target detection object is obtained; Based on the fusion features, the detection result of the target image corresponding to the target detection object is determined.

2. The method according to claim 1, characterized in that, The step of performing feature enhancement processing on one of the visual features and the text features to obtain the text-enhanced visual features of the target image and the visually enhanced text features of the text information includes: The text features are enhanced by performing feature enhancement processing on the visual features to obtain visually enhanced text features of the text information. By performing feature enhancement processing on the visual features using the text features, the text-enhanced visual features of the target image are obtained.

3. The method according to claim 2, characterized in that, The step of performing feature enhancement processing on the visual features using the text features to obtain the text-enhanced visual features of the target image includes: Based on the visual features, a first query matrix is ​​determined for the target image; the visual features include visual sub-features for each spatial region in the target image. Based on the text features, a first key matrix and a first value matrix are determined for the target image; the text features include the context features of each word in the text information. By performing weight calculations using the first query matrix and the first key matrix, a first attention weight matrix is ​​obtained corresponding to the target image and the target detection object; the first attention weight matrix is ​​used to characterize the degree of attention of each spatial region to each word. By aggregating the first value matrix using the first attention weight matrix, the text-enhanced visual features of the target image are obtained.

4. The method according to claim 2, characterized in that, The step of performing feature enhancement processing on the text features using the visual features to obtain visually enhanced text features of the text information includes: Based on the text features, a second query matrix is ​​determined for the text information; the text features include the context features of each word in the text information. Based on the visual features, a second key matrix and a second value matrix are determined for the text information; the visual features include visual sub-features of each spatial region in the target image. By performing weight calculations using the second query matrix and the second key matrix, a second attention weight matrix is ​​obtained corresponding to the target detection object in the text information; the second attention weight matrix is ​​used to characterize the degree of attention of each word element to each spatial region. The visually enhanced text features of the text information are obtained by aggregating the second value matrix using the second attention weight matrix.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the detection result of the target image corresponding to the target detection object based on the fusion features includes: Obtain the detection task type of the target object; Based on the task detection strategy that matches the detection task type and the fusion features, the detection result of the target image corresponding to the target detection object is determined.

6. The method according to claim 5, characterized in that, The fusion features include global fusion features; The determination of the detection result of the target image corresponding to the target detection object based on the task detection strategy matching the detection task type and the fusion features includes: When the detection task type is an image-text matching task, the global fusion features are input into a preset classifier to obtain a confidence score between the target image and the text information; the confidence score is used to characterize the degree of matching between the target image and the text information.

7. The method according to claim 5, characterized in that, The fusion features include spatial fusion features; determining the detection result of the target image corresponding to the target detection object based on the task detection strategy matching the detection task type and the fusion features includes: When the detection task type is a visual localization task, the spatial fusion features are input into a preset heatmap generation model to generate an attention heatmap of the target image relative to the text information. The attention heatmap is binarized according to a preset pixel threshold to obtain a binary image. Find connected components in the binary graph and set a rectangle for each connected component; For each of the rectangular frames, the confidence level of the rectangular frame is determined based on the thermal values ​​of each pixel in the corresponding connected region of the rectangular frame. Output the highest confidence score among all the confidence scores, and the target bounding box corresponding to the highest confidence score.

8. An object detection device, characterized in that, The device includes: The information acquisition module is used to acquire the target image to be detected, as well as text information describing the target object to be detected; The feature extraction module is used to extract the visual features of the target image and the text features of the text information; A cross-modal interaction module is used to perform feature enhancement processing on one of the visual features and the text features to obtain the text-enhanced visual features of the target image and the visual-enhanced text features of the text information. The feature fusion module is used to fuse the visually enhanced text features and the text-enhanced visual features to obtain the fused features of the target image corresponding to the target detection object; An object detection module is used to determine the detection result of the target image corresponding to the target detection object based on the fused features.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images