Target detection method and device based on reference features, storage medium and electronic device

Abstract

The invention provides a target detection method and device based on reference features, a storage medium and an electronic device.The method comprises the steps that a reference image of a marked target and text description information of the reference image are collected; extracting reference features of the reference image and the text description information; generating an image feature sequence of a multi-scale feature map by adopting the reference image and the reference features; the image feature sequence and the reference feature are adopted to construct a query decoding network of the reference feature, the reference feature corresponds to the target category of the annotation target, and the image feature sequence corresponds to a bounding box; and calling the query decoding network to carry out image detection on the annotation target. According to the method and the device, the technical problem of poor flexibility of a target detection model in related technologies is solved, the method and the device have higher practicability in practical application, and the development cost of new target identification can be effectively reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a target detection method and device based on reference features, a storage medium, and an electronic device. Background Art

[0002] Intelligent recognition devices are increasingly used in modern technology. These devices use visual sensors to acquire image information and perform object recognition tasks. Traditional intelligent recognition devices often rely on large, precisely labeled training datasets to ensure accuracy and reliability when performing computer vision tasks such as classification or segmentation. However, when detecting unknown objects, collecting large-scale, high-quality labeled data covering all potential categories is extremely challenging, or even impossible. This not only increases the cost of data preparation but also limits the scope of application of intelligent recognition devices.

[0003] To overcome this limitation, researchers are focusing on developing technologies that can effectively identify unknown categories. This progress is crucial for expanding the application scenarios of intelligent recognition devices, especially in areas that require a high degree of autonomy and adaptability. Currently, improved intelligent recognition technology has demonstrated great potential in several key areas, such as early warning of abnormal behavior in security monitoring, understanding and responding to the environment for autonomous vehicles, and real-time control of product quality in industrial environments. These advances indicate that intelligent recognition devices are gradually becoming a key driving force for innovation and efficiency improvements across various industries.

[0004] The target recognition models in related technologies have certain limitations in practical applications: Lack of flexibility: This method essentially divides target detection into two stages, known and unknown, rather than an end-to-end architecture, which to a certain extent affects its flexibility. Dependence on CLIP image-text matching: Since the detection accuracy depends largely on the image-text matching capabilities of CLIP, the detection speed is slow in application scenarios with high real-time requirements (such as lawn mowers, autonomous driving, etc.), and cannot meet the needs of rapid response. In summary, although this method provides valuable exploration for solving the problem of unknown category detection, in the pursuit of efficient and real-time application scenarios, further optimization is still needed to overcome existing challenges.

[0005] For the above-mentioned problems existing in related technologies, no efficient and accurate solutions have been found yet. Summary of the Invention

[0006] The present invention provides a target detection method and device based on reference features, a storage medium, and an electronic device to solve the above-mentioned technical problems existing in the related art.

[0007] According to one embodiment of the present invention, a method for target detection based on reference features is provided, comprising: acquiring a reference image of a labeled target and text description information of the reference image; extracting reference features of the reference image and the text description information; generating an image feature sequence of a multi-scale feature map using the reference image and the reference features; constructing a query decoding network for the reference features using the image feature sequence and the reference features, wherein the reference features correspond to the target category of the labeled target and the image feature sequence corresponds to a bounding box; and calling the query decoding network to perform image detection on the labeled target.

[0008] Optionally, extracting reference features of the reference image and the text description information includes: calling a trained comparative language image pre-trained CLIP model, and embedding the reference image and the text description information into the same semantic space through comparative learning of the CLIP model; calling an image editor and a text editor of the CLIP model to perform feature extraction on the reference image and the text description information respectively in the semantic space to obtain reference features.

[0009] Optionally, using the reference image and the reference features to generate an image feature sequence of a multi-scale feature map includes: using a lightweight network structure to extract a first feature map, a second feature map, and a third feature map of the reference image at multiple levels, wherein the resolution of the first feature map is higher than that of the second feature map, the resolution of the second feature map is higher than that of the third feature map, the amount of semantic information of the first feature map is lower than that of the second feature map, and the amount of semantic information of the second feature map is lower than that of the third feature map; and performing feature fusion on the reference features, the first feature map, the second feature map, and the third feature map to obtain an image feature sequence of a multi-scale feature map.

[0010] Optionally, feature fusion is performed on the reference feature, the first feature map, the second feature map, and the third feature map to obtain an image feature sequence of a multi-scale feature map, including: fusing the reference feature with the third feature map to obtain a first fused feature; upsampling the first fused feature and fusing it with the second feature map to obtain a second fused feature; upsampling the second fused feature and fusing it with the first feature map to obtain a third fused feature; downsampling the third fused feature and fusing it with the second feature map to obtain a fourth fused feature; downsampling the fourth fused feature and fusing it with the third feature map to obtain a fifth fused feature; splicing the third fused feature, the fourth fused feature, and the fifth fused feature into a multi-scale feature map; and flattening the multi-scale feature map into an image feature sequence.

[0011] Optionally, using the image feature sequence and the reference feature to construct a query decoding network for the reference feature includes: using a multi-layer multi-head attention module to perform a linear transformation on the reference feature to obtain a first feature vector; using a multi-head cross-attention module to perform a linear transformation on the first feature vector and the image feature sequence to obtain a second feature vector; using a feedforward neural network to construct the second feature vector and model parameters to obtain a query decoding network for the reference feature.

[0012] Optionally, before calling the query decoding network to perform image detection on the labeled target, the method further includes: constructing a loss function, wherein the loss function includes a classification loss function and a bounding box regression loss function, wherein the classification loss function is used to calculate the cross entropy loss between the predicted category probability and the true category label, and the bounding box regression loss function is used to calculate the generalized intersection-over-union loss GIoU and the mean absolute error L1 loss between the predicted bounding box coordinates and the true bounding box coordinates; and using the loss function and sample data to train the query decoding network until the model of the query decoding network converges.

[0013] Optionally, calling the query decoding network to perform image detection on the annotated target includes: obtaining an image frame of the annotated target, and extracting image features of the image frame through an encoder of the query decoding network; calling a decoder of the query decoding network to detect the target category and bounding box of the reference features in the image frame; and outputting the target category and the bounding box as a detection result.

[0014] According to another embodiment of the present invention, a reference feature-based target detection device is provided, comprising: an acquisition module for acquiring a reference image of a labeled target and text description information of the reference image; an extraction module for extracting reference features of the reference image and the text description information; a generation module for generating an image feature sequence of a multi-scale feature map using the reference image and the reference features; a first construction module for constructing a query decoding network of the reference features using the image feature sequence and the reference features, wherein the reference features correspond to the target category of the labeled target and the image feature sequence corresponds to a bounding box; and a detection module for calling the query decoding network to perform image detection on the labeled target.

[0015] Optionally, the extraction module includes: an embedding unit, used to call a trained comparative language image pre-trained CLIP model, and embed the reference image and the text description information into the same semantic space through comparative learning of the CLIP model; an extraction unit, used to call an image editor and a text editor of the CLIP model to perform feature extraction on the reference image and the text description information respectively in the semantic space to obtain reference features.

[0016] Optionally, the generation module includes: an extraction unit, used to use a lightweight network structure to extract the first feature map, the second feature map, and the third feature map of the reference image at multiple levels, wherein the resolution of the first feature map is higher than that of the second feature map, the resolution of the second feature map is higher than that of the third feature map, the amount of semantic information of the first feature map is lower than that of the second feature map, and the amount of semantic information of the second feature map is lower than that of the third feature map; a fusion unit, used to perform feature fusion on the reference feature, the first feature map, the second feature map, and the third feature map to obtain an image feature sequence of a multi-scale feature map.

[0017] Optionally, the fusion unit includes: a first fusion subunit, used to fuse the reference feature and the third feature map to obtain a first fusion feature; a second fusion subunit, used to upsample the first fusion feature and fuse it with the second feature map to obtain a second fusion feature; a third fusion subunit, used to upsample the second fusion feature and fuse it with the first feature map to obtain a third fusion feature; a fourth fusion subunit, used to downsample the third fusion feature and fuse it with the second feature map to obtain a fourth fusion feature; a fifth fusion subunit, used to downsample the fourth fusion feature and fuse it with the third feature map to obtain a fifth fusion feature; a splicing subunit, used to splice the third fusion feature, the fourth fusion feature, and the fifth fusion feature into a multi-scale feature map; and a flattening subunit, used to flatten the multi-scale feature map into an image feature sequence.

[0018] Optionally, the first construction module includes: a first transformation unit, used to use a multi-layer multi-head attention module to perform a linear transformation on the reference feature to obtain a first feature vector; a second transformation unit, used to use a multi-head cross-attention module to perform a linear transformation on the first feature vector and the image feature sequence to obtain a second feature vector; a construction unit, used to use a feedforward neural network to construct the second feature vector and model parameters to obtain a query decoding network of the reference feature.

[0019] Optionally, the device also includes: a second construction module, used to construct a loss function before the detection module calls the query decoding network to perform image detection on the labeled target, wherein the loss function includes a classification loss function and a bounding box regression loss function, wherein the classification loss function is used to calculate the cross entropy loss between the predicted category probability and the true category label, and the bounding box regression loss function is used to calculate the generalized intersection-over-union loss GIoU and the mean absolute error L1 loss between the predicted bounding box coordinates and the true bounding box coordinates; a training module, used to train the query decoding network using the loss function and sample data until the model of the query decoding network converges.

[0020] Optionally, the detection module includes: a processing unit, used to obtain the image frame of the labeled target and extract image features of the image frame through the encoder of the query decoding network; a detection unit, used to call the decoder of the query decoding network to detect the target category and bounding box of the reference feature in the image frame; and an output unit, used to output the target category and the bounding box as a detection result.

[0021] According to yet another embodiment of the present invention, a storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above-mentioned apparatus embodiments when run.

[0022] According to another embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above device embodiments.

[0023] According to the embodiment of the present invention, a reference image of a labeled target and text description information of the reference image are collected; reference features of the reference image and the text description information are extracted; an image feature sequence of a multi-scale feature map is generated using the reference image and the reference features; a query decoding network of the reference features is constructed using the image feature sequence and the reference features, wherein the reference features correspond to the target category of the labeled target and the image feature sequence corresponds to the bounding box; the query decoding network is called to perform image detection on the labeled target, and an open target detection algorithm based on reference features is implemented by providing the reference image and description information of the target. The query decoding network with a self-attention mechanism is used to capture long-distance dependencies in the image, and image detection of the labeled target can be quickly realized, thereby quickly adapting to the recognition task of the new scene, solving the technical problem of poor flexibility of the target detection model in the related art, having stronger practicality in actual application, and being able to effectively reduce the development cost of new target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 This is a hardware structure block diagram of a computer according to an embodiment of the present invention;

[0026] Figure 2 is a flow chart of a method for object detection based on reference features according to an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of a feature fusion module according to an embodiment of the present invention;

[0028] Figure 4 is a flow chart of open target detection according to an embodiment of the present invention;

[0029] Figure 5 4 is a structural block diagram of a target detection device based on reference features according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of this application. It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Example 1

[0033] The method embodiment provided in the first embodiment of the present application can be executed in a computer, a server, a security terminal or a similar computing device. Taking running on a computer as an example, Figure 1 This is a hardware structure diagram of a computer according to an embodiment of the present invention. Figure 1 As shown, the computer may include one or more ( Figure 1 Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data. Optionally, the above computer may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0034] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to a reference feature-based target detection method in an embodiment of the present invention. The processor 102 executes the computer program stored in the memory 104 to perform various functional applications and data processing, thereby implementing the above-mentioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories may be connected to the computer via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0035] The transmission device 106 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by a computer's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0036] In this embodiment, a target detection method based on reference features is provided. Figure 2 is a flow chart of a target detection method based on reference features according to an embodiment of the present invention. Figure 2As shown, the process includes the following steps:

[0037] Step S202: collecting a reference image of the marked object and text description information of the reference image;

[0038] For the labeled objects to be detected and identified, we collected open-source real-world image annotation data, such as COCO (Common Objects in Context) data. We also obtained reference images of the labeled objects and used multimodal text descriptions of the corresponding reference images. We used normalization and resized the images to (640*640*3).

[0039] Optionally, the annotation target can be a person, an object, an action, etc.

[0040] Step S204, extracting reference features of the reference image and the text description information;

[0041] Step S206, generating an image feature sequence of a multi-scale feature map using the reference image and the reference features;

[0042] Step S208, constructing a query decoding network of the reference feature using the image feature sequence and the reference feature, wherein the reference feature corresponds to the object category of the labeled object, and the image feature sequence corresponds to a bounding box;

[0043] The query decoding network of this embodiment is a reference feature query decoding network based on a multi-layer Transformer structure. By using a multi-layer Transformer structure, the reference features are interacted with the image feature sequence, which can accurately capture the semantic association between the reference features and the image feature sequence, improve detection accuracy, and generate target category and bounding box predictions.

[0044] Step S210: calling the query decoding network to perform image detection on the marked object.

[0045] Through the above steps, the reference image of the labeled target and the text description information of the reference image are collected; the reference features of the reference image and the text description information are extracted; the reference image and the reference features are used to generate an image feature sequence of a multi-scale feature map; the image feature sequence and the reference features are used to construct a query decoding network of the reference features, wherein the reference features correspond to the target category of the labeled target and the image feature sequence corresponds to the bounding box; the query decoding network is called to perform image detection on the labeled target. By providing the reference image and description information of the target, an open target detection algorithm based on reference features is realized. The query decoding network with a self-attention mechanism is used to capture long-distance dependencies in the image, which can quickly realize image detection of the labeled target, thereby quickly adapting to the recognition task of the new scene, solving the technical problem of poor flexibility of the target detection model in the related technology, having stronger practicality in actual application, and can effectively reduce the development cost of new target recognition.

[0046] In this embodiment, extracting reference features of the reference image and the text description information includes: calling a trained comparative language image pre-trained CLIP model, and embedding the reference image and the text description information into the same semantic space through comparative learning of the CLIP model; calling an image editor and a text editor of the CLIP model to perform feature extraction on the reference image and the text description information in the semantic space, respectively, to obtain reference features.

[0047] The CLIP (Contrastive Language-Image Pre-training) of this embodiment is a multimodal pre-training model that aims to establish associations between images and natural language through contrastive learning. The CLIP model is pre-trained using large-scale image-text alignment data, mapping images and text into the same feature space, thereby achieving cross-modal understanding and retrieval. The CLIP model consists of two components: an image encoder (Image Encoder): which typically uses a convolutional neural network (such as ResNet) or Vision Transformer (ViT) to extract feature representations of images; and a text encoder (Text Encoder): which converts text descriptions into high-dimensional vectors based on the Transformer architecture. During training, the goal of CLIP is to bring the feature vectors of matching image-text pairs closer together and push unmatched pairs further apart. The CLIP model uses a contrastive loss function (such as InfoNCE loss) to optimize the similarity between image and text features. CLIP's training data comes from images captured from the Internet and their corresponding text descriptions. In this way, CLIP can learn the semantic associations between images and text.

[0048] This embodiment uses the pre-trained CLIP model to embed images and text into the same semantic space through contrastive learning, thereby achieving a joint understanding of images and text. The CLIP image editor and text editor are used to extract features from the target image and target text information, respectively, to obtain reference features F.

[0049] In one embodiment of the present embodiment, using the reference image and the reference features to generate an image feature sequence of a multi-scale feature map includes: using a lightweight network structure to extract a first feature map, a second feature map, and a third feature map of the reference image at multiple levels, wherein the resolution of the first feature map is higher than that of the second feature map, the resolution of the second feature map is higher than that of the third feature map, the amount of semantic information of the first feature map is lower than that of the second feature map, and the amount of semantic information of the second feature map is lower than that of the third feature map; and performing feature fusion on the reference features, the first feature map, the second feature map, and the third feature map to obtain an image feature sequence of a multi-scale feature map.

[0050] Different feature maps can be extracted by building a basic feature extractor and using a lightweight network structure such as Moiblenet and Resnet. This network reduces the spatial dimension of the image while increasing the depth, thereby generating corresponding feature maps at different stages (S3, S4, and S5). S3: Low-level feature map (first feature map), with high resolution and rich detail information. S4: Mid-level feature map (second feature map), with moderate resolution, containing certain semantic information and detail information. S5: High-level feature map (third feature map), with low resolution and rich semantic information.

[0051] In one example, feature fusion is performed on the reference feature, the first feature map, the second feature map, and the third feature map to obtain an image feature sequence of a multi-scale feature map, including: fusing the reference feature with the third feature map to obtain a first fused feature; upsampling the first fused feature and fusing it with the second feature map to obtain a second fused feature; upsampling the second fused feature and fusing it with the first feature map to obtain a third fused feature; downsampling the third fused feature and fusing it with the second feature map to obtain a fourth fused feature; downsampling the fourth fused feature and fusing it with the third feature map to obtain a fifth fused feature; splicing the third fused feature, the fourth fused feature, and the fifth fused feature into a multi-scale feature map; and flattening the multi-scale feature map into an image feature sequence.

[0052] Figure 3: is a schematic diagram of a feature fusion module in an embodiment of the present invention. S3 to S5 correspond to the first feature map to the third feature map, respectively. fusion is a fusion operation, and c is a splicing operation. It includes a reference feature fusion module and a cross-scale feature fusion module. Feature fusion is achieved by constructing a reference feature fusion module, and cross-scale feature fusion is achieved by constructing a cross-scale feature fusion module. The network structure of the reference feature fusion module is composed of a single-layer Transformer editor. The reference feature is obtained and fused with the high-level feature map (the third feature map S5) through the attention mechanism, thereby realizing the connection of high semantic features and outputting the feature map (first fusion feature) F5 after interaction. The calculation formula is as follows:

[0053] F5=TransformerEncoder(S5)

[0054] The calculation formula of TransformerEncoder is:

[0055]

[0056] Where Q, K, and V are query, key, and value respectively. k It represents the feature dimension and T is the transpose symbol. It is obtained by linearly transforming S5 and F.

[0057] Q=F*W Q

[0058] K=F*W k

[0059] V=S5*W v

[0060] Among them, W Q ,W k ,W v is a learnable weight matrix.

[0061] Construct a cross-scale feature fusion module to fuse features of different scales and enhance the multi-scale target detection capability. A top-down path is adopted: starting from F5, cross-scale fusion is performed layer by layer with S4 (second feature map) and S3 (first feature map). The resolution of the high-level feature map is adjusted to the same level as the next level feature map through upsampling operation. Figure 1 The features are fused using fusion blocks (such as 1×1 convolution and RepBlock). Then the bottom-up path is continued: starting from the fused S3, it is fused with S4 and F5 layer by layer. The resolution of the low-level feature map is adjusted to the same level as the previous layer feature map through downsampling operation. Figure 1 Use the fusion block to fuse the features. Finally, output the fused multi-scale feature map F encoder, flatten the multi-scale feature map into an image feature sequence so that it can be input into the Transformer Decoder. These feature maps contain rich details and semantic information and can effectively detect multi-scale targets. F4 is the second fusion feature, F3 is the third fusion feature, F4 ′ The fourth fusion feature, F5 ′ is the fifth fusion feature, F encoder is a multi-scale feature map, and the calculation formula is as follows:

[0062] F4=FusionBlock(Upsample(F5)+S4)

[0063] F3=FusionBlock(Upsample(F4)+S3)

[0064] F4 ′ =FusionBlock(Downsample(F3)+S4)

[0065] F5 ′ =FusionBlock(Downsample(F4 ′ )+F5)

[0066] F encoder =Concat(Flatten(F3),Flatten(F4 ′ ),Flatten(F5 ′ ))

[0067] FusionBlock(x)=RepBlock(Conv 1x1 (x))

[0068] RepConv(x)=Conv 3x3 (x)+Conv 1x1 (x)

[0069] Flatten(F i )=reshape(F i ,(H i *W i ,C i ))i={3,4,5}

[0070] H, W, and C are dimension symbols, representing height (Height), width (Width), and number of channels (Channel), respectively.

[0071] Multi-scale fusion enables multi-scale detection capabilities, enabling simultaneous detection of both near and far targets, improving detection comprehensiveness. High detection accuracy can be maintained even in complex monitoring scenarios (such as lighting changes and occlusions).

[0072] In one embodiment of the present invention, using the image feature sequence and the reference feature to construct a query decoding network for the reference feature includes: using a multi-layer multi-head attention module to perform a linear transformation on the reference feature to obtain a first feature vector; using a multi-head cross-attention module to perform a linear transformation on the first feature vector and the image feature sequence to obtain a second feature vector; using a feedforward neural network to construct the second feature vector and model parameters to obtain a query decoding network for the reference feature.

[0073] Multi-head self-attention improves the model's ability to express and learn input data by processing multiple independent attention modules ("heads") in parallel. Queries, keys, and values come from the same sequence, capturing dependencies within the sequence. A multi-layer multi-head attention mechanism stacks multiple multi-head attention layers within the model to enhance the model's ability to learn complex patterns. Multi-head cross-attention uses queries from one sequence and keys and values from another sequence to capture alignment between the two sequences.

[0074] This embodiment uses a multi-layer multi-head attention mechanism and a multi-head cross attention mechanism structure to construct a query decoding network for reference features, converting the reference features and image feature sequences into target categories and bounding boxes. The formula is as follows:

[0075]

[0076]

[0077] CrossAttention(F,F encoder ,F encoder )=F attention , the feature vector F output by the cross attention mechanism attention Perform linear changes and use feedforward neural network to finally get F out , its size is (N, K, C), and the calculation formula is as follows:

[0078] F out =Relu(W1*F attention +b1)W2+b2

[0079] Where N is the batch size, K is the number of query vectors, C is the feature dimension, W1 and W2 are weight matrices, and b1 and b2 are bias terms.

[0080] This paper implements an end-to-end detection model, simplifying the model training and inference process and improving the system's usability. Through the Transformer architecture, it accurately captures the semantic associations between reference features and image features, improving detection accuracy, adapting to a variety of new scenarios and targets, and reducing the frequency of model updates and maintenance.

[0081] In this embodiment, before calling the query decoding network to perform image detection on the labeled target, it also includes: constructing a loss function, wherein the loss function includes a classification loss function and a bounding box regression loss function, wherein the classification loss function is used to calculate the cross entropy loss between the predicted category probability and the true category label, and the bounding box regression loss function is used to calculate the generalized intersection-over-union loss GIoU and the mean absolute error L1 loss between the predicted bounding box coordinates and the true bounding box coordinates; and using the loss function and sample data to train the query decoding network until the model of the query decoding network converges.

[0082] Optionally, construct classification and bounding box regression losses for model training. The classification loss function is used to calculate the cross entropy loss between the predicted class probability and the true class label, and the bounding box regression loss function is used to calculate the GIoU (Generalized Intersection over Union Loss) loss and L1 (Mean Absolute Error, MAE) loss between the predicted bounding box coordinates and the true bounding box coordinates. The calculation formula is as follows:

[0083]

[0084] Among them, y i,j is the true category label (0,1) of the j-th query vector of the i-th sample, P i,j is the predicted class probability of the jth query vector of the i-th sample. ij are the coordinates of the ground-truth bounding box of the j-th query vector for the i-th sample. L is the coordinate of the predicted bounding box of the jth query vector of the i-th sample. giou is the GIoU loss, which measures the degree of bounding box overlap. α and β are weight coefficients. N is the total number of samples, and k is the total number of query vectors.

[0085] The total loss is: L total =L cls +L box

[0086] During model training, the Adam algorithm was used to optimize model parameters. Image or text reference features were randomly used during training. The number of iterations was set to 5000, the initial learning rate was set to 0.001, and the weight decay was set to 0.0005. Every 50 iterations, the learning rate was decayed to 1 / 10 of its original value until the model converged.

[0087] In one embodiment of the present embodiment, calling the query decoding network to perform image detection on the annotated target includes: obtaining an image frame of the annotated target, and extracting image features of the image frame through an encoder of the query decoding network; calling a decoder of the query decoding network to detect the target category and bounding box of the reference features in the image frame; and outputting the target category and the bounding box as a detection result.

[0088] During the inference detection phase, by inputting a reference image or text description of the target, the system can quickly determine whether the target area exists in the image being detected. For open object detection, by providing a reference image or text description, the system can quickly identify the target area in the image. This eliminates the need to retrain data on unknown targets, allowing for rapid adaptation to new application scenarios. This enhances the scalability of on-device AI applications.

[0089] The solution of this embodiment is aimed at existing end-side AI target recognition, which cannot quickly adapt to the recognition of new scenarios due to the complexity of application scenarios. An open target detection algorithm based on reference features is proposed. Figure 4 This is a flowchart of an embodiment of the present invention for open target detection, including: S1: collecting training data, marking coordinate information, and obtaining feature description information of each target image and target text. S2: using a pre-trained CLIP encoder to perform feature encoding on the target image or text respectively, S3: using a lightweight convolutional neural network to build a feature extractor. Extract feature maps of different scales in the image. S4: constructing a reference feature fusion module, using the attention mechanism to perform feature interaction between the feature map of the last layer and the reference feature, and capture the semantic connection between high-level features. S5: constructing a cross-scale feature fusion module through top-down and bottom-up paths to achieve efficient fusion of features of different scales and convert them into image feature sequences. S6: constructing a reference feature query decoding network, using a multi-layer Transformer structure to convert the reference features and image feature sequences into target categories and bounding boxes, constructing category loss and border loss for model training, and realizing end-to-end target detection. S7: constructing category loss and border loss for model training. S8: realizing open target detection by providing reference target graphics or text information.

[0090] This embodiment targets complex end-side AI recognition scenarios. By providing a single target image or description information of the target image, it can quickly adapt to the target recognition task of the new scenario. It has stronger practicality in actual application and can effectively reduce the development cost of new target recognition.

[0091] In the solution of this embodiment, an open target detection algorithm based on reference features is proposed. By introducing target image or target text description information as reference features, it can quickly adapt to target recognition tasks in new scenarios. Technical details: Use the pre-trained CLIP encoder to encode the features of the target image or text to generate reference features. Through the reference feature fusion module, the attention mechanism is used to interact the reference features with the image features to capture high-level semantic information. Advantages: It solves the problem that traditional end-side AI target recognition cannot quickly adapt to new scenarios in complex scenarios. Through reference features, the model's recognition ability of new targets is significantly improved, reducing development costs.

[0092] The solution of this embodiment also designs an efficient cross-scale feature fusion module, which realizes the efficient fusion of multi-scale features through top-down and bottom-up paths. Technical details: A lightweight convolutional neural network is used to extract the multi-scale feature map of the image. Feature maps of different scales are fused through top-down and bottom-up paths to generate an image feature sequence. Advantages: The model's ability to detect multi-scale targets is enhanced, especially the recognition effect of small targets and distant targets in complex scenarios is significantly improved. It adopts a lightweight design and is suitable for the real-time inference needs of end-side devices.

[0093] The solution of this embodiment also constructs a reference feature query decoding network based on a multi-layer Transformer structure to convert reference features and image feature sequences into target categories and bounding boxes. Technical details: Using a multi-layer Transformer structure, reference features are interacted with image feature sequences to generate target category and bounding box predictions. End-to-end model training is performed using category loss and border loss. Advantages: End-to-end target detection is achieved, simplifying the model training and inference process. By introducing reference features, the generalization ability and practicality of the model in new scenarios are significantly improved.

[0094] In one application scenario, target detection is required within an intelligent security monitoring system. The system needs to be able to quickly identify newly emerging suspicious targets (such as new dangerous goods or suspicious behavior) and generate real-time alerts. When a new dangerous good (such as a new explosive) appears in the monitoring system, traditional target detection models require collecting large amounts of labeled data and retraining the model, which is time-consuming and labor-intensive. This proposal introduces reference features, enabling rapid adaptation to new scenarios with only a single image of the dangerous good or a text description (e.g., "cylindrical metal object with a red label"). The pre-trained CLIP encoder is used to encode the target image or text to generate reference features. Rapid adaptation to new scenarios: No model retraining is required; new targets can be identified simply by providing reference features. Reduced development costs: Data collection and annotation workload is reduced, significantly lowering development costs. Improved practicality: In practical applications, this approach enables rapid response to emerging threats, improving the real-time and reliability of security systems. In surveillance videos, targets can appear at varying scales, such as large objects at close range and small objects at a distance. This proposal uses a cross-scale feature fusion module to extract multi-scale feature maps from the image and efficiently fuse them through top-down and bottom-up approaches. For example, for small dangerous objects at a distance, low-level feature maps (high resolution) capture detailed information; for large dangerous objects at close range, high-level feature maps (low resolution) capture semantic information. Multi-scale detection capability: It can simultaneously detect objects at both near and far distances, improving detection comprehensiveness. Efficient computing: Its lightweight design makes it suitable for real-time operation on end-devices, meeting the real-time requirements of security systems. Strong adaptability to complex scenarios: It can maintain high detection accuracy even in complex surveillance scenarios (such as lighting changes and occlusions). During the detection process, the system needs to match reference features with image features in the surveillance video to determine the target category and location. This proposal uses a multi-layer Transformer structure to interact with the reference features and image feature sequences to generate target category and bounding box predictions. This simplifies the model training and inference process, improving the system's usability. High-precision matching: The Transformer structure can accurately capture the semantic associations between reference features and image features, improving detection accuracy. Strong generalization ability: Able to adapt to a variety of new scenarios and new goals, reducing the frequency of model updates and maintenance.

[0095] In the monitoring scene, a new type of explosive appears, whose reference feature is a "cylindrical metal object with a red label". The following is the workflow:

[0096] Reference feature generation: Provide a single image or text description of the explosive and use the CLIP encoder to generate reference features.

[0097] Multi-scale feature extraction: Use lightweight convolutional neural networks to extract multi-scale feature maps of surveillance video frames.

[0098] Reference feature fusion: Through the reference feature fusion module, reference features are interacted with image features to capture semantic associations.

[0099] Cross-scale feature fusion: Through the cross-scale feature fusion module, feature maps of different scales are fused to generate an image feature sequence.

[0100] Object detection: By querying the decoding network with reference features, the reference features and image feature sequences are converted into object categories and bounding boxes, and the detection results are output.

[0101] Real-time alarm: The system identifies the object as a new type of explosive and issues an alarm in real time.

[0102] The solution of this embodiment can be used to quickly adapt to new scenarios: new targets can be identified without retraining the model. Strong multi-scale detection capabilities: able to detect targets at both close and long distances. Efficient computing: suitable for real-time operation on end-side devices. High-precision matching: Accurately capture semantic associations through the Transformer structure. Reduced development costs: Reduce the workload of data collection and annotation. The technical features and advantages of the solution of this embodiment in the intelligent security monitoring system can significantly improve the real-time performance, practicality and generalization capability of the system, while reducing development costs.

[0103] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0104] Example 2

[0105] This embodiment also provides a reference feature-based target detection device for implementing the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated here. As used below, the term "module" may include a combination of software and hardware that implements the specified functionality. Although the devices described in the following embodiments are preferably implemented in software, implementations using hardware, or a combination of software and hardware, are also contemplated.

[0106] Figure 5 is a structural block diagram of a target detection device based on reference features according to an embodiment of the present invention. Figure 5 As shown, the device includes:

[0107] An acquisition module 50 is configured to acquire a reference image of a marked object and text description information of the reference image;

[0108] An extraction module 52, configured to extract reference features of the reference image and the text description information;

[0109] A generating module 54, configured to generate an image feature sequence of a multi-scale feature map using the reference image and the reference features;

[0110] A first construction module 56 is configured to construct a query decoding network for the reference features using the image feature sequence and the reference features, wherein the reference features correspond to the object category of the labeled object and the image feature sequence corresponds to a bounding box;

[0111] The detection module 58 is configured to call the query decoding network to perform image detection on the labeled object.

[0112] Optionally, the extraction module includes: an embedding unit, used to call a trained comparative language image pre-trained CLIP model, and embed the reference image and the text description information into the same semantic space through comparative learning of the CLIP model; an extraction unit, used to call an image editor and a text editor of the CLIP model to perform feature extraction on the reference image and the text description information respectively in the semantic space to obtain reference features.

[0113] Optionally, the generation module includes: an extraction unit, used to use a lightweight network structure to extract the first feature map, the second feature map, and the third feature map of the reference image at multiple levels, wherein the resolution of the first feature map is higher than that of the second feature map, the resolution of the second feature map is higher than that of the third feature map, the amount of semantic information of the first feature map is lower than that of the second feature map, and the amount of semantic information of the second feature map is lower than that of the third feature map; a fusion unit, used to perform feature fusion on the reference feature, the first feature map, the second feature map, and the third feature map to obtain an image feature sequence of a multi-scale feature map.

[0114] Optionally, the fusion unit includes: a first fusion subunit, used to fuse the reference feature and the third feature map to obtain a first fusion feature; a second fusion subunit, used to upsample the first fusion feature and fuse it with the second feature map to obtain a second fusion feature; a third fusion subunit, used to upsample the second fusion feature and fuse it with the first feature map to obtain a third fusion feature; a fourth fusion subunit, used to downsample the third fusion feature and fuse it with the second feature map to obtain a fourth fusion feature; a fifth fusion subunit, used to downsample the fourth fusion feature and fuse it with the third feature map to obtain a fifth fusion feature; a splicing subunit, used to splice the third fusion feature, the fourth fusion feature, and the fifth fusion feature into a multi-scale feature map; and a flattening subunit, used to flatten the multi-scale feature map into an image feature sequence.

[0115] Optionally, the first construction module includes: a first transformation unit, used to use a multi-layer multi-head attention module to perform a linear transformation on the reference feature to obtain a first feature vector; a second transformation unit, used to use a multi-head cross-attention module to perform a linear transformation on the first feature vector and the image feature sequence to obtain a second feature vector; a construction unit, used to use a feedforward neural network to construct the second feature vector and model parameters to obtain a query decoding network of the reference feature.

[0116] Optionally, the device also includes: a second construction module, used to construct a loss function before the detection module calls the query decoding network to perform image detection on the labeled target, wherein the loss function includes a classification loss function and a bounding box regression loss function, wherein the classification loss function is used to calculate the cross entropy loss between the predicted category probability and the true category label, and the bounding box regression loss function is used to calculate the generalized intersection-over-union loss GIoU and the mean absolute error L1 loss between the predicted bounding box coordinates and the true bounding box coordinates; a training module, used to train the query decoding network using the loss function and sample data until the model of the query decoding network converges.

[0117] Optionally, the detection module includes: a processing unit, used to obtain the image frame of the labeled target and extract image features of the image frame through the encoder of the query decoding network; a detection unit, used to call the decoder of the query decoding network to detect the target category and bounding box of the reference feature in the image frame; and an output unit, used to output the target category and the bounding box as a detection result.

[0118] Example 3

[0119] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0120] Optionally, in this embodiment, the storage medium may be configured to store a computer program for executing:

[0121] S1, collecting reference images of the marked objects and text description information of the reference images;

[0122] S2, extracting reference features of the reference image and the text description information;

[0123] S3, generating an image feature sequence of a multi-scale feature map using the reference image and the reference features;

[0124] S4, constructing a query decoding network of the reference features using the image feature sequence and the reference features, wherein the reference features correspond to the object category of the labeled object, and the image feature sequence corresponds to a bounding box;

[0125] S5: Call the query decoding network to perform image detection on the marked target.

[0126] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0127] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0128] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0129] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0130] S1, collecting reference images of the marked objects and text description information of the reference images;

[0131] S2, extracting reference features of the reference image and the text description information;

[0132] S3, generating an image feature sequence of a multi-scale feature map using the reference image and the reference features;

[0133] S4, constructing a query decoding network of the reference features using the image feature sequence and the reference features, wherein the reference features correspond to the object category of the labeled object, and the image feature sequence corresponds to a bounding box;

[0134] S5: Call the query decoding network to perform image detection on the marked target.

[0135] The electronic device provided in the embodiments of the present application may be a module capable of realizing communication functions or a terminal device including the module, etc. The terminal device may be a mobile terminal or a smart terminal. The mobile terminal may be at least one of a mobile phone, a tablet computer, a laptop computer, etc.; the smart terminal may be a terminal including a wireless communication module such as a smart car, a smart watch, a shared bicycle, a smart cabinet, etc.; the module may be a wireless communication module, such as any one of a 2G communication module, a 3G communication module, a 4G communication module, a 5G communication module, an NB-IOT communication module, etc.

[0136] Optionally, specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0137] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0138] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0140] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0141] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0142] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0143] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A target detection method based on reference features, characterized in that: include: Collecting reference images of the marked objects and text description information of the reference images; extracting reference features of the reference image and the text description information; generating an image feature sequence of a multi-scale feature map using the reference image and the reference features; constructing a query decoding network for the reference features using the image feature sequence and the reference features, wherein the reference features correspond to the object category of the labeled object and the image feature sequence corresponds to a bounding box; The query decoding network is called to perform image detection on the labeled target.

2. The method according to claim 1, characterized in that Extracting reference features of the reference image and the text description information includes: Calling a trained comparative language image pre-training CLIP model, and embedding the reference image and the text description information into the same semantic space through comparative learning of the CLIP model; The image editor and text editor of the CLIP model are called to perform feature extraction on the reference image and the text description information in the semantic space respectively to obtain reference features.

3. The method according to claim 1, characterized in that Generating an image feature sequence of a multi-scale feature map using the reference image and the reference features includes: A lightweight network structure is used to extract a first feature map, a second feature map, and a third feature map of the reference image at multiple levels, wherein the resolution of the first feature map is higher than that of the second feature map, the resolution of the second feature map is higher than that of the third feature map, the amount of semantic information of the first feature map is lower than that of the second feature map, and the amount of semantic information of the second feature map is lower than that of the third feature map; Feature fusion is performed on the reference feature, the first feature map, the second feature map, and the third feature map to obtain an image feature sequence of a multi-scale feature map.

4. The method according to claim 3, characterized in that Performing feature fusion on the reference feature, the first feature map, the second feature map, and the third feature map to obtain an image feature sequence of a multi-scale feature map, including: Fusing the reference feature and the third feature map to obtain a first fused feature; Upsampling the first fused feature and fusing it with the second feature map to obtain a second fused feature; Upsampling the second fused feature and fusing it with the first feature map to obtain a third fused feature; Downsampling the third fused feature and fusing it with the second feature map to obtain a fourth fused feature; Downsampling the fourth fused feature and fusing it with the third feature map to obtain a fifth fused feature; splicing the third fused feature, the fourth fused feature, and the fifth fused feature into a multi-scale feature map; The multi-scale feature map is flattened into an image feature sequence.

5. The method according to claim 1, characterized in that Constructing a query decoding network of the reference feature using the image feature sequence and the reference feature includes: Using a multi-layer multi-head attention module to perform a linear transformation on the reference feature to obtain a first feature vector; Using a multi-head cross attention module to perform a linear transformation on the first feature vector and the image feature sequence to obtain a second feature vector; A feedforward neural network is used to construct the second feature vector and model parameters to obtain a query decoding network of the reference feature.

6. The method according to claim 1, characterized in that Before calling the query decoding network to perform image detection on the labeled target, the method further includes: Constructing a loss function, wherein the loss function includes a classification loss function and a bounding box regression loss function, wherein the classification loss function is used to calculate the cross entropy loss between the predicted category probability and the true category label, and the bounding box regression loss function is used to calculate the generalized intersection-over-union loss (GIoU) and the mean absolute error (L1) loss between the predicted bounding box coordinates and the true bounding box coordinates; The query decoding network is trained using the loss function and sample data until a model of the query decoding network converges.

7. The method according to claim 1, characterized in that Calling the query decoding network to perform image detection on the labeled target includes: Acquire an image frame of the labeled target, and extract image features of the image frame through an encoder of the query decoding network; Invoking a decoder of the query decoding network to detect an object category and a bounding box of the reference feature in the image frame; The target category and the bounding box are output as a detection result.

8. A target detection device based on reference features, characterized in that: include: An acquisition module, configured to acquire reference images of the marked objects and text description information of the reference images; An extraction module, configured to extract reference features of the reference image and the text description information; A generating module, configured to generate an image feature sequence of a multi-scale feature map using the reference image and the reference features; a first construction module, configured to construct a query decoding network of the reference features using the image feature sequence and the reference features, wherein the reference features correspond to the object category of the labeled object and the image feature sequence corresponds to a bounding box; A detection module is used to call the query decoding network to perform image detection on the marked target.

9. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 7 when run.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 7.

Images