A target detection method, electronic device and computer readable storage medium

Abstract

The present application discloses a target detection method, an electronic device and a computer-readable storage medium, the target detection method comprising: obtaining a captured image and detection prompt information of a target object, the captured image comprising an object to be detected; determining a fusion feature of the captured image according to image features of the captured image and text features of the detection prompt information; performing detection processing on the captured image based on the fusion feature of the captured image, and obtaining target object information corresponding to the detection prompt information in the captured image. Thus, the accuracy of target detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a target detection method, an electronic device and a computer-readable storage medium. Background Art

[0002] With the rapid development of computer vision and deep learning technology, object detection methods have been widely used in various fields and have achieved significant breakthroughs. Traditional object detection methods detect objects in images based on pre-determined object categories. However, the real world is diverse and predefined categories usually cannot achieve full coverage, resulting in the inability to efficiently identify new objects that do not exist in the predefined categories.

[0003] Based on this, how to achieve open-world target detection has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a target detection method, an electronic device and a computer-readable storage medium, which can improve the accuracy of target detection.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: a target detection method is provided, the target detection method comprising: obtaining a captured image and detection prompt information of a target object, the captured image comprising an object to be detected; determining a fusion feature of the captured image based on image features of the captured image and text features of the detection prompt information; performing detection processing on the captured image based on the fusion features of the captured image, and obtaining target object information in the captured image corresponding to the detection prompt information.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above target detection method.

[0007] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a computer-readable storage medium including program data stored therein, wherein the program data is used to implement the above target detection method when executed by a processor.

[0008] Compared with the problem that traditional target detection can only detect trained object categories and cannot detect new targets that do not exist in predefined categories, the above scheme provides a target detection method, including: obtaining detection prompt information of a captured image and a target object, the captured image including the object to be detected; determining the fusion features of the captured image based on the image features of the captured image and the text features of the detection prompt information; performing detection processing on the captured image based on the fusion features of the captured image to obtain the target object information corresponding to the detection prompt information in the captured image. The introduction of the detection prompt information can realize the open set detection of the target, and the combination of the image features of the captured image and the text features of the detection prompt information enhances the interaction between the text and the image, thereby improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0010] Figure 1 is a flowchart of an exemplary embodiment of a target detection method shown in the present application;

[0011] Figure 2 yes Figure 1 FIG. 1 is a flow chart of an exemplary embodiment of step S120 in the target detection method;

[0012] Figure 3 is a schematic structural diagram of an exemplary embodiment of a target detection device;

[0013] Figure 4 It is a structural diagram of the feature pyramid network shown in this application;

[0014] Figure 5 yes Figure 1 FIG. 1 is a flow chart of an exemplary embodiment of step S130 in the target detection method;

[0015] Figure 6 It is a specific flow chart of an exemplary embodiment of the target detection method shown in the present application;

[0016] Figure 7 is a schematic structural diagram of another exemplary embodiment of an object detection device;

[0017] Figure 8 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application;

[0018] Fig. 9 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.

[0020] First of all, it should be noted that with the development of computer vision, the role of image acquisition equipment is not limited to video acquisition, but also needs to be combined with the video screen to respond to timely alarms. For example, the image acquisition device detects the target object in real time and sends a reminder to the detection party when the target object leaves a fixed area. The detection program of the current image acquisition device is usually trained with labeled training data. Such methods can only detect targets of known categories. When facing target object detection tasks at different stages, accurate target detection is achieved by requiring additional fine-grained annotations or adding new fine-grained classification models. In order to solve the problem that the target detection model can only detect pre-set object categories, the open vocabulary detection algorithm is an algorithm that can recognize and understand the semantic information in the text, and can directly detect different semantic categories without additional fine-grained data annotation. At present, open vocabulary target detection algorithms are usually divided into two categories: grounding and referring. The grounding-based OVD (Open-Vocabulary Object Detection) algorithm mainly relies on a large number of image-text pairs for training, which requires a large amount of annotation resources. The referring-based OVD algorithm takes detection box regression and category judgment as two subtasks. It extracts category tokens from external input text, and then converts these categories into vectors through the CLIP model. These vectors are similar to the features extracted by the image encoding model, and the largest value is the category of the current detection box.

[0021] Based on this, the present application proposes a target detection method, an electronic device and a computer-readable storage medium. Figure 1 , Figure 1 It is a flowchart of an exemplary embodiment of the target detection method shown in the present application.

[0022] The target detection method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The target detection method may also be executed by a target detection device. In some possible implementations, the target detection method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0023] Specifically, the target detection method of this embodiment includes the following steps:

[0024] S110: Acquire a captured image and detection prompt information of a target object, wherein the captured image includes an object to be detected.

[0025] The captured image may be an image that needs to be detected. Exemplarily, the captured image may be acquired by an image acquisition device, which may be a camera or an infrared camera installed in a room, road, etc. that can capture the object to be detected. In addition, an image whose image quality is greater than a preset quality threshold among multiple images captured by the image acquisition device may be used as a captured image. In other application scenarios, relevant data of each captured image may also be obtained through the network, for example, the captured image to be processed may be obtained from an online gallery, a social media platform, or a professional database.

[0026] The target object is an object to be detected, which may be, but is not limited to, a vehicle, an animal, etc. In practical applications, the captured image is generally an image captured in real time, so the target object may or may not exist in the captured image.

[0027] The detection prompt information is used to guide the target detection device to detect the target object. Exemplarily, the detection prompt information may include the category, color, etc. of the target object. The detection prompt information may be manually predetermined, and the object to be detected including the detection prompt information in the captured image may be detected through the detection prompt information, and the information of the object to be detected including the detection prompt information may be output.

[0028] The object to be detected refers to an object present in the collected image. Exemplarily, the object to be detected may include but is not limited to pets, vehicles, etc. When it is necessary to detect a target animal in a collected image, the animal present in the collected image is determined as the object to be detected, and the target animal is determined as the target object. In practical applications, the object to be detected may not exist in the collected image acquired in real time, so a preliminary detection may be performed after the collected image is acquired, and the collected image containing the object to be detected may be subjected to target detection.

[0029] The target detection device acquires the image acquired by the image acquisition device in real time, performs preliminary detection on the acquired image, determines whether there is an object to be detected, and if so, determines the acquired image as an acquired image and acquires detection prompt information of the target object to be detected. Of course, in other embodiments, the target detection device can also use the original image acquired by the image acquisition device as the acquired image for target detection.

[0030] S120: Determine fusion features of the collected image according to image features of the collected image and text features of the detection prompt information.

[0031] Image features refer to information extracted from a captured image that can characterize its content, such as describing the texture, shape, edge, and color attributes in an image. Exemplarily, the image features of the captured image can be obtained by extracting features from the captured image through an image feature extraction network model. The image feature extraction network model can be a CNN (Convolutional Neural Network), ShufflenetV2 model, ViT (Vision Transformer), Swin Transformer, and VGGNet (Visual Geometry Group Network), etc. It can also be obtained by distillation learning based on a multimodal image feature extraction network model. The multimodal image feature extraction network model can understand the association between images and texts. Exemplarily, the multimodal image feature extraction network model can be pre-trained using CLIP (Contrastive Language-Image Pretraining).

[0032] Text features refer to the attributes extracted from the detection prompt information that characterize its semantics and structure. For example, the text features of the detection prompt information can be extracted using a Transformer text encoder pre-trained by CLIP (Contrastive Language-Image Pretraining). The CLIP model is a multimodal pre-training model developed by OpenAI. It is first pre-trained through a large-scale contrastive learning task, using text and image data from the Internet. The Transformer text encoder can thus convert the detection prompt information into vector representations, which are compared and contrasted with the image vectors generated by the image feature extraction network model, thereby achieving cross-modal reasoning.

[0033] The fusion feature is obtained by fusing the image features of the collected image and the text features of the detection prompt information by the feature fusion network model. The text features are fused with the image features to enhance the interaction between the text features and the image features, thereby improving the detection accuracy of open targets.

[0034] The target detection device inputs the captured image into the image feature extraction network model for image encoding processing to obtain the image features of the captured image, inputs the detection prompt information into the text encoder pre-trained by CLIP for text encoding processing to obtain the text features of the detection prompt information, and inputs the image features of the captured image and the text features of the detection prompt information into the feature fusion network model for feature fusion processing to obtain fusion features.

[0035] S130: Perform detection processing on the collected image based on the fusion features of the collected image to obtain target object information corresponding to the detection prompt information in the collected image.

[0036] The target object information includes the category, color, and posture of the target object, which belongs to fine-grained information detection. Traditional target object detection usually only detects the coarse-grained information of the target object.

[0037] The object detection device performs object detection on the collected image by fusing the features and the text features of the detection prompt information. Combining text and images, untrained object categories can be detected, and more fine-grained object detection can be achieved based on the detection prompt information.

[0038] It can be seen that the target detection method of the embodiment of the present application obtains the detection prompt information of the collected image and the target object, the collected image includes the object to be detected; the fusion features of the collected image are determined according to the image features of the collected image and the text features of the detection prompt information; the collected image is detected and processed based on the fusion features of the collected image to obtain the target object information corresponding to the detection prompt information in the collected image. The introduction of the detection prompt information can realize the open set detection of the target, and the combination of the image features of the collected image and the text features of the detection prompt information enhances the interaction between the text and the image, thereby improving the accuracy of target detection.

[0039] Based on the above embodiments, the embodiments of the present application adopt Figure 2 The flowchart explains in detail how to obtain fusion features, please refer to Figure 2 , Figure 2 yes Figure 1 FIG. 1 is a flow chart of an exemplary embodiment of step S120 in the target detection method. Specifically, the process of determining the fusion features of the collected image according to the image features of the collected image and the text features of the detection prompt information in step S120 includes the following steps:

[0040] S210: Perform feature extraction processing at different resolutions on the collected image to obtain image features at different resolutions.

[0041] First, in order to more intuitively understand the target detection device of this embodiment, please refer to Figure 3 , Figure 3 It is a structural diagram of an exemplary embodiment of a target detection device. The target detection device includes a detection network and CLIP. The detection network includes input, which mainly includes images (captured images) and categories of objects in the images (detection prompt information), a backbone network (image feature extraction network model), a fusion network (feature fusion network model) and a detection head (regression detection head and classification detection head). CLIP includes an image encoder (multimodal image feature extraction network model) and a text encoder (multimodal text feature extraction network model). The image encoder of CLIP only exists in the training process. During the training process, the CLIP image encoder performs feature distillation on the backbone network in the detection network.

[0042] The target detection device inputs the captured image into the image feature extraction network model for feature extraction processing to obtain image features of the captured image at different resolutions. The image feature extraction network model can include five stages, each stage corresponds to feature extraction of different scales, that is, the features output by each stage have different resolutions. As an example, a captured image with an image dimension of 3*H*W is input into the image feature extraction network model for convolution processing, and image features of three dimensions are output, and the output image features correspond to 8 times, 16 times, and 32 times the downsampling resolution, respectively.

[0043] Among them, the image feature extraction network model belongs to the backbone network of the target detection device. In some embodiments, the image feature extraction network model can be obtained by distillation learning based on the multimodal image feature extraction network model. The distillation learning process includes the following steps: obtaining training samples, the training samples include sample images and corresponding sample labels; inputting the training samples into the trained multimodal image feature extraction network model and the image feature extraction network model to be trained respectively, and obtaining the first image feature output by the multimodal image feature extraction network model and the second image feature output by the image feature extraction network model to be trained; calculating the loss value of the loss function based on the first image feature and the second image feature; training the image feature extraction network model to be trained with the loss value reduced as the goal, until the loss value meets the preset requirements, and obtaining the trained image feature extraction network model. In this way, the implicit knowledge between the image text in the multimodal image feature extraction network model can be transferred to the image feature extraction network model, which reduces the computational burden while ensuring the accuracy of target detection. At the same time, the corresponding detection network can be selected according to actual needs for model building.

[0044] Specifically, the multimodal image feature extraction network model can be an image encoder pre-trained by CLIP. In some embodiments, in order to improve the effective detection of the image feature extraction network model, the first image feature and the second image feature can be aligned. The alignment process can be performed by flattening the second image feature of the sample image after the image feature extraction network model downsamples the sample image by 32 times, thereby using the self-attention module to align the dimensions of the second image feature with the first image feature.

[0045] The training process of the image feature extraction network model to be trained is also the process of updating the model parameters. In some embodiments, in order to enable the image feature extraction network model to learn the implicit knowledge of feature extraction in the multimodal image feature extraction network model structure and take into account the target detection capability, the multimodal image feature extraction network model can be used as a teacher model and the image feature extraction network model as a student model through momentum distillation, and the image feature extraction network model can be trained based on the multimodal image feature extraction network model. Exemplarily, the parameter update formula of the image feature extraction network model satisfies the following formula:

[0046]

[0047] in, represents the parameters of this round of the image feature extraction network model, Indicates the preset distillation parameters, Represents the parameters of the previous round of the image feature extraction network model, It represents the current round parameters of the image feature extraction network model when momentum distillation is not used, which can be obtained by performing feature distillation on the multimodal image feature extraction network model and back propagation of the detection network in the target detection device.

[0048] In order to make the image feature extraction network model more suitable for target detection tasks, the preset distillation parameters can be controlled at the beginning of training to reduce the impact of the multimodal image feature extraction network model on the image feature extraction network model. After the detection network has a certain detection performance, the preset distillation parameters are adjusted, and the implicit knowledge between the multimodal image feature extraction network model and the multimodal text feature extraction network model is introduced by distillation training. In this way, momentum distillation is used to avoid the failure of the detection network to converge due to excessive structural differences, and at the same time stabilize the detection ability of the detection network.

[0049] During the training process of the detection network, its total loss value is composed of the backbone network and the detection head. For example, the calculation of the total loss value of the detection network satisfies the following formula:

[0050]

[0051] in, Represents the total loss value of the detection network, represents a loss value of a loss function between a first image feature output by a multimodal image feature extraction network model and a second image feature output by the image feature extraction network model, and the loss value can be obtained by calculating the feature distribution between the first image feature and the second image feature using KL divergence; Represents the classification contrast loss value in the detection network, Represents the iou (Intersection over Union) loss of the regression loss in the detection network, DFL (Dynamic Focal Loss) represents the regression loss in the detection network. represents the distillation control parameter, Indicates a preset value. For example, Gaussian loss decay can be used, which first increases and then decreases during training, presenting a bell-shaped function state.

[0052] In other embodiments, the target detection device may also perform a single-scale feature extraction process on the collected image to obtain image features of the collected image, and perform feature fusion process on the image features of the collected image and the text features of the detection prompt information to obtain fusion features.

[0053] S220: Perform feature fusion processing on the image features of the acquired image at different resolutions and the text features of the detection prompt information to obtain fusion features of the acquired image at different resolutions.

[0054] After obtaining the image features of the collected image at different resolutions, the target detection device inputs the image features of the collected image at different resolutions and the text features of the detection prompt information into the feature fusion network model for feature fusion processing to obtain the fusion features of the collected image at different resolutions. In some embodiments, the target detection device may perform feature extraction processing on the collected image through the image feature extraction network model to obtain the image features of the collected image at different resolutions; then perform feature fusion processing on the image features of the collected image at different resolutions and the text features of the detection prompt information to obtain the fusion features of the collected image at different resolutions. In other embodiments, the target detection device may perform feature extraction processing on the collected image to obtain the image features of the collected image at different resolutions; then perform feature fusion processing on the image features of the collected image at different resolutions and the text features of the detection prompt information through the feature fusion network model to obtain the fusion features of the collected image at different resolutions. In other embodiments, the target detection device may perform feature extraction processing on the collected image through the image feature extraction network model to obtain the image features of the collected image at different resolutions; then perform feature fusion processing on the image features of the collected image at different resolutions and the text features of the detection prompt information through the feature fusion network model to obtain the fusion features of the collected image at different resolutions. When the image feature extraction network model is not used, the image features of the acquired image can be extracted by using methods such as color histogram and texture analysis; when the image fusion network model is not used, the image features of the acquired image at different resolutions and the text features of the detection prompt information can be feature spliced ​​to obtain the fusion features of the acquired image at different resolutions.

[0055] Among them, in some embodiments, the feature fusion network model can be a feature pyramid network constructed in accordance with the idea of ​​path aggregation network (PAN), and each layer of the pyramid corresponds to a different image resolution, so as to extract and utilize features at different scales. In other embodiments, different levels in the feature fusion network model can be set with different sizes of convolution kernels, and different image features are processed by convolution kernels of different sizes to generate fusion features of different resolutions. In other embodiments, since the input is the image features of the acquired image at different resolutions, the image features at different resolutions can be sequentially input into the feature fusion network model, and the feature fusion network model sequentially fuses the image features and text features at different resolutions to obtain fusion features corresponding to different resolutions. By performing target detection through fusion features of different resolutions, high-resolution image features can provide fine-grained detail information, while low-resolution image features can better capture global context and larger objects. By combining these image features of different scales, the target detection model can more accurately detect targets of various sizes.

[0056] As an example, the image features at different resolutions include a first image feature and a second image feature, and the feature fusion network model includes a first fusion module, a second fusion module, and a third fusion module. The process of obtaining fusion features of different resolutions through a feature pyramid network includes the following steps: upsampling the first image feature to obtain the first image feature after upsampling; performing a first feature fusion process on the first image feature, the second image feature, and the text feature of the detection prompt information after upsampling through the first fusion module to obtain the fusion feature after the first feature fusion process; performing a second feature fusion process on the fusion feature and the text feature after the first feature fusion process through the second fusion module to obtain the fusion feature corresponding to the second image feature; downsampling the fusion feature corresponding to the second image feature to obtain the fusion feature after downsampling; performing a third feature fusion process on the first image feature, the fusion feature after downsampling, and the text feature through the third fusion module to obtain the fusion feature corresponding to the first image feature. Thus, by fusing multi-scale image features and text features through a feature pyramid network, not only the amount of calculation parameters is reduced, but also the multi-scale feature information and text features of the acquired image can be effectively utilized, thereby generating richer and more accurate fusion features.

[0057] As another example, see Figure 4 , the image features at different resolutions may further include a third image feature. The fusion feature determination process corresponding to the third image feature may refer to the above, and the first image feature is upsampled to obtain the first image feature after the upsampling process; the first fusion module performs a first feature fusion process on the first image feature after the upsampling process, the second image feature and the text feature of the detection prompt information to obtain the fusion feature after the first feature fusion process; the fusion feature after the first feature fusion process is upsampled to obtain the fusion feature after the upsampling process; the fourth fusion module performs a second feature fusion process on the fusion feature after the upsampling process and the text feature to obtain the third image feature corresponding to the image feature. fusion features; downsampling the fusion features corresponding to the third image features to obtain the fusion features corresponding to the third image features after the downsampling; performing a third feature fusion process on the fusion features corresponding to the third image features after the downsampling, the fusion features after the first feature fusion process and the text features through the second fusion module to obtain the fusion features corresponding to the second image features; downsampling the fusion features corresponding to the second image features to obtain the fusion features corresponding to the downsampling second image features; performing a fourth feature fusion process on the fusion features corresponding to the second image features after the downsampling and the text features through the third fusion module to obtain the fusion features corresponding to the first image features.

[0058] The feature pyramid network includes two stages: top-down and bottom-up. Figure 4 The left half of the middle part is the process of upsampling from the first image feature to the third image feature, from bottom to top, including Figure 4 The middle right half shows the process of downsampling from the third image feature to the first image feature in sequence. Taking the image features at different resolutions including the first image feature and the second image feature as an example, from top to bottom, it includes upsampling the first image feature to obtain the first image feature after upsampling; performing a first feature fusion process on the first image feature, the second image feature and the text feature of the detection prompt information after the upsampling process by the first fusion module to obtain the fusion feature after the first feature fusion process. From bottom to top, it includes performing a second feature fusion process on the fusion feature after the first feature fusion process and the text feature by the second fusion module to obtain the fusion feature corresponding to the second image feature; downsampling the fusion feature corresponding to the second image feature to obtain the fusion feature after downsampling; performing a third feature fusion process on the first image feature, the fusion feature after downsampling and the text feature by the third fusion module to obtain the fusion feature corresponding to the first image feature. Taking image features at different resolutions including a first image feature, a second image feature and a third image feature as an example, the top-to-bottom process includes upsampling the first image feature to obtain the first image feature after the upsampling process; performing a first feature fusion process on the first image feature, the second image feature and the text feature of the detection prompt information after the upsampling process through a first fusion module to obtain a fused feature after the first feature fusion process; upsampling the fused feature after the first feature fusion process to obtain the fused feature after the upsampling process; performing a second feature fusion process on the fused feature after the upsampling process and the text feature through a fourth fusion module to obtain a fused feature corresponding to the third image feature. The bottom-up process includes downsampling the fused features corresponding to the third image features to obtain the fused features corresponding to the third image features after the downsampling process; performing a third feature fusion process on the fused features corresponding to the third image features after the downsampling process, the fused features after the first feature fusion process and the text features through a second fusion module to obtain the fused features corresponding to the second image features; downsampling the fused features corresponding to the second image features to obtain the fused features corresponding to the downsampling second image features; performing a fourth feature fusion process on the fused features corresponding to the second image features after the downsampling process and the text features through a third fusion module to obtain the fused features corresponding to the first image features.

[0059] In some embodiments, the top-down process may be to embed text features into image features of the captured image at different resolutions. Exemplarily, in the first feature fusion process, the max-Sigmoid method may be used to weight the image features of each layer in the feature pyramid network, and the calculation process includes: performing feature splicing processing on the first image features and the second image features after upsampling processing to obtain image features after feature splicing processing; performing attention calculation on the image features after feature splicing processing and the text features of the detection prompt information to obtain attention weights; and determining the product of the attention weights and the image features after feature splicing processing as the fused features after the first feature fusion processing. The bottom-up process may be to aggregate image features of different resolutions into text features. Specifically, the calculation of embedding text features into image features of the captured image at different resolutions may satisfy the following formula:

[0060]

[0061] in, represents the fusion features output by each fusion module, Represents the image features of each fusion module input, Represents the text features input by each fusion module, Represents the sigmoid operation. For example, in the first feature fusion process, represents the fusion features output by the first fusion module, represents the image feature after feature splicing obtained by splicing the first image feature and the second image feature after upsampling, Text features representing detection prompt information.

[0062] For example, in the third feature fusion process, the cross-attention mechanism can be used to aggregate the image features into the text features, and the calculation process includes: performing multi-head attention calculation on the first image feature, the fused feature after downsampling, and the text feature to obtain the fused feature representation; adding the fused feature representation and the text feature to obtain the fused feature corresponding to the first image feature. Specifically, the calculation of embedding the image features of the captured image at different resolutions into the text features can satisfy the following formula:

[0063]

[0064] in, represents the fusion features output by each fusion module, Represents the text features input by each fusion module, Represents the image features of each fusion module input, Represents a multi-head attention operation. For example, in the third feature fusion process, represents the fusion features output by the third fusion module, The text features representing the detection prompt information, represents the first image feature, Represents the fused features after downsampling.

[0065] Of course, in other embodiments, the product of the fused feature representation and the text feature can also be determined as the fused feature corresponding to the first image feature. Thus, the text feature and the image feature are fused through multi-head attention, so as to achieve more comprehensive feature capture, enhance the richness of feature representation and the ability of task processing, and improve the performance of the feature fusion network model.

[0066] In other embodiments, the top-down and bottom-up feature fusion processing can use the max-sigmoid method to aggregate text features into image features. Of course, the cross-attention mechanism can also be used to aggregate image features into text features. In other embodiments, the feature fusion processing method can also splice image features and text features to obtain corresponding fusion features.

[0067] Based on the above embodiments, the embodiments of the present application adopt Figure 5 The flowchart details how to detect the target object information in the acquired image. Figure 5 , Figure 5 yes Figure 1 The flowchart of an exemplary embodiment of step S130 in the target detection method is shown. Specifically, step S130 performs detection processing on the collected image based on the fusion features of the collected image, and the process of obtaining the target object information corresponding to the detection prompt information in the collected image includes the following steps:

[0068] S510: Performing initial detection on the fusion features of the collected image to obtain initial object information.

[0069] The initial object information is object information that has not been verified by the detection prompt information. In other embodiments, in order to improve the detection efficiency, the initial object information obtained by the initial detection can be determined as the target object information. Exemplarily, the fusion features can be processed by the detection head in the target detection device to obtain the initial object information of the captured image. The initial object information includes the category, color, and posture of the corresponding object in each initial area. Among them, the detection head includes a regression branch and a classification branch. The regression branch is responsible for detecting the initial area. In some embodiments, the structure of the regression branch includes two 3×3 depth-separable convolution layers, and the depth-separable convolution layer is composed of a 3×3 depth-wise convolution and a 1×1 point-by-point convolution; in other embodiments, the structure of the regression branch may include two 3*3 convolution layers. The initial area of ​​the captured image is obtained by using the box regression method of DFL in anchor-free. The box regression calculation of DFL satisfies the following formula:

[0070]

[0071] in, Represents the distance from the center point of the grid in the initial area to the predicted box, Represents the discrete distance weight between the grid center point and the prediction box, Represents the regression component of the regression branch output. Can be set to 8, 16, etc.

[0072] The classification branch is responsible for outputting the initial object information of the initial region. In other embodiments, the classification branch further includes a comparison layer for comparing the initial object information with the detection prompt information to obtain the target object information.

[0073] S520: Obtain feature similarity between initial text features of initial object information and text features of detection prompt information.

[0074] The target detection device can compare the initial object information and the detection prompt information through the comparison layer of the classification branch in the detection head. Exemplarily, the comparison method can be to obtain the feature similarity between the initial text features of the initial object information and the text features of the detection prompt information. It should be noted that a captured image may include multiple objects, that is, there may be multiple initial object information, and the detection prompt information may also include multiple detection prompt words, that is, multiple text features. The calculation of feature similarity can be but is not limited to cosine similarity, Euclidean distance, normalization, etc. As an example, L2 norm normalization can be used to normalize the detection prompt information and the initial object information to obtain feature similarity, and its calculation satisfies the following formula:

[0075]

[0076] in, represents feature similarity, represents the similarity coefficient, represents L2 norm normalization (regularization operation), represents the initial text features of the kth initial object information, represents the jth text feature, Represents the similarity bias. Thus, the training of the target detection device is effectively stabilized through L2 norm normalization.

[0077] S530: In response to the feature similarity being greater than a preset feature similarity threshold, determining the corresponding initial object information as the target object information.

[0078] The target detection device obtains the feature similarity between each initial object information and the detection prompt information in the collected image output by the detection head, and determines the initial object information whose feature similarity is greater than the preset feature similarity threshold as the target object information. In some embodiments, the target detection device can first calculate the feature similarity between the initial text feature of the same initial object information and the text feature of each detection prompt word in the detection prompt information, and determine the detection prompt word corresponding to the maximum feature similarity as the target prompt word corresponding to the initial object information; then select the initial object information whose feature similarity between each initial object information and the target prompt word is greater than the preset feature similarity threshold as the target object information. In other embodiments, the average feature similarity between the initial text feature of the same initial object information and the text feature of each detection prompt word in the detection prompt information is calculated; and the initial object information whose average feature similarity is greater than the preset feature similarity threshold is selected as the target object information. The preset feature similarity threshold can be set according to experience, for example, set to 0.7, 0.8, 0.9, etc. In other embodiments, the target detection device can calculate the feature similarity between the same initial object information and each detection prompt word in the detection prompt information, and determine the detection prompt word corresponding to the maximum feature similarity as the target prompt word corresponding to the initial object information; unify the classification scores of each initial region and the regional quality of the initial region through task alignment learning (TAL); and select the target object information from each initial object information based on the classification score and the regional quality.

[0079] In order to elaborate on the target detection method of the present application, Figure 6 The flowchart shown further illustrates it, and the details are as follows:

[0080] First, the captured image is obtained. The captured image can be used as a training sample in the training process or as an image to be detected in the application process. The captured image is preprocessed. The preprocessing methods include but are not limited to image cropping, image denoising, etc. Feature extraction is performed on the preprocessed captured image through the backbone network (image feature extraction network model). If it is in the training stage, feature distillation is performed through the CLIP image encoder (multimodal image feature extraction network model) to obtain the feature distillation loss value of the loss function between the first image feature output by the CLIP image encoder and the second image feature output by the backbone network. After that, the image features extracted by the backbone network and the text features of the detection prompt information output by the CLIP text encoder are input into the feature fusion layer for feature fusion processing to obtain fused features, and the fused features are fused through the detection head to obtain the prediction box and the predicted text description corresponding to the prediction box, and the prediction box and the captured image are calculated. The regression loss value between the actual object area of ​​the object to be detected is included, and the regression loss may include iou loss and DEL loss. The classification contrast loss value between the predicted text description and the sample label of the object to be detected is calculated, and the detection network is trained with the purpose of reducing the sum of the feature distillation loss value, the regression loss value and the classification contrast loss value to obtain a trained detection network; if it is not in the training stage, the image features extracted by the backbone network are input into the feature fusion layer, and the text features of the detection prompt information output by the CLIP text encoder are input into the feature fusion layer; the image features of the collected image and the text features of the detection prompt information are subjected to feature fusion processing through the feature fusion layer to obtain fused features; the fused features are input into the detection head, the fused features are processed through the classification branch and the regression branch in the detection head, and the detection prompt information and the initial object information are subjected to feature comparison based on the comparison layer of the classification branch to obtain the target object information.

[0081] See also Figure 7 , Figure 7 It is a structural diagram of another exemplary embodiment of the target detection device shown in the present application. The target detection device 700 includes an acquisition module 710, a feature fusion module 720 and a detection module 730. The acquisition module 710 is used to acquire the detection prompt information of the collected image and the target object, and the collected image includes the object to be detected; the feature fusion module 720 is used to determine the fusion features of the collected image according to the image features of the collected image and the text features of the detection prompt information; the detection module 730 is used to perform detection processing on the collected image based on the fusion features of the collected image, and obtain the target object information corresponding to the detection prompt information in the collected image.

[0082] In the above scheme, the target detection device obtains the collected image and the detection prompt information of the target object, the collected image includes the object to be detected; determines the fusion feature of the collected image according to the image feature of the collected image and the text feature of the detection prompt information; performs detection processing on the collected image based on the fusion feature of the collected image, and obtains the target object information corresponding to the detection prompt information in the collected image. The introduction of the detection prompt information can realize the open set detection of the target, and the combination of the image feature of the collected image and the text feature of the detection prompt information enhances the interaction between the text and the image, thereby improving the accuracy of target detection.

[0083] Among them, the functions of each module can be found in the target detection method embodiment, and will not be repeated here.

[0084] In order to implement the target detection method of the above embodiment, the present application proposes another electronic device, which is specifically referred to as Figure 8 , Figure 8 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application.

[0085] The electronic device 800 includes a memory 810 and a processor 820 , wherein the memory 810 and the processor 820 are coupled.

[0086] The memory 810 is used to store program data, and the processor 820 is used to execute the program data to implement the target detection method of the above embodiment.

[0087] In this embodiment, the processor 820 may also be referred to as a CPU (Central Processing Unit). The processor 820 may be an integrated circuit chip having signal processing capabilities. The processor 820 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 820 may also be any conventional processor, etc.

[0088] The present application also provides a computer-readable storage medium, such as Fig. 9 As shown, the computer-readable storage medium 900 is used to store program data 910. When the program data 910 is executed by the processor, it is used to implement the target detection method in the method embodiment of the present application.

[0089] The method involved in the target detection method embodiment of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0090] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A target detection method, characterized in that: The target detection method comprises: Acquire a captured image and detection prompt information of a target object, wherein the captured image includes an object to be detected; Performing feature extraction processing on the collected image through an image feature extraction network model to obtain image features of the collected image at different resolutions, wherein the image feature extraction network model is obtained based on distillation learning of a multimodal image feature extraction network model; Inputting the image features of the collected image at different resolutions and the text features of the detection prompt information into a feature fusion network model to obtain fusion features of the collected image at different resolutions output by the feature fusion network model; Performing detection processing on the collected image based on the fusion features of the collected image to obtain target object information corresponding to the detection prompt information in the collected image; The image features at different resolutions include a first image feature and a second image feature, the feature fusion network model includes a first fusion module, a second fusion module and a third fusion module, and the step of inputting the image features at different resolutions of the acquired image and the text features of the detection prompt information into the feature fusion network model to obtain the fusion features of the acquired image at different resolutions output by the feature fusion network model includes: Performing upsampling processing on the first image feature to obtain a first image feature after upsampling processing; Performing a first feature fusion process on the first image feature after the upsampling process, the second image feature, and the text feature of the detection prompt information by the first fusion module to obtain a fusion feature after the first feature fusion process; Performing a second feature fusion process on the fused features after the first feature fusion process and the text features through the second fusion module to obtain a fused feature corresponding to the second image feature; Downsampling the fused feature corresponding to the second image feature to obtain a downsampled fused feature; The third fusion module performs a third feature fusion process on the first image feature, the fused feature after the downsampling process, and the text feature to obtain a fused feature corresponding to the first image feature.

2. The target detection method according to claim 1, characterized in that: The step of performing a first feature fusion process on the first image feature after the upsampling process, the second image feature and the text feature of the detection prompt information by the first fusion module to obtain a fusion feature after the first feature fusion process comprises: Performing feature splicing processing on the first image feature and the second image feature after the upsampling processing to obtain image features after the feature splicing processing; Performing attention calculation on the image features after the feature splicing processing and the text features of the detection prompt information to obtain attention weights; The product of the attention weight and the image feature after the feature splicing process is determined as the fusion feature after the first feature fusion process.

3. The target detection method according to claim 1, characterized in that: The step of performing a third feature fusion process on the first image feature, the fused feature after the downsampling process, and the text feature through the third fusion module to obtain a fused feature corresponding to the first image feature includes: Performing multi-head attention calculation on the first image feature, the fused feature after the downsampling process, and the text feature to obtain a fused feature representation; The fused feature representation and the text feature are added together to obtain a fused feature corresponding to the first image feature.

4. The target detection method according to claim 1, characterized in that: The step of performing detection processing on the collected image based on the fusion feature of the collected image to obtain target object information corresponding to the detection prompt information in the collected image includes: Performing initial detection on the fusion features of the acquired image to obtain initial object information; Acquire feature similarity between initial text features of the initial object information and text features of the detection prompt information; In response to the feature similarity being greater than a preset feature similarity threshold, the corresponding initial object information is determined as the target object information.

5. The target detection method according to claim 1, characterized in that: The method further comprises: Acquire a training sample, wherein the training sample includes a sample image and a corresponding sample label; Inputting the training samples into the trained multimodal image feature extraction network model and the image feature extraction network model to be trained respectively, to obtain the first image feature output by the multimodal image feature extraction network model and the second image feature output by the image feature extraction network model to be trained; Calculate a loss value of a loss function based on the first image feature and the second image feature; The image feature extraction network model to be trained is trained with the goal of reducing the loss value until the loss value meets the preset requirements, thereby obtaining a trained image feature extraction network model.

6. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: include: Program data is stored, and when the program data is executed by a processor, it is used to implement the method according to any one of claims 1 to 5.

Images