Target detection method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a target detection method, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a collection image and detection prompt information of a target object, the collection image comprises a to-be-detected object, and the detection prompt information comprises image prompt information and text prompt information; performing feature fusion processing on the image features of the image prompt information and the text features of the text prompt information to obtain fused prompt features; and performing detection processing on the acquired image based on the fusion prompt feature to obtain target object information matched with the detection prompt information in the acquired image. Therefore, the accuracy of target detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a target detection method, an electronic device and a computer-readable storage medium. Background Art

[0002] With the rapid development of computer vision and deep learning technology, object detection methods have been widely used in various fields and have achieved significant breakthroughs. Traditional object detection methods detect objects in images based on pre-determined object categories. However, the real world is diverse and predefined categories usually cannot achieve full coverage, resulting in the inability to efficiently identify new objects that do not exist in the predefined categories.

[0003] Based on this, how to achieve open-world target detection has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a target detection method, an electronic device and a computer-readable storage medium, which can improve the accuracy of target detection.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: a target detection method is provided, the target detection method comprising: obtaining detection prompt information of a captured image and a target object, the captured image comprising an object to be detected, the detection prompt information comprising image prompt information and text prompt information; performing feature fusion processing on image features of the image prompt information and text features of the text prompt information to obtain fused prompt features; performing detection processing on the captured image based on the fused prompt features to obtain target object information in the captured image that matches the detection prompt information.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above target detection method.

[0007] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a computer-readable storage medium including program data stored therein, wherein the program data is used to implement the above target detection method when executed by a processor.

[0008] Compared with the problem that traditional target detection can only detect trained object categories and cannot detect new targets that do not exist in predefined categories, the above scheme provides a target detection method, including: obtaining detection prompt information of a captured image and a target object, the captured image includes the object to be detected, and the detection prompt information includes image prompt information and text prompt information; performing feature fusion processing on the image features of the image prompt information and the text features of the text prompt information to obtain fused prompt features; performing detection processing on the captured image based on the fused prompt features to obtain target object information in the captured image that matches the detection prompt information. In this way, multi-modal target detection is performed through image prompt information and text prompt information, which combines the rich image features of the image prompt information and the high generalization ability of the text features, thereby improving the accuracy of target detection and realizing open set detection of target objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0010] Figure 1 is a flowchart of an exemplary embodiment of a target detection method shown in the present application;

[0011] Figure 2 is a structural diagram of an exemplary embodiment of a text encoder shown in the present application;

[0012] Figure 3 yes Figure 1 FIG. 1 is a flow chart of an exemplary embodiment of step S130 in the target detection method;

[0013] Figure 4 is a schematic structural diagram of an exemplary embodiment of a first target detection model shown in the present application;

[0014] Figure 5 It is a specific flow chart of an exemplary embodiment of the target detection method shown in the present application;

[0015] Figure 6 is a schematic diagram of an application scenario of an exemplary embodiment of the target detection method shown in the present application;

[0016] Figure 7 is a structural schematic diagram of an exemplary embodiment of a target detection device shown in the present application;

[0017] Figure 8It is a structural schematic diagram of an embodiment of an electronic device provided by the present application;

[0018] Fig. 9 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.

[0020] First of all, it should be noted that target detection is to detect the collected images through computer technology to obtain the target object information in the collected images. In practical applications, this target detection method can be applied to non-motor vehicle detection, motor vehicle detection, animal detection, etc. Taking non-motor vehicles as an example, the current non-motor vehicle detection based on closed set detection category intervals requires high-cost manual annotations, can only detect known non-motor vehicle categories, and cannot adapt to open-world target detection. However, the existing public detection datasets only have a limited number of targets, and it is difficult to collect training data for all categories. In addition, due to the long-tail problem between categories, it is still very difficult to achieve general open-world target detection. Among them, the public detection datasets include COCO80 category, Object365 365 category, and LVIS1203 category. At the same time, the current open vocabulary target detection lacks fine-grained target text descriptions, cannot accurately provide key text information of similar categories, and is prone to false detection of similar categories.

[0021] Based on this, the present application proposes a target detection method, an electronic device and a computer-readable storage medium. Figure 1 , Figure 1 It is a flowchart of an exemplary embodiment of the target detection method shown in the present application.

[0022] The execution subject of the target detection method can be a terminal device or a server or other processing device, wherein the terminal device can be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The execution subject of the target detection method can also be a target detection device. In some possible implementations, the target detection method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0023] Specifically, the target detection method of this embodiment includes the following steps:

[0024] S110: Acquire a captured image and detection prompt information of a target object, wherein the captured image includes an object to be detected, and the detection prompt information includes image prompt information and text prompt information.

[0025] The captured image may be an image that needs to be detected. Exemplarily, the captured image may be acquired by an image acquisition device, which may be a camera or an infrared camera installed in a room, road, etc. that can capture the object to be detected. In addition, an image whose image quality is greater than a preset quality threshold among multiple images captured by the image acquisition device may be used as a captured image. In other application scenarios, relevant data of each captured image may also be obtained through the network, for example, the captured image to be processed may be obtained from an online gallery, a social media platform, or a professional database.

[0026] The target object is an object to be detected, which may be, but is not limited to, a non-motor vehicle, a motor vehicle, an animal, etc. In practical applications, the captured image is generally a real-time captured image, so the target object may or may not exist in the captured image.

[0027] The detection prompt information is used to guide the target detection model to detect the target object. Exemplarily, the detection prompt information includes image prompt information and text prompt information. It can be predetermined by humans, and the object to be detected including the detection prompt information in the collected image can be detected through the detection prompt information, and the information of the object to be detected including the detection prompt information can be output. For example, in order to prevent residents from bringing electric vehicles into the elevator and causing dangerous accidents, it is necessary to detect whether there is an electric vehicle in the elevator. Then, the electric vehicle image and the electric vehicle text can be used as image prompt information and text prompt information respectively to obtain the detection prompt information, so as to guide the target detection model to detect whether there is an electric vehicle in the elevator before the elevator door closes. If it appears, the detected electric vehicle information is output and an alarm is issued.

[0028] Among them, image hint information includes the shape, color, posture, etc. of the target object. Text hint information includes the shape, category, color, posture, etc. of the target object. Image hint information can provide more information about the target object, solving the problem that limited text is difficult to accurately describe fine-grained target object information; text hint information has better generalization ability and can improve the target detection ability in the open world.

[0029] The object to be detected refers to an object present in the collected image. The object to be detected may be, but is not limited to, a non-motor vehicle, a motor vehicle, an animal, etc. Exemplarily, an initial detection is performed on the collected image to obtain the object to be detected included in the collected image. In practical applications, the object to be detected may not exist in the collected image acquired in real time, so the collected image may also be detected after the collected image is acquired, and target detection is performed on the collected image where the object to be detected exists.

[0030] The target detection device acquires the image captured by the acquisition device in real time, detects the acquired image, and determines whether there is an object to be detected. If so, the acquired image is determined as the acquisition image and the image prompt information and text prompt information of the target object are obtained.

[0031] In other embodiments, the detection prompt information may also include only image prompt information, or only text prompt information.

[0032] S120: Perform feature fusion processing on the image features of the image prompt information and the text features of the text prompt information to obtain fused prompt features.

[0033] Image features refer to information extracted from image prompt information that can characterize its content, such as describing the texture, shape, edge, color and other attributes in the image. Exemplarily, the image features of the collected image can be extracted by an image encoder to obtain the image features of the collected image. After the image features of the image prompt information are obtained, a label position is added before the image features and marked as image features. The image encoder can be VIT (Vision Transformer), SwinTransformer and VGGNet, etc. Exemplarily, a non-motor vehicle image template can be built by a picture-text alignment algorithm, and the category of non-motor vehicles in the non-motor vehicle image template corresponds to the offline vocabulary corresponding to the text prompt information, and the number of templates is set to more than 65536, so that the image encoder can obtain the corresponding image prompt information from the non-motor vehicle image template through the text prompt information. Thus, through the pre-training of large-scale image-text pairs, multimodal representation is learned, and the picture-text alignment function is possessed. Among them, the picture-text learning algorithm can be an algorithm such as ALBEF (Align before Fuse).

[0034] Text features refer to the attributes extracted from text prompt information that represent its semantics and structure. Exemplarily, the text prompt information can be subjected to feature extraction through a text encoder to obtain text features. A label is added before the text feature to mark it as a text feature. As an example, a label [0] can be added before the text feature and a label [1] can be added before the image feature to distinguish between text features and image features. The text encoder can adopt the CoOP scheme (Conditional Prompt Learning for Vision-Language Models) and use a learnable prompt template to obtain better text features. Please refer to Figure 2 , CoOP introduces M learnable context vectors {[v] 1 ,[v] 2 ,...,[v] M}, the learnable context vector is prompted according to each input instance (image), rather than being fixed once learned, the learnable context vector and text prompt information are input into the text encoder (Test Encoder) for encoding, and the text prompt information can be the class of the target object (class), such as person, vehicle, pet, etc. The text encoder can thus convert the text prompt information into vector representations, which are compared and contrasted with the image vectors generated by the image encoder to achieve cross-modal reasoning. The text encoder can also be a bag of words model, a BERT model, etc.

[0035] The target detection device extracts features of image prompt information through an image feature extraction network to obtain image features; extracts features of text prompt information through a text encoder to obtain text features; and performs feature fusion processing on image features and text features to obtain fused prompt features. The feature fusion processing can adopt a multimodal attention mechanism to adjust the weights of image features and text features to obtain fused prompt features.

[0036] S130: Perform detection processing on the collected image based on the fused prompt feature to obtain target object information in the collected image that matches the detection prompt information.

[0037] The target object information includes the category, color, and posture of the target object, including the fine-grained information of the target object. Traditional target object detection usually only detects the coarse-grained information of the target object. When facing the target detection tasks of non-motor vehicles, such as electric vehicles, bicycles, mountain bikes, tricycles, etc., additional fine-grained information is required to achieve accurate detection or classification.

[0038] The target detection device performs target detection on the collected image based on the fusion prompt features, combines text and images to obtain richer target object information, and realizes fine-grained detection of open-set targets.

[0039] It can be seen that the target detection method of the embodiment of the present application obtains the detection prompt information of the collected image and the target object, the collected image includes the object to be detected, and the detection prompt information includes image prompt information and text prompt information; the image features of the image prompt information and the text features of the text prompt information are subjected to feature fusion processing to obtain fused prompt features; the collected image is subjected to detection processing based on the fused prompt features to obtain the target object information in the collected image that matches the detection prompt information. Thus, multi-modal target detection is performed through image prompt information and text prompt information, which combines the rich image features of the image prompt information and the high generalization ability of the text features, thereby improving the accuracy of target detection and realizing open set detection of target objects.

[0040] Based on the above embodiments, the embodiments of the present application adopt Figure 3 The flowchart details how to perform object detection on the acquired image based on the fusion hint features. Figure 3 , Figure 3 yes Figure 1 The flowchart of an exemplary embodiment of step S130 in the target detection method is shown. Specifically, step S130 performs detection processing on the collected image based on the fusion prompt feature, and the process of obtaining the target object information matching the detection prompt information in the collected image includes the following steps:

[0041] S310: Performing initial detection on the captured image using a first target detection model to obtain regional features of a region to be detected in the captured image, where the region to be detected includes an object to be detected.

[0042] The first target detection model is used to detect the object to be detected in the collected image, and output the classification information and position regression information of the collected image. The classification information and the position regression information have a corresponding relationship. The position regression information is used to determine the areas to be detected in the collected image, and the classification information is used to determine whether there are objects to be detected in each area to be detected. The position regression information includes 4-dimensional position regression parameters, which are divided into XYWH, where XY is the coordinate of the center point of the area to be detected, and W and H are the length and width of the area to be detected, respectively. The classification information includes 2 categories, which are divided into objects to be detected and objects without objects to be detected. It can be seen that the output dimension of the first target detection model is C=N+4, where N is the number of categories 2.

[0043] In some embodiments, the first target detection model may use a YOLOv5 structure. Figure 4, the step of performing initial detection on the collected image through the first target detection model to obtain the regional features of the area to be detected in the collected image may include: performing multi-scale feature extraction processing on the collected image to obtain image features of the collected image at different resolutions; performing feature fusion processing on image features of different resolutions to obtain fusion features corresponding to different resolutions; performing initial detection on fusion features of different resolutions to obtain regional features of the area to be detected in the collected image at different resolutions. In other embodiments, the target detection device may also perform single-scale feature extraction processing on the collected image to obtain image features of the collected image; perform initial detection on image features of the collected image to obtain regional features of the area to be detected in the collected image.

[0044] Exemplarily, the backbone network in the first target detection model can be used to extract multi-scale features of the input acquisition image to obtain image features at different resolutions. The backbone network includes five stages of feature extraction, and the five stages are stage 1 to stage 5, stage 1 includes 1 MobileOne-Block, stage 2 includes 2 MobileOne-Blocks, stage 3 includes 8 MobileOne-Blocks, stage 4 includes 4 MobileOne-Blocks, and stage 5 includes 2 MobileOne-Blocks; the resolution of the image features output in each stage is different. For example, when the size of the input acquisition image is 640*640*3, 640*640 represents the resolution of the acquisition image, and 3 represents the number of channels of the acquisition image. Stages 3, 4, and 5 can correspond to image features with 8 times, 16 times, and 32 times downsampled resolutions, respectively, thereby obtaining image features at different resolutions.

[0045] Among them, MobileOne-Block is a lightweight module that integrates the idea of ​​depthwise separable convolution and structural reparameterization, including two-stage operation: the first stage includes a 1×1 channel-by-channel convolution (DWConv) branch, a 3×3DWConv branch controlled by a hyperparameter k, and a batch normalization (BN) layer branch. In MobileOneLite, the hyperparameter k=4, and the activation function used is the ReLU function with low computational delay; the second stage includes a 1×1 ordinary convolution branch and a batch normalization (BN) layer branch, and the activation function used is also the ReLU function with low computational delay. Therefore, the standard convolution is decomposed into depthwise convolution and point-by-point convolution through depthwise separable convolution, which reduces the amount of calculation and parameters, improves the computational efficiency, and reduces the complexity of the first target detection model.

[0046] After obtaining the image features of the collected images at different resolutions, the image features at different resolutions are fused through the feature pyramid network, so as to extract and utilize image features at different scales. Figure 4, feature fusion processing is performed on the image features output by the third stage, the fourth stage and the fifth stage, and MobileOne-Block is a lightweight module. Specifically, taking the image features output by the fourth stage and the fifth stage as an example, the image features output by the fifth stage and the fourth stage are respectively determined as the first image feature and the second image feature, and the first target detection model includes a first lightweight module, a second lightweight module and a third lightweight module. The process of determining fusion features of different resolutions includes the following steps: performing convolution processing on the first image feature through the first lightweight module to obtain the first image feature after convolution processing (P30). In some application scenarios, the first image feature can also be convoluted before being input into the first lightweight module; performing upsampling processing on the first image feature after convolution processing to obtain the first image feature after upsampling processing; performing feature fusion processing on the first image feature and the second image feature after upsampling processing through the second lightweight module to obtain the fusion feature after feature fusion processing (P20). Specifically, the first image feature and the second image feature after upsampling processing can be concatenated (Concat) to obtain the fusion feature after feature fusion processing. The processed features are input into the second lightweight module for feature fusion processing to obtain the fused features after feature fusion processing; the fused features after feature fusion processing are convoluted by the third lightweight module to obtain the fused features (P2) corresponding to the second image features; the first image features (P30) after convolution processing and the fused features (P'20) corresponding to the second image features are feature spliced ​​to obtain the fused features (P3) corresponding to the first image features. In order to further optimize the computational efficiency and feature extraction capabilities, the features after feature splicing processing can also be convoluted by RepGhostBottleneck (convolutional network module) to obtain the fused features (P3) corresponding to the first image features. Therefore, by fusing the lightweight modules through the feature pyramid network, not only the amount of computational parameters is reduced, but also the multi-scale feature information of the collected image can be effectively utilized, thereby generating richer and more accurate fused features.

[0047] In some other embodiments, the image features of different resolutions also include a third image feature output by stage 3. The fusion feature determination process corresponding to the third image feature can refer to the above, firstly performing convolution processing on the first image feature, including a convolution layer (Conv) and a first lightweight module (MobileOne-Block), to obtain the first image feature after convolution processing (P30); performing upsampling processing on the first image feature after convolution processing to obtain the first image feature after upsampling processing; performing splicing processing on the first image feature after upsampling processing and the second image feature, and inputting the spliced ​​image feature into the second lightweight module for feature fusion processing to obtain the fusion feature after feature fusion processing (P20); performing feature fusion processing on the fusion feature after feature fusion processing (P20 ) is upsampled (UpSample) to obtain a fused feature after the upsampling process; the fused feature after the upsampling process and the third image feature are spliced, and the spliced ​​image feature (P10) is input into the fourth lightweight module for feature fusion processing to obtain a fused feature (P1) corresponding to the third image feature; the fused feature (P'10) corresponding to the third image feature and the fused feature (P20) after the feature fusion process are spliced, and the spliced ​​image feature is input into the third lightweight module for feature fusion processing to obtain a fused feature (P2) corresponding to the second image feature; the fused feature (P'20) corresponding to the second image feature and the first image feature (P30) after the convolution process are spliced, and the spliced ​​image feature is convoluted through RepGhost Bottleneck to obtain a fused feature (P3) corresponding to the first image feature.

[0048] In other embodiments, convolution kernels of different sizes may be set at different levels in the first target detection model, and different image features may be processed by convolution kernels of different sizes to generate fusion features of different resolutions. In other embodiments, since the input acquired images are image features at different resolutions, the image features at different resolutions may be sequentially input into the first target detection model, and the first target detection model sequentially fuses the image features and text features at different resolutions to obtain fusion features corresponding to different resolutions. By performing target detection using fusion features of different resolutions, high-resolution image features may provide fine-grained detail information, while low-resolution image features may better capture global context and larger objects. By combining these image features of different scales, the target detection model may more accurately detect targets of various sizes.

[0049] After obtaining the fusion features at different resolutions, the detection head of the first target detection model performs 1×1 convolution processing on the fusion features at each resolution. Exemplarily, the fusion features (P3) corresponding to the first image feature, the fusion features (P2) corresponding to the second image feature, and the fusion features (P1) corresponding to the third image feature are respectively processed into 20*20*16 in size, the fusion features (P2) corresponding to the second image feature are processed into 40*40*16 in size, and the fusion features (P1) corresponding to the third image feature are processed into 80*80*16 in size; the fusion features (P3) corresponding to the first image feature, the fusion features (P2) corresponding to the second image feature, and the fusion features (P1) corresponding to the third image feature after the convolution processing are respectively classified and regressed to obtain the regional features corresponding to the first image feature, the regional features corresponding to the second image feature, and the regional features corresponding to the third image feature.

[0050] It should be noted that the first target detection model used above is a trained first target detection model. Specifically, the training process of the first target detection model includes: obtaining a training sample set, the training sample set including a sample image and an actual area of ​​the sample image including an actual object; inputting the sample image into the first target detection model to obtain the area to be detected output by the first target detection model; calculating the loss value of the loss function between the area to be detected and the actual area; training the first target detection model with the goal of reducing the loss value until the loss value meets the preset requirements, thereby obtaining a first target detection model that meets the requirements.

[0051] The area to be detected output by the first target detection model includes position regression information and classification information. Therefore, the loss function may include classification loss and regression loss. The classification loss is used to calculate the error between the category of the area to be detected and the category of the actual area, and the regression loss is used to calculate the error between the position of the area to be detected and the position of the actual area. Exemplarily, the classification loss uses QFL LOSS, and the regression loss uses DFL LOSS. The calculation of the classification loss QFL LOSS satisfies the following formula:

[0052] QFL(σ)=-|y-σ| β ((1-y)log(1-σ)+ylog(σ))

[0053] Among them, σ represents the classification prediction value of the first target detection model, y represents the label value; β is the preset parameter value (usually 1 or 2).

[0054] The calculation of regression loss DFL LOSS satisfies the following formula:

[0055] DFL(S i , S i+1)=-((y i+1 -y)log(S i )+(yy i-1 )log(S i+1 )

[0056] Among them, y represents the label value, y i-1 and i+1 Represents the left label value and right label value of y respectively, S i+1 Represents y i-1 The probability value, S i Represents y i+1 The probability value of y i-1 and i+1 is the adjacent value of y. For example, when y is 2, y i-1 Can be 1.5, y i+1 It can be 2.5.

[0057] S320: Determine the target area and target object information in the target area from the area to be detected in the captured image by fusing the prompt feature and the area feature through a second target detection model.

[0058] The second target detection model is used to obtain the target area corresponding to the detection prompt information in the captured image, and the target area includes the target object. Exemplarily, the second target detection model can be CLIP (Contrastive Language–Image Pre-training), which is a multimodal learning model developed by OpenAI. CLIP uses a contrastive learning method to embed images and texts into the same semantic space, so that tasks such as image classification, image retrieval, and text generation can be performed. As an example, if it is necessary to detect electric vehicles in an elevator, the target detection device inputs the captured image into the first target detection model for initial detection, and determines the area to be detected including non-motor vehicles in the captured image; the area to be detected and the fused prompt features are input into the second target detection model, and the fused prompt features are used to perform specific prompts for the electric vehicle, and the target area including the electric vehicle is determined from the area to be detected, and the fine-grained information of the electric vehicle is output.

[0059] In some embodiments, the process of determining the target area from the area to be detected based on the fusion prompt feature includes the following steps: obtaining the feature similarity between the fusion prompt feature and the regional feature; in response to the feature similarity being greater than or equal to the preset feature similarity threshold, the object to be detected in the corresponding area to be detected is determined as the target object, and the target object information of the target object is obtained. Thus, by comparing the regional features of each area to be detected in the acquired image with the fusion prompt feature, the target object closest to the detection prompt information can be found, thereby improving the accuracy of target detection. Among them, the feature similarity can be calculated using cosine similarity, Euclidean distance, etc. The preset feature similarity threshold is an empirical value and can be set to 0.7, 0.8, 0.9, etc.

[0060] In some other embodiments, the process of determining the target area from the area to be detected based on the fused prompt feature may also include the following steps: fusing the fused prompt feature with the area feature to obtain the target fused feature; inputting the target fused feature into the second target detection model to perform target detection to obtain target object information. In this case, the second target detection model may be a classification model. In some other embodiments, target detection may also be performed on the target fused feature in the second target detection model to obtain an initial object; and the initial object may be verified based on the fused prompt feature to obtain the target object.

[0061] It should be noted that the second target detection model used above is a trained second target detection model. Specifically, the training process of the second target detection model includes: obtaining a training sample set, the training sample set including a sample image and text prompt information and image prompt information corresponding to each object to be detected in the sample image; performing feature fusion processing on the text features of the text prompt information and the image features of the image prompt information to obtain the fused prompt features of each object to be detected; inputting the fused prompt features and the image features of the sample image into the second target detection model to obtain the predicted object area output by the second target detection model and the predicted text description corresponding to the predicted object area; calculating the first loss value of the loss function between the predicted object area and the actual object area of ​​the sample image, and calculating the second loss value of the loss function between the predicted text description and the text description corresponding to the object to be detected; training the second target detection model with the goal of reducing the sum of the first loss value and the second loss value until the sum of the first loss value and the second loss value meets the preset requirements, thereby obtaining a second target detection model that meets the requirements.

[0062] In some embodiments, the sample images can be obtained by sampling from the first preset sample image set and the second preset sample image set respectively. The first preset sample image set may include various objects such as pets, plants, motor vehicles and non-motor vehicles. Exemplarily, the first preset sample image set can be sampled from a general target detection dataset, such as a COCO dataset, an Objects365 dataset and a LVIS1203 class. When detecting non-motor vehicles, the second preset sample image set may only include non-motor vehicle categories. Exemplarily, a fine-grained non-motor vehicle dataset can be pre-constructed to obtain a second preset sample image set. The non-motor vehicle dataset includes region-text pairs, and the region text description includes non-motor vehicle categories (such as electric vehicles, bicycles, mountain bikes, tricycles, etc.), non-motor vehicle colors, and non-motor vehicle postures (such as falling down, etc.). According to the number of samples set for training, the sample images are obtained by sampling from the first preset sample image set and the second preset sample image set according to a certain sampling ratio. Thus, a fine-grained non-motor vehicle dataset is added to train the target detection model with an open vocabulary, meet various non-motor vehicle fine-grained detection requirements, and expand the applicability of the model in actual scenarios. In other embodiments, non-motor vehicle images collected in a certain geographical area during a historical time period can also be used as sample images.

[0063] After obtaining the training sample set, the training sample set is split into a training set and a validation set, wherein the actual objects in each sample image of the validation set are different from the actual objects in each sample image of the training set, that is, there are actual objects of new categories in the validation set. As an example, there are electric vehicles in the validation set, but not in the training set. This can verify the ability of the second object detection model to detect new objects.

[0064] During the feature fusion process, the feature fusion network model can be used to perform feature fusion processing on the text features and image features of the object to be detected to obtain fusion prompt features. Among them, the feature fusion network model can be a multimodal attention mechanism. In some embodiments, in order to better solve the problem of lazy learning caused by the feature fusion network model, the text features of each object to be detected in the sample image can be randomly occluded to obtain the text features after occlusion processing; the text features after occlusion processing and the corresponding image features are subjected to feature fusion processing to obtain the fusion prompt features of each object to be detected. In this way, the feature fusion network model can pay more attention to the image features of the image prompt information and balance the prompt capabilities of the image features and text features. It should be noted that the occlusion processing of the text features of each object to be detected is usually to occlude part of the text features, such as covering the first half or the second half of the text features. Exemplarily, the feature fusion network model can be a MOCO momentum model. After the new parameters are trained, the weighted moving smoothing method is used to update the template image weights. Of course, in other embodiments, the image features of each object to be detected in the sample image can be randomly occluded to obtain the image features after occlusion processing; the image features after occlusion processing and the corresponding text features are fused to obtain the fused prompt features of each object to be detected. This makes the model pay more attention to the text features. In other embodiments, the text features and image features can be directly spliced ​​to obtain the fused prompt features.

[0065] It can be seen that the target detection method of the embodiment of the present application performs initial detection on the collected image through the first target detection model to obtain the regional features of the area to be detected in the collected image, and the area to be detected includes the object to be detected; the second target detection model is used to fuse the prompt features and the regional features to determine the target area and the target object information in the target area from the area to be detected in the collected image. Therefore, the first target detection model is used to perform initial detection on the collected image to determine the object to be detected included in the collected image; and the second target detection model is used to compare and screen the detection object with the detection prompt information to obtain the final target object information, which can improve the accuracy of target detection.

[0066] In other embodiments, the target detection device may perform target detection on the captured image through the first target detection model, and use the object to be detected output by the first target detection model as the target object. Similarly, in other embodiments, the target detection device may also input the fused prompt feature and the captured image into the second target detection model for target detection, and determine the output of the second target detection model as the target object. In this way, the speed of target detection can be improved while reducing the accuracy.

[0067] In order to elaborate on the target detection method of the present application, Figure 5 The flowchart shown further illustrates it, and the details are as follows:

[0068] During the training process, an open world detection data set (training sample set) is processed, including image data (sample images) and label data corresponding to the image data, wherein the test set contains target types that do not exist in the training set; the first target detection model is trained, image encoding is performed, and image features are extracted. Specifically, the image data in the data set is input into the first target detection model for detection, and the backbone network first performs multi-scale feature extraction processing on the image data to obtain image features of the image data at different resolutions; then, the feature pyramid network is used to perform feature fusion processing on the image features of different resolutions to obtain fusion features corresponding to different resolutions; the detection head performs initial detection on the fusion features of different resolutions to obtain regional features of the image data to be detected at different resolutions; the loss value of the loss function between the area to be detected and the actual area is calculated; the first target detection model is trained with the goal of reducing the loss value until the loss value meets the preset requirements, and a first target detection model that meets the requirements is obtained;

[0069] Then, CLIP is used to encode text and extract text features, that is, feature extraction processing is performed on text prompt information through CLIP to obtain text features of text prompt information; multimodal image-text feature alignment is performed on text prompt information and image prompt information of image data, that is, text features of text prompt information and image features of image prompt information are fused through a multimodal attention mechanism to obtain fused prompt features; the fused prompt features and regional features of image data are input into the second target detection model to generate a predicted object region and a corresponding predicted text description; the first loss value of the loss function between the predicted object region and the actual object region of the sample image is calculated, and the second loss value of the loss function between the predicted text description and the text description corresponding to the object to be detected is calculated; the second target detection model is trained with the goal of reducing the sum of the first loss value and the second loss value, until the sum of the first loss value and the second loss value meets the preset requirements, and a second target detection model that meets the requirements is obtained. When training the second target detection model, the loss function can use the cross entropy classification loss.

[0070] In the application process, it should be noted that the target detection model used in the application stage is the target detection model trained as described above. First, a custom offline vocabulary, custom categories or prompts can be pre-set. The custom offline vocabulary includes detection prompt information, and the text encoder encodes and stores it to construct an offline vocabulary. The text features can be re-parameterized as weights of a convolutional layer or a linear layer for deployment. That is, in actual applications, the process of deploying open vocabulary (detection prompt information) target detection on a mobile platform based on the re-parameterized weights includes: inputting the collected image into the first target detection model for detection, and obtaining the regional features of the area to be detected in the collected image at different resolutions; extracting features of the image prompt information to obtain image features of the image prompt information, and fusing the image features of the image prompt information and the deployed text features through a multimodal attention mechanism to obtain fused prompt features; inputting the fused prompt features and the regional features of the area to be detected in the collected image at different resolutions into the second target detection model to obtain target object information. Therefore, a "prompt first, then detect" strategy is adopted in the application stage. The text encoder encodes and stores the detection prompt information to build an offline vocabulary. The text features of the offline vocabulary can be re-parameterized into weights of the convolutional layer or linear layer for deployment. Therefore, when using the re-parameterized open vocabulary target detector, there is no need to load the text encoder. Only the first target detection model, the feature fusion network model and the second target detection model are required. This simplifies the target detection process, improves the detection efficiency, and makes the open vocabulary target detector more real-time and easier to deploy.

[0071] As an example, see Figure 6When detecting non-motor vehicles, the text encoder adopts the CoOP scheme (Conditional Prompt Learning for Vision-Language Models), inputs the learnable context vector (learnable context) and text prompt information into the text encoder (Test Encoder) to obtain text features. The text prompt information is mainly category prompts (Class), such as bicycles (bicycle), electric vehicles (moto) or tricycles (Tricycle); the image prompt information (n, 224, 224, 3) is input into the image encoder (momentum encoder) to obtain image features, momentum The encoder includes a main network and a momentum network. The momentum network is a sliding average of the weights of the main network. In the training phase, a non-motor vehicle image template is constructed, and the category of the non-motor vehicle corresponds to the category prompt in the offline vocabulary. The number of templates is more than 65,536, and the image size is 224×224×3. In the application phase, the corresponding image prompt information can be directly obtained from the non-motor vehicle image template through the category prompt in the text prompt information. The multimodal attention mechanism is used to perform feature fusion processing on the text features and image features to obtain the fusion prompt features. The collected image is input into the first target detection model (Detector) for initial detection to obtain the regional features of the area to be detected in the collected image at different resolutions. The fusion prompt features and the regional features of the area to be detected at different resolutions are input into the second target detection model (CLIP) for target detection to obtain the target object information. The loss function of the second target detection model during the training process can be BCE Loss.

[0072] See also Figure 7 , Figure 7 It is a structural diagram of an exemplary embodiment of the target detection device shown in the present application. The target detection device 700 includes an acquisition module 710, a feature fusion module 720 and a detection module 730. The acquisition module 710 is used to acquire the detection prompt information of the collected image and the target object. The collected image includes the object to be detected, and the detection prompt information includes image prompt information and text prompt information; the feature fusion module 720 is used to perform feature fusion processing on the image features of the image prompt information and the text features of the text prompt information to obtain fusion prompt features; the detection module 730 is used to perform detection processing on the collected image based on the fusion prompt features to obtain the target object information in the collected image that matches the detection prompt information.

[0073] In the above scheme, the target detection device obtains the detection prompt information of the collected image and the target object, the collected image includes the object to be detected, and the detection prompt information includes image prompt information and text prompt information; the image features of the image prompt information and the text features of the text prompt information are subjected to feature fusion processing to obtain fusion prompt features; the collected image is subjected to detection processing based on the fusion prompt features to obtain the target object information in the collected image that matches the detection prompt information. Thus, multi-modal target detection is performed through image prompt information and text prompt information, which combines the rich image features of the image prompt information and the high generalization ability of the text features, thereby improving the accuracy of target detection and realizing open set detection of target objects.

[0074] Among them, the functions of each module can be found in the target detection method embodiment, and will not be repeated here.

[0075] In order to implement the target detection method of the above embodiment, the present application proposes another electronic device, which is specifically referred to as Figure 8 , Figure 8 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application.

[0076] The electronic device 800 includes a memory 810 and a processor 820 , wherein the memory 810 and the processor 820 are coupled.

[0077] The memory 810 is used to store program data, and the processor 820 is used to execute the program data to implement the target detection method of the above embodiment.

[0078] In this embodiment, the processor 820 may also be referred to as a CPU (Central Processing Unit). The processor 820 may be an integrated circuit chip having signal processing capabilities. The processor 820 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 820 may also be any conventional processor, etc.

[0079] The present application also provides a computer-readable storage medium, such as Fig. 9 As shown, the computer-readable storage medium 900 is used to store program data 910. When the program data 910 is executed by the processor, it is used to implement the target detection method in the method embodiment of the present application.

[0080] The method involved in the target detection method embodiment of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the prior art that contributes to the technical solution or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0081] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A target detection method, characterized in that: The target detection method comprises: Acquire a captured image and detection prompt information of a target object, wherein the captured image includes the object to be detected, and the detection prompt information includes image prompt information and text prompt information; Performing feature fusion processing on the image features of the image prompt information and the text features of the text prompt information to obtain fused prompt features; The collected image is detected and processed based on the fusion prompt feature to obtain target object information in the collected image that matches the detection prompt information.

2. The target detection method according to claim 1, characterized in that: The step of performing detection processing on the collected image based on the fusion prompt feature to obtain target object information in the collected image that matches the detection prompt information includes: Performing an initial detection on the acquired image by using a first target detection model to obtain regional features of a region to be detected in the acquired image, wherein the region to be detected includes an object to be detected; The target area and the target object information in the target area are determined from the area to be detected in the acquired image by using the fusion prompt feature and the area feature through the second target detection model.

3. The target detection method according to claim 2, characterized in that: The step of performing initial detection on the acquired image by using the first target detection model to obtain regional features of the area to be detected in the acquired image includes: Performing multi-scale feature extraction processing on the collected image to obtain image features of the collected image at different resolutions; Perform feature fusion processing on image features of different resolutions to obtain fusion features corresponding to different resolutions; Initial detection is performed on the fused features of different resolutions to obtain the regional features of the to-be-detected area of ​​the acquired image at different resolutions.

4. The target detection method according to claim 3, characterized in that: The image features of different resolutions include a first image feature and a second image feature, the first target detection model includes a first lightweight module, a second lightweight module and a third lightweight module, and the step of performing feature fusion processing on the image features of different resolutions to obtain fusion features corresponding to different resolutions includes: Performing convolution processing on the first image feature through the first lightweight module to obtain a first image feature after the convolution processing; Performing upsampling processing on the first image feature after the convolution processing to obtain the first image feature after the upsampling processing; Performing feature fusion processing on the first image features after the upsampling processing and the second image features through the second lightweight module to obtain fused features after the feature fusion processing; Performing convolution processing on the fused features after the feature fusion processing by the third lightweight module to obtain fused features corresponding to the second image features; Feature splicing processing is performed on the first image feature after the convolution processing and the fusion feature corresponding to the second image feature to obtain the fusion feature corresponding to the first image feature.

5. The target detection method according to claim 2, characterized in that: The step of determining the target area and the target object information in the target area from the area to be detected in the acquired image by using the second target detection model with the fusion prompt feature and the area feature comprises: Acquire feature similarity between the fusion prompt feature and the regional feature; In response to the feature similarity being greater than or equal to a preset feature similarity threshold, the object to be detected in the corresponding area to be detected is determined as the target object, and target object information of the target object is acquired.

6. The target detection method according to claim 2, characterized in that: The method further comprises: Acquire a training sample set, wherein the training sample set includes a sample image and an actual area of ​​the sample image including an actual object; Inputting the sample image into a first target detection model to obtain an area to be detected output by the first target detection model; Calculating the loss value of the loss function between the area to be detected and the actual area; The first target detection model is trained with the goal of reducing the loss value until the loss value meets the preset requirements, thereby obtaining a first target detection model that meets the requirements.

7. The target detection method according to claim 2, characterized in that: The method further comprises: Acquire a training sample set, wherein the training sample set includes a sample image and text prompt information and image prompt information corresponding to each object to be detected in the sample image; Performing feature fusion processing on the text features of the text prompt information and the image features of the image prompt information to obtain fused prompt features of each object to be detected; Inputting the fused prompt feature and the image feature of the sample image into the second object detection model to obtain a predicted object region output by the second object detection model and a predicted text description corresponding to the predicted object region; Calculating a first loss value of a loss function between the predicted object region and the actual object region of the sample image, and calculating a second loss value of a loss function between the predicted text description and a text description corresponding to the object to be detected; The second target detection model is trained with the goal of reducing the sum of the first loss value and the second loss value until the sum of the first loss value and the second loss value meets preset requirements, thereby obtaining a second target detection model that meets the requirements.

8. The target detection method according to claim 7, characterized in that: The step of performing feature fusion processing on the text features of the text prompt information and the image features of the image prompt information to obtain fused prompt features of each object to be detected includes: Randomly performing occlusion processing on the text features of each object to be detected in the sample image to obtain the text features after occlusion processing; The text features after the occlusion processing and the corresponding image features are subjected to feature fusion processing to obtain fusion prompt features of each object to be detected.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: include: Program data is stored, and when the program data is executed by a processor, it is used to implement the method according to any one of claims 1 to 8.

Images