Target detection method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a target detection method, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a collection image and detection prompt information of a target object, and the collection image comprises a to-be-detected object; inputting the image features of the collected image and the text features of the detection prompt information into a feature fusion network model for feature fusion processing to obtain fusion features; and performing detection processing on the acquired image based on the fusion feature and the text feature of the detection prompt information to obtain target object information matched with the detection prompt information in the acquired image. Therefore, the target detection precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a target detection method, an electronic device and a computer-readable storage medium. Background Art

[0002] With the rapid development of computer vision and deep learning technology, object detection methods have been widely used in various fields and have achieved significant breakthroughs. Traditional object detection methods detect objects in images based on pre-determined object categories. However, the real world is diverse and predefined categories usually cannot achieve full coverage, resulting in the inability to efficiently identify new objects that do not exist in the predefined categories.

[0003] Based on this, how to achieve open-world target detection has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a target detection method, an electronic device and a computer-readable storage medium, which can improve the accuracy of target detection.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: a target detection method is provided, the target detection method comprising: obtaining a captured image and detection prompt information of a target object, the captured image comprising an object to be detected; inputting image features of the captured image and text features of the detection prompt information into a feature fusion network model for feature fusion processing to obtain fusion features; performing detection processing on the captured image based on the fusion features and text features of the detection prompt information to obtain target object information in the captured image that matches the detection prompt information.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above target detection method.

[0007] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a computer-readable storage medium including program data stored therein, wherein the program data is used to implement the above target detection method when executed by a processor.

[0008] Compared with the problem that traditional target detection can only detect trained object categories and cannot detect new targets that do not exist in predefined categories, the above scheme provides a target detection method, including: obtaining detection prompt information of a captured image and a target object, the captured image including the object to be detected; inputting the image features of the captured image and the text features of the detection prompt information into a feature fusion network model for feature fusion processing to obtain fusion features; performing detection processing on the captured image based on the fusion features and the text features of the detection prompt information to obtain target object information in the captured image that matches the detection prompt information. In this way, the image features of the captured image and the text features of the detection prompt information are combined to enhance the interaction between text and image, and the introduction of detection prompt information can realize open-world target detection, thereby improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0010] Figure 1 is a flowchart of an exemplary embodiment of a target detection method shown in the present application;

[0011] Figure 2 It is a schematic diagram of the structure of the ShufflenetV2 model shown in this application;

[0012] Figure 3 yes Figure 1 FIG. 1 is a flow chart of an exemplary embodiment of step S120 in the target detection method;

[0013] Figure 4 It is a structural diagram of the feature pyramid network shown in this application;

[0014] Figure 5 It is a structural schematic diagram of a lightweight module shown in this application;

[0015] Figure 6 yes Figure 1 FIG. 1 is a flow chart of an exemplary embodiment of step S130 in the target detection method;

[0016] Figure 7 It is a structural diagram of the target detection model shown in this application;

[0017] Figure 8 A specific flow chart of an exemplary embodiment of a target detection method shown in the present application;

[0018] Fig. 9 is a schematic structural diagram of an exemplary embodiment of a target detection device shown in the present application;

[0019] Fig.10 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application;

[0020] Fig.11 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.

[0022] First of all, it should be noted that target detection refers to identifying the location and category of a specific target in an image or video. In this solution, the main purpose is to identify the attributes of pets in the image, including pet category, pet color, and pet posture. In order to solve the problem that the target detection model can only detect pre-set pet categories, most of the current large detectors with heavy backbone networks are used to process images and texts for prediction at the same time, resulting in heavy computational burdens, complex edge device deployment, and other problems, and cannot be inferred and applied in real time.

[0023] Based on this, the present application proposes a target detection method, an electronic device and a computer-readable storage medium. Figure 1 , Figure 1 It is a flowchart of an exemplary embodiment of the target detection method shown in the present application.

[0024] The execution subject of the target detection method can be a terminal device or a server or other processing device, wherein the terminal device can be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The execution subject of the target detection method can also be a target detection device. In some possible implementations, the target detection method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0025] Specifically, the target detection method of this embodiment includes the following steps:

[0026] S110: Acquire a captured image and detection prompt information of a target object, wherein the captured image includes an object to be detected.

[0027] The captured image refers to an image that needs to be detected. Exemplarily, the captured image can be acquired by an image acquisition device, and the image acquisition device can be installed in a room, road, or other location where the object to be detected can be captured. In addition, an image whose image quality is greater than a preset quality threshold among multiple images captured by the image acquisition device can be used as a captured image. In other application scenarios, relevant data of each captured image can also be obtained through the network, for example, the captured image to be processed can be obtained from an online gallery, a social media platform, or a professional database.

[0028] The detection prompt information is used to guide the target detection model to detect the target object. Exemplarily, the detection prompt information may include the category, color, etc. of the target object. It can be predetermined by humans. For example, if a pet owner has a black dog and a white cat and wants to know the status of the black dog, the detection prompt information can be determined as a black dog. The detection prompt information can be used to detect the object to be detected in the captured image including the detection prompt information, and output the information of the object to be detected including the detection prompt information.

[0029] The object to be detected refers to the object present in the collected image. Exemplarily, the objects to be detected are mainly the major categories that need to be detected, such as pets, vehicles, etc. When it is necessary to detect the target pet in the collected image, the pet present in the collected image is determined as the object to be detected, and the target pet is determined as the target object. In practical applications, the object to be detected may not exist in the collected image acquired in real time, so a preliminary detection can be performed after the collected image is acquired, and the collected image with the object to be detected can be subjected to target detection.

[0030] The target detection device acquires the image captured by the acquisition device in real time, performs preliminary detection on the acquired image to determine whether there is an object to be detected. If so, the acquired image is determined as the acquisition image and the detection prompt information of the target object to be detected is obtained.

[0031] S120: Inputting the image features of the collected image and the text features of the detection prompt information into a feature fusion network model for feature fusion processing to obtain fusion features.

[0032] Image features refer to information extracted from a captured image that can characterize its content, such as describing the texture, shape, edge, and color of an image. For example, the image feature extraction network can be used to extract features from the captured image to obtain the image features of the captured image. The image feature extraction network can be a ShufflenetV2 model. The structure of the core module ShuffleBlock of the ShufflenetV2 model is as follows: Figure 2 As shown, Figure 2 The left side shows a structure with a step size of 1. Figure 2The right side shows a structure with a stride of 2. In the structure with a stride of 1, the acquired image is firstly channel-separated to obtain the first channel and the second channel, and the second channel is subjected to multi-layer convolution processing, including sequentially performing 1×1Conv (convolution), BN ReLU (Batch NormalizationRectified Linear Unit, normalization-activation function) operations, 3×3DWConv (Depthwise separable convolution, depthwise separable convolution), BN (Batch Normalization, normalization), 1×1Conv (convolution), and BN ReLU (BatchNormalizationRectified Linear Unit, normalization-activation function) operations, and the second channel after multi-layer convolution processing is concat (connected) with the first channel, and then Channel Shuffle (channel shuffle) is performed on the first channel and the second channel after the concat processing. In the structure with a stride of 2, the collected image is firstly channel-separated to obtain the first channel and the second channel, and the first channel is subjected to multi-layer convolution processing, including 3×3 or 3×2 DWConv (Depthwiseseparable convolution), BN (Batch Normalization), 1×1Conv (convolution) and BN ReLU (Batch NormalizationRectified Linear Unit, normalization-activation function) operations; the second channel is subjected to multi-layer convolution processing, including 1×1Conv (convolution), BN ReLU (Batch NormalizationRectified Linear Unit, normalization-activation function) operations, 3×3 or 3×2 DWConv (Depthwise separable convolution), BN (BatchNormalization), 1×1Conv (convolution) and BN ReLU (BatchNormalizationRectified Linear Unit) operations in sequence. Unit, normalization-activation function) operation; Concat (connect) the first channel after the multi-layer convolution processing and the second channel after the multi-layer convolution processing, and perform Channel Shuffle (channel shuffle) on the first channel and the second channel after the Concat processing.

[0033] The ShufflenetV2 model of this embodiment optimizes and improves the original ShufflenetV2 model, deletes the last 1×1 convolution layer of the ShufflenetV2 model, and reduces the parameter processing amount. There are five remaining stages, and the network structure parameters of the five stages are shown in Table 1. Each stage corresponds to the image features of different resolutions of the collected image, for example, stage 3, stage 4 and stage 5 correspond to image features of 8 times, 16 times and 32 times resolution, respectively. In Table 1, the first stage is convolution processing, the second stage is maximum pooling processing, and the third to fifth stages are the core module ShuffleBlock. Among them, the input refers to the image resolution of the input image. In the third stage, when the step size of ShuffleBlock is 2, the number of ShuffleBlocks is 1, and when the step size of ShuffleBlock is 1, the number of ShuffleBlocks is 3; in the fourth stage, when the step size of ShuffleBlock is 2, the number of ShuffleBlocks is 1, and when the step size of ShuffleBlock is 1, the number of ShuffleBlocks is 7; the fifth stage is similar.

[0034] Table 1

[0035]

[0036] Text features refer to the attributes extracted from the detection prompt information that characterize its semantics and structure. For example, the text features of the detection prompt information can be extracted using a Transformer text encoder pre-trained by CLIP (Contrastive Language-Image Pretraining). The CLIP model is a multimodal pre-training model developed by OpenAI. It is first pre-trained through a large-scale contrastive learning task, using text and image data from the Internet. The Transformer text encoder can thus convert the detection prompt information into vector representations, which are compared and contrasted with the image vectors generated by the image encoder, thereby achieving cross-modal reasoning.

[0037] For example, the text features encoded by the CLIP pre-trained Transformer text encoder can be expressed as follows:

[0038] W = TextEncoder(T)∈R C∪D

[0039] Among them, W represents text features, T represents detection prompt information, C represents the number of nouns in the detection prompt information, and D represents feature dimension, which can be preset and is generally set to 512. The number of nouns in the detection prompt information can be obtained by analyzing the part of speech of each word in the detection prompt information, for example, cat is a name and white is an adjective.

[0040] The feature fusion network model is used to fuse the image features of the collected image and the text features of the detection prompt information. The fused features are obtained by the feature fusion network model after fusing the image features of the collected image and the text features of the detection prompt information. The text features are fused into the image features to enhance the interaction between the text features and the image features, thereby improving the detection accuracy of open objects.

[0041] The target detection device inputs the captured image into the ShufflenetV2 model for image encoding processing to obtain image features of the captured image, inputs the detection prompt information into the Transformer text encoder pre-trained by CLIP for text encoding processing to obtain text features of the detection prompt information, and inputs the image features of the captured image and the text features of the detection prompt information into the feature fusion network model for feature fusion processing to obtain fused features.

[0042] S130: Perform detection processing on the collected image based on the fusion feature and the text feature of the detection prompt information to obtain target object information in the collected image that matches the detection prompt information.

[0043] Target object information includes the category, color, and posture of the target object, which belongs to fine-grained information detection. Traditional target object detection usually only detects coarse-grained information of the target object. In pet detection, with the increase in the number of pet owners, many families have more than one pet and more than one type. It is necessary to detect the fine-grained information of pets to distinguish different pets and analyze pet behaviors.

[0044] The object detection device performs object detection on the collected image by fusing the features and the text features of the detection prompt information. Combining text and images, untrained object categories can be detected, and more fine-grained object detection can be achieved based on the detection prompt information.

[0045] It can be seen that the target detection method of the embodiment of the present application obtains the detection prompt information of the collected image and the target object, and the collected image includes the object to be detected; the image features of the collected image and the text features of the detection prompt information are input into the feature fusion network model for feature fusion processing to obtain fusion features; the collected image is detected and processed based on the fusion features and the text features of the detection prompt information to obtain the target object information in the collected image that matches the detection prompt information. Thus, the image features of the collected image and the text features of the detection prompt information are combined to enhance the interaction between text and image, and the introduction of detection prompt information can realize open-world target detection and improve the accuracy of target detection.

[0046] Based on the above embodiments, the embodiments of the present application adopt Figure 3 The flowchart details how the feature fusion network model fuses the image features of the collected image with the text features of the detection prompt information. Figure 3 , Figure 3 yes Figure 1 The flowchart of an exemplary embodiment of step S120 in the target detection method is shown. Specifically, step S120 inputs the image features of the collected image and the text features of the detection prompt information into the feature fusion network model for feature fusion processing, and the process of obtaining the fusion feature includes the following steps:

[0047] S310: Perform multi-scale feature extraction processing on the collected image to obtain image features of the collected image at different resolutions.

[0048] The target detection device inputs the collected image into the ShufflenetV2 model for feature extraction. The ShufflenetV2 model includes five stages, each of which corresponds to feature extraction of different scales, that is, the features output by each stage have different resolutions. For example, the image features output by the third, fourth, and fifth stages correspond to downsampling resolutions of 8 times, 16 times, and 32 times, respectively. The image features output by the third to fifth stages are subjected to subsequent operations.

[0049] In other embodiments, the target detection device may also perform common feature extraction processing on the collected image to obtain image features of the collected image, and perform feature fusion processing on the image features of the collected image and the text features of the detection prompt information to obtain fusion features.

[0050] S320: Perform feature fusion processing on the image features of the collected image at different resolutions and the text features of the detection prompt information through a feature fusion network model to obtain fusion features corresponding to different resolutions.

[0051] The feature fusion network model can be a feature pyramid network constructed following the idea of ​​Path Aggregation Network (PAN), where each layer of the pyramid corresponds to a different image resolution, thereby extracting and utilizing features at different scales. Figure 4 As shown in the figure, F1, F2, and F3 are the image features output by the 3rd, 4th, and 5th stages of the ShufflenetV2 model, respectively. T-CSPLayer is a lightweight module, and P1, P2, and P3 are the fusion features corresponding to F1, F2, and F3, respectively. Specifically, taking F3 and F2 as examples, F3 and F2 are determined as the first image feature and the second image feature, respectively, and the feature fusion network model includes the first lightweight module, the second lightweight module, and the third lightweight module. The determination process of fusion features of different resolutions includes the following steps: upsampling the first image feature to obtain the first image feature after upsampling; performing a first feature fusion process on the first image feature after upsampling, the second image feature and the text feature of the detection prompt information through the first lightweight module to obtain the fusion feature after the first feature fusion process; performing a second feature fusion process on the fusion feature after the first feature fusion process and the text feature through the second lightweight module to obtain the fusion feature corresponding to the second image feature; downsampling the fusion feature corresponding to the second image feature to obtain the fusion feature after downsampling; performing a third feature fusion process on the first image feature, the fusion feature after downsampling and the text feature through the third lightweight module to obtain the fusion feature corresponding to the first image feature. Therefore, by fusing the lightweight module through the feature pyramid network, not only the amount of calculation parameters is reduced, but also the multi-scale feature information and text features of the collected image can be effectively utilized to generate richer and more accurate fusion features.

[0052] Furthermore, F1 corresponds to the third image feature, and the process of determining the fusion feature corresponding to the third image feature can refer to the above, and the fusion feature after the first feature fusion processing is upsampled to obtain the fusion feature after the upsampling processing; the third image feature and the fusion feature after the upsampling processing are subjected to feature fusion processing through the second lightweight module to obtain the fusion feature corresponding to the third image feature.

[0053] In other embodiments, convolution kernels of different sizes may be set at different levels in the feature fusion network model, and different image features may be processed by convolution kernels of different sizes to generate fusion features of different resolutions. In other embodiments, since the input is the image features of the captured image at different resolutions, the image features at different resolutions may be sequentially input into the feature fusion network model, and the feature fusion network model sequentially fuses the image features and text features at different resolutions to obtain fusion features corresponding to different resolutions. By performing target detection using fusion features of different resolutions, high-resolution image features can provide fine-grained detail information, while low-resolution image features can better capture global context and larger objects. By combining these image features of different scales, the target detection model can more accurately detect targets of various sizes.

[0054] Among them, the structure of the lightweight module is as follows Figure 5 As shown, taking the first lightweight module as an example, the first lightweight module performs a first feature fusion process on the first image feature, the second image feature and the text feature of the detection prompt information after the upsampling process, and obtains the fusion feature after the first feature fusion process, including: performing channel separation process on the first image feature and the second image feature after the upsampling process by the first lightweight module to obtain the image features under each channel; performing multi-layer convolution process on the image features under the first channel to obtain the image features after the convolution process; performing multi-head attention calculation on the image features and text features after the convolution process to obtain the attention weight; performing full connection process on the image features after the convolution process, the attention weight and the fusion feature under the second channel to obtain the fusion feature after the first feature fusion process. Thus, the image features and text features are fused by the lightweight module, which greatly reduces the amount of calculation parameters.

[0055] Among them, channel separation can be performed according to the resolution of the input image features. For example, the input image features can be divided into three channels, the first channel is processed by multi-layer convolution, and the remaining channels are directly input to the connection layer as the second channel. The process of multi-layer convolution processing includes: 3×3DWConv, 1×1Conv, 3×3DWConv and 1×1Conv. The deep separable convolution can reduce the number of parameters for target detection and effectively control the increase in parameters caused by the branch decoupling of the feature fusion model.

[0056] The multi-head attention calculation of image features and text features after convolution processing can use Max-Sigmoid attention, and its calculation formula satisfies the following formula:

[0057]

[0058] Among them, X′ lrepresents the attention weight, X l represents the image features after convolution processing, sigmoid represents the activation function, and max represents the maximum value selection. represents the transposition of the jth text feature, and C represents the number of detection prompt texts in the detection prompt information, that is, the number of text features.

[0059] In some application scenarios, before the first image features and the second image features after upsampling are input into the first lightweight module, the first image features and the second image features after upsampling are spliced ​​to obtain spliced ​​image features; the spliced ​​image features are input into the first lightweight module; the first image features are subjected to 1×1Conv (convolution), and then the spliced ​​image features after convolution are subjected to channel separation processing to obtain image features under each channel; the image features under the first channel are subjected to multi-layer convolution processing to obtain graph features after convolution processing; multi-head attention calculation is performed on the image features and text features after convolution processing to obtain attention weights; the image features after convolution processing, the attention weights, and the fused features under the second channel are fully connected to obtain fully connected features; the fully connected features are subjected to 1×1Conv (convolution) to obtain fused features after the first feature fusion processing.

[0060] Based on the above embodiments, the embodiments of the present application adopt Figure 6 The flowchart details how to detect the target object information in the acquired image. Figure 6 , Figure 6 yes Figure 1 The flowchart of an exemplary embodiment of step S130 in the target detection method is shown. Specifically, step S130 performs detection processing on the captured image based on the fusion feature and the text feature of the detection prompt information, and the process of obtaining the target object information matching the detection prompt information in the captured image includes the following steps:

[0061] S610: Performing region detection processing on the captured image with fusion features through the target detection model to obtain the target object region in the captured image.

[0062] The target detection model can also be set to a lightweight structure, such as Figure 7 As shown in the figure, the target detection model includes a region detection branch and a text description generation branch. Each branch includes two 3×3 depthwise separable convolutional layers. The depthwise separable convolutional layer consists of a 3×3 depthwise convolution and a 1×1 pointwise convolution. The depthwise separable convolutional layer can reduce the number of computational parameters of the target detection model and reduce the computational burden.

[0063] In the region detection branch, the input fusion features are subjected to two depth-separable convolution layers for depth-separable convolution processing, and then the fusion features after the depth-separable convolution processing are subjected to 1×1 convolution processing, wherein the parameter reg_max for calculating the region coordinates is set to 16, and the number of region coordinates of a region is generally 4. The probability distribution of the region coordinates is obtained through the region detection branch, the target region coordinates are determined according to the probability distribution, and the target object region is determined according to the target region coordinates. For example, the region coordinates with a probability greater than a preset probability threshold can be determined as the target region coordinates.

[0064] The target object region includes the target object. Exemplarily, the region where the target object exists is defined by a border to obtain the target object region.

[0065] The target detection device inputs the fused features into the target detection model, and in the region detection branch, performs multi-layer depth-separable convolution processing on the fused features to obtain the probability distribution of the region coordinates, determines the target region coordinates according to the probability distribution, and determines the target object region according to the target region coordinates.

[0066] S620: Perform text description generation processing on the target object area with the fusion features through the target detection model to obtain the target object description of the target object area.

[0067] The target object description refers to the text description corresponding to the target object area. For example, the category, color, posture, etc. of the target object in the target object area are described to form the target object description. As an example, a black Garfield cat is lying down to eat, which is a target object description of the cat target object.

[0068] The text description generation branch of the object detection model performs two depth-wise separable convolutional layers on the input fused features to obtain the target object description of the target object area.

[0069] The target detection device performs target detection by fusing features and can learn richer target object information, thereby making the detection of target objects more accurate.

[0070] In some other embodiments, the method for acquiring target object information further includes the following steps: performing region detection processing on the captured image with fusion features through the target detection model to obtain an initial region in the captured image; performing text description generation processing on the initial region with fusion features through the target detection model to obtain an initial text description of the initial region; determining the target object region from the initial region according to the text features and the initial text description in the initial region; and determining the initial text description corresponding to the target object region as the target object description. Thus, the detected initial region and the initial text description are further verified by detecting the text features of the prompt information, so as to improve the accuracy of target detection.

[0071] You can continue to read Figure 7 In the text description generation branch, in this embodiment, after processing the fused features through the deep separable convolutional layer, the initial area of ​​the captured image and the initial text description corresponding to the initial area are obtained, and then the detection prompt information and the initial text description are compared. The initial area and the initial text description corresponding to the initial area are verified according to the text description of the detection prompt information to obtain the target object area and the corresponding target object description.

[0072] In practical applications, the detection prompt information may include multiple detection prompt texts. For example, a family has four pets, a white cat, a black dog, a black cat and a white rabbit. If you want to get the information of the white cat and the black dog at the same time, you can input the two detection prompt texts of the white cat and the black dog, and perform text feature extraction on the two detection prompt texts of the white cat and the black dog respectively to obtain the text features of the white cat and the black dog, and encode the text features of the white cat as 1 and the text features of the black dog as 2; then, the initial text description of each initial area detected in the collected image is respectively calculated with the text features of the white cat and the black dog, and the text feature closest to the initial text description is determined, and the encoding is used as the classification label of the initial area; through task alignment learning (Task Alignment Learning (TAL), unify the classification scores of each initial region and the regional quality of the initial region, assign positive and negative samples, where the positive sample is the initial region with a classification score greater than or equal to a preset score and a regional quality greater than or equal to a preset quality, and the negative sample is the initial region with a classification score less than a preset score and a regional quality less than a preset quality; determine the positive sample as the target object region and the corresponding initial text description as the target object description.

[0073] Among them, the similarity calculation between the text features and the initial text features during the verification process satisfies the following formula:

[0074] s k,j =α*L2norm(e k )*L2norm(wj ) T +β

[0075] Among them, s k,j represents similarity, α represents a learnable scaling factor for affine transformation, β represents a learnable offset factor for radial transformation, L2norm represents L2 norm normalization, e k represents the kth initial text feature, w j Represents the jth text feature. Thus, it is normalized by L2 norm and affine transformed to effectively stabilize the training of regional text description.

[0076] Furthermore, the target detection model used above is a trained target detection model. The training process of the target detection model is as follows: first, a sample image set is obtained, and the sample image set includes a sample image and a text description corresponding to the sample image; the fusion features between the image features of the sample image and the text features of the corresponding text description are input into the target detection model to obtain at least one candidate detection area output by the target detection model and a predicted text description corresponding to the candidate detection area; the first loss value of the loss function between the candidate detection area and the actual target area of ​​the sample image is calculated; the second loss value of the loss function between the predicted text description corresponding to the candidate detection area and the text description corresponding to the sample image is calculated; the target detection model is trained with the goal of reducing the sum of the first loss value and the second loss value, until the sum of the first loss value and the second loss value meets the preset requirements. Thus, through the combined training of area and text, a target detection model with more generalization ability is obtained.

[0077] Wherein, the sample image includes at least one sample object area, and the text description corresponding to the sample image includes the area text description corresponding to the sample object area. When obtaining the sample image set, the sample image set can be obtained by sampling from the first preset sample image set and the second preset sample image set respectively. The first preset sample image set may include objects to be detected and objects not to be detected, such as various objects including pets, plants, and vehicles. Exemplarily, the first preset sample image set can be sampled from a general object detection dataset, such as the COCO dataset and the Objects365 dataset. The bounding box-label pair Ω in the general object detection dataset is converted into a vector. i ,c i} is redefined as region-text pair Ω = {B i ,t i} constitutes a first preset sample image set, r i It is area B i The corresponding label, t i It is area B iThe corresponding regional text description is a fine-grained description of the label, which may include descriptions such as category, name, and posture. The second preset sample image set may only include the object to be detected. Exemplarily, a fine-grained pet dataset may be pre-constructed to obtain the second preset sample image set. Among them, t i ′ is area B i ′The corresponding regional text description includes pet category (such as Garfield, American Shorthair, Rural Cat, Bichon Frise, Teddy Dog, etc.), pet color, pet posture (such as standing, lying, etc.). According to the number of samples set in the training, samples are collected from the first preset sample image set and the second preset sample image set according to a certain sampling ratio to obtain a sample image set. In this way, a fine-grained pet dataset is added for training the open vocabulary target detection model, which meets the fine-grained detection needs of various pets and expands the applicability of the model in actual scenarios.

[0078] During the training process, multi-scale feature extraction is performed on sample images in the sample image set to obtain image features at different resolutions; the text feature extraction of the regional text description corresponding to each region in the sample image is performed using the Transformer text encoder pre-trained by CLIP to obtain text features, and the text features of the text description of each region in the sample image are encoded to obtain the text index of each text feature; the image features and text features at different resolutions are combined by combining the feature pyramid network and the lightweight module to obtain fusion features at different resolutions; the fusion features at different resolutions are input into the target detection model to obtain the candidate detection region output by the target detection model and the corresponding predicted text description; the similarity between the predicted text feature and the text feature of the predicted text description is obtained, the text feature most similar to the predicted text feature is determined, and the text index of the most similar text feature is determined as the classification label of the corresponding predicted text feature; based on task alignment learning (TAL), the classification scores and regional qualities of each candidate detection region are unified, and positive and negative samples are allocated; the first loss value of the loss function between the candidate detection region and the actual target region of the sample image is calculated, wherein the loss function can be combined with the CIoU (Complete-IoU) loss function and the DFL (Distribution The method adopts a novel object detection method to detect the target image in the training set, and adopts a Focal Loss loss function; calculates the second loss value of the loss function between the predicted text description corresponding to the candidate detection area and the text description corresponding to the sample image; uses the stochastic gradient descent method to update the weight parameters of each network node until the sum of the first loss value and the second loss value converges, thereby obtaining a trained target detection model and an initial offline text vocabulary consisting of the text descriptions corresponding to the sample images.

[0079] In other embodiments, the calculation of the second loss value between the predicted text description and the text description also includes the following steps: respectively calculating the feature similarity between the predicted text features of the predicted text description corresponding to the candidate detection area and the regional text features of each regional text description; selecting a target detection area from the candidate detection area and selecting a target text description from each predicted text description based on the feature similarity; obtaining the selection rate of each predicted text description selected as the target text description; and determining the second loss value based on the cross entropy between the feature similarity and the selection rate.

[0080] Specifically, the feature similarity between the predicted text description and the regional text description is calculated, and the regional text description that is most similar to each predicted text description is determined through the feature similarity, and the text index of the most similar text feature is determined as the classification label of the corresponding predicted text feature; based on task alignment learning (TAL), the classification score and regional quality of each candidate detection region are unified, and each predicted text description is divided into a positive sample and a negative sample; the second loss value of each regional text description is obtained according to the feature similarity and the cross entropy between the positive and negative samples.

[0081] The total loss value of the target detection model is the sum of the first loss value and the second loss value, and its calculation formula satisfies the following formula:

[0082] Loss total =Loss con +Loss iou +Loss dfl

[0083] Among them, Loss total Represents the total loss value, Loss con Represents the second loss value, Loss iou Represents the first loss value calculated using the CIoU (Complete-IoU) loss function, Loss dfl Represents the first loss value calculated using the DFL (Distribution Focal Loss) loss function.

[0084] In order to elaborate on the target detection method of the present application, Figure 8 The flowchart shown further illustrates it, and the details are as follows:

[0085] During the training process, a general object detection dataset is obtained and a fine-grained household pet dataset (fine-grained pet dataset) is constructed. The dataset annotations are redefined as region-text pairs. Specifically, the region-label pairs in the general object detection dataset are redefined as region-text pairs, and the text is an extended description of the label. The general object detection dataset and the fine-grained household pet dataset are sampled according to a preset sampling ratio to obtain a sample image set; an open vocabulary object detector is constructed. Specifically, the open vocabulary object detector includes an image feature extraction network, a text encoder, a feature fusion network model, and an object detection model; online vocabulary training is carried out based on the open vocabulary object detector, the visual-language large model CLIP, and the sample image set to obtain a trained open vocabulary object detector. Specifically, the sample image is input into the image feature extraction network for multi-scale feature extraction to obtain an open vocabulary object detector. Image features at the same resolution; inputting the text description corresponding to the sample object into the text encoder pre-trained by the visual-language model CLIP for text encoding to obtain the corresponding text features; inputting the image features and text features at different resolutions into the feature fusion network model for feature fusion to obtain fused features at different resolutions; inputting the fused features into the target detection model to obtain at least one candidate detection area output by the target detection model and the predicted text description corresponding to the candidate detection area; calculating the first loss value of the loss function between the candidate detection area and the actual target area of ​​the sample image; calculating the second loss value of the loss function between the predicted text description corresponding to the candidate detection area and the text description corresponding to the sample image; training the target detection model with the sum of the first loss value and the second loss value becoming smaller as the goal, until the sum of the first loss value and the second loss value meets the preset requirements.

[0086] During the application process, it should be noted that the target detection model used in the application stage is the target detection model trained as above. First, a custom offline vocabulary can be pre-set, which includes detection prompt information, and the detection prompt information is re-parameterized into the weights of the open vocabulary target detector and deployed on the mobile platform, that is, the detection prompt information is encoded through a text encoder to obtain text features; based on the re-parameterized weights, open vocabulary (detection prompt information) target detection is performed on the mobile platform, that is, target detection is performed on the collected image through the detection prompt information to obtain the target object corresponding to the detection prompt information, specifically, the text encoding of the detection prompt information and the image features of the collected image are subjected to feature fusion processing to obtain fusion features; the fusion features are input into the target detection model for detection processing to obtain the target object information corresponding to the detection prompt information. Therefore, a "prompt first, then detect" strategy is adopted in the application stage. The text encoder encodes and stores the detection prompt information to build an offline vocabulary. The text features of the offline vocabulary can be re-parameterized into weights of the convolutional layer or linear layer for deployment. Therefore, when using the re-parameterized open vocabulary object detector, there is no need to load the text encoder. Only the image feature extraction network, feature fusion network model and object detection model are required, which simplifies the object detection process, improves the detection efficiency, and makes the open vocabulary object detector more real-time and easier to deploy.

[0087] See also Fig. 9 , Fig. 9 It is a structural diagram of an exemplary embodiment of the target detection device shown in the present application. The target detection device 900 includes an acquisition module 910, a feature fusion module 920 and a detection module 930. The acquisition module 910 is used to acquire the detection prompt information of the collected image and the target object, and the collected image includes the object to be detected; the feature fusion module 920 is used to input the image features of the collected image and the text features of the detection prompt information into the feature fusion network model for feature fusion processing to obtain fusion features; the detection module 930 is used to perform detection processing on the collected image based on the fusion features and the text features of the detection prompt information to obtain the target object information in the collected image that matches the detection prompt information.

[0088] In the above scheme, the target detection device obtains the collected image and the detection prompt information of the target object, and the collected image includes the object to be detected; the image features of the collected image and the text features of the detection prompt information are input into the feature fusion network model for feature fusion processing to obtain the fusion features; the collected image is detected based on the fusion features and the text features of the detection prompt information to obtain the target object information in the collected image that matches the detection prompt information. In this way, the image features of the collected image and the text features of the detection prompt information are combined to enhance the interaction between the text and the image, and the introduction of the detection prompt information can realize the open world target detection and improve the target detection accuracy.

[0089] Among them, the functions of each module can be found in the target detection method embodiment, and will not be repeated here.

[0090] In order to implement the target detection method of the above embodiment, the present application proposes another electronic device, which is specifically referred to as Fig.10 , Fig.10 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application.

[0091] The electronic device 1000 includes a memory 1001 and a processor 1002 , wherein the memory 1001 and the processor 1002 are coupled.

[0092] The memory 1001 is used to store program data, and the processor 1002 is used to execute the program data to implement the target detection method of the above embodiment.

[0093] In this embodiment, the processor 1002 may also be referred to as a CPU (Central Processing Unit). The processor 1002 may be an integrated circuit chip having signal processing capabilities. The processor 1002 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 1002 may also be any conventional processor, etc.

[0094] The present application also provides a computer-readable storage medium, such as Fig.11 As shown, the computer-readable storage medium 1100 is used to store program data 1101. When the program data 1101 is executed by the processor, it is used to implement the target detection method in the method embodiment of the present application.

[0095] The method involved in the target detection method embodiment of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the prior art that contributes to the technical solution or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0096] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A target detection method, characterized in that: The target detection method comprises: Acquire a captured image and detection prompt information of a target object, wherein the captured image includes an object to be detected; Inputting the image features of the collected image and the text features of the detection prompt information into a feature fusion network model for feature fusion processing to obtain fusion features; The collected image is detected and processed based on the fusion feature and the text feature of the detection prompt information to obtain target object information in the collected image that matches the detection prompt information.

2. The target detection method according to claim 1, characterized in that: The step of inputting the image features of the collected image and the text features of the detection prompt information into a feature fusion network model for feature fusion processing to obtain fusion features comprises: Performing multi-scale feature extraction processing on the collected image to obtain image features of the collected image at different resolutions; The feature fusion network model is used to perform feature fusion processing on the image features of the acquired image at different resolutions and the text features of the detection prompt information to obtain fusion features corresponding to different resolutions.

3. The target detection method according to claim 2, characterized in that: The image features include a first image feature and a second image feature, the feature fusion network model includes a first lightweight module, a second lightweight module and a third lightweight module, and the step of performing feature fusion processing on the image features of the acquired image at different resolutions and the text features of the detection prompt information by the feature fusion network model to obtain fusion features corresponding to different resolutions includes: Performing upsampling processing on the first image feature to obtain a first image feature after upsampling processing; Performing a first feature fusion process on the first image feature after the upsampling process, the second image feature, and the text feature of the detection prompt information by the first lightweight module to obtain a fusion feature after the first feature fusion process; Performing a second feature fusion process on the fused features after the first feature fusion process and the text features through the second lightweight module to obtain a fused feature corresponding to the second image feature; Downsampling the fused feature corresponding to the second image feature to obtain a downsampled fused feature; The third lightweight module performs a third feature fusion process on the first image feature, the fused feature after the downsampling process, and the text feature to obtain a fused feature corresponding to the first image feature.

4. The target detection method according to claim 3, characterized in that: The step of performing a first feature fusion process on the first image feature after the upsampling process, the second image feature, and the text feature of the detection prompt information by the first lightweight module to obtain a fused feature after the first feature fusion process comprises: Performing channel separation processing on the first image features and the second image features after the upsampling processing by the first lightweight module to obtain image features under each channel; Perform multi-layer convolution processing on the image features under the first channel to obtain the image features after convolution processing; Performing multi-head attention calculation on the image features after the convolution processing and the text features to obtain attention weights; The image features after the convolution processing, the attention weights, and the fused features under the second channel are fully connected to obtain the fused features after the first feature fusion processing.

5. The target detection method according to claim 1, characterized in that: The target object information includes a target object area and a target object description, and the step of performing detection processing on the acquired image based on the fusion feature and the text feature of the detection prompt information to obtain the target object information in the acquired image that matches the detection prompt information includes: Performing region detection processing on the acquired image using the fusion feature through a target detection model to obtain a target object region in the acquired image; The target object detection model uses the fusion features to perform text description generation processing on the target object area to obtain a target object description of the target object area.

6. The target detection method according to claim 1, characterized in that: The target object information includes a target object area and a target object description, and the step of performing detection processing on the acquired image based on the fusion feature and the text feature of the detection prompt information to obtain the target object information in the acquired image that matches the detection prompt information further includes: Performing region detection processing on the acquired image using the fusion feature through a target detection model to obtain an initial region in the acquired image; Performing text description generation processing on the initial region using the fusion feature through the target detection model to obtain an initial text description of the initial region; Determine the target object region from the initial region according to the text features and the initial text description in the initial region; An initial text description corresponding to the target object area is determined as the target object description.

7. The target detection method according to any one of claims 5-6, characterized in that: The method also includes a step of training the target detection model, the training step including: Acquire a sample image set, wherein the sample image set includes sample images and text descriptions corresponding to the sample images; Inputting fusion features between the image features of the sample image and the text features of the corresponding text description into an object detection model to obtain at least one candidate detection area output by the object detection model and a predicted text description corresponding to the candidate detection area; Calculating a first loss value of a loss function between the candidate detection area and the actual target area of ​​the sample image; Calculating a second loss value of a loss function between a predicted text description corresponding to the candidate detection area and a text description corresponding to the sample image; The target detection model is trained with the goal of reducing the sum of the first loss value and the second loss value until the sum of the first loss value and the second loss value meets a preset requirement.

8. The target detection method according to claim 7, characterized in that: The sample image includes at least one sample object area, the text description corresponding to the sample image includes a regional text description corresponding to the sample object area, and the step of calculating a second loss value of a loss function between a predicted text description corresponding to the candidate detection area and a text description corresponding to the sample image includes: Calculate the feature similarity between the predicted text features of the predicted text description corresponding to the candidate detection area and the regional text features of each regional text description respectively; Selecting a target detection region from the candidate detection regions and a target text description from each predicted text description according to the feature similarity; Obtaining a selection rate of each predicted text description being selected as a target text description; A second loss value is determined according to the cross entropy between the feature similarity and the selection rate.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: include: Program data is stored, and when the program data is executed by a processor, it is used to implement the method according to any one of claims 1 to 8.

Images