Target detection method, device, storage medium and equipment

Abstract

The present application discloses a target detection method, device, storage medium and equipment, the method comprising: first obtaining an environmental image of a scene in which an edge device is located as a target image to be detected; then inputting the target image into a pre-constructed target detection model, predicting the classification results and position information of each target object in the target image and the classification results at the pixel level; then fusing the classification results and position information of each target object and the classification results at the pixel level to obtain the detection results of the target object in the target image. Since the present application first constructs a target detection model based on the CLIP model and feature encoding network, and uses sample images for training, the model can learn the image representation ability of the CLIP model, so that when the model is used in the edge device to detect the target object in the target image, the accuracy and stability of the detection results can be effectively improved, thereby improving the target detection effect of the edge device.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a target detection method, device, storage medium and equipment. Background Art

[0002] Object detection refers to finding the specific location, size and other positioning information of all objects of interest in an image. It has been widely used in vehicle identification and analysis, smart home, video surveillance and analysis, face recognition and other fields.

[0003] At present, the existing target detection methods generally include three types: one is a two-stage target detection algorithm, which requires first automatically extracting the area to be detected that may contain the target object, and then further completing a series of detection or classification tasks on the area to be detected, which inevitably leads to higher latency and larger model parameters, which also makes it difficult to deploy this solution on edge devices with limited computing resources. The second is an end-to-end visual transformer detection method based on image block matching. Due to the particularity of its structure, this method also has a strong dependence on computing resources and is not suitable for edge devices with limited computing resources. The third method is a single-stage target detection algorithm. Although this method can make up for the shortcomings of the above two solutions in terms of model structure and reasoning logic, its post-processing of the prediction results depends on artificial pre-designed rules, which inevitably affects the detection effect of the model. It can be seen that the above-mentioned target detection methods cannot achieve ideal results, especially for edge devices such as sweeping robots, learning machines or home robots with limited computing resources. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a target detection method, apparatus, storage medium and equipment, which can improve the accuracy and stability of target detection by edge devices and achieve ideal detection effects.

[0005] The present application provides a target detection method, including:

[0006] Acquire a target image to be detected; the target image is an environmental image of the scene where the edge device is located;

[0007] The target image is input into a pre-built target detection model to predict the classification results and position information of each target object in the target image and the classification results at the pixel level; wherein the target detection model includes a backbone network layer, a neck network layer, a semantic branch network layer, a classification network layer, and a regression network layer; the target detection model is based on the CLIP model and feature encoding network pre-trained by contrasting language images, and is obtained by optimizing and training using pre-collected sample images; the target detection model learns the image representation ability of the CLIP model;

[0008] The classification results and position information of each target object and the classification results at the pixel level are fused to obtain the detection results of the target objects in the target image.

[0009] In a possible implementation, inputting the target image into a pre-built target detection model to predict the classification result and position information of each target object in the target image and the classification result at the pixel level includes:

[0010] Inputting the target image into the backbone network layer of a pre-built target detection model for feature extraction to obtain multi-scale features corresponding to the target image;

[0011] Inputting the multi-scale features into the neck network layer of the target detection model for feature fusion processing to obtain fusion features corresponding to the target image;

[0012] Inputting the fused features into the semantic branch network layer of the target detection model to obtain a pixel-level classification result corresponding to the target image;

[0013] The fused features are respectively input into the classification network layer and the regression network layer of the target detection model to obtain the classification results and position information of each target object in the target image.

[0014] In a possible implementation, the target detection model is constructed as follows:

[0015] Acquire a sample image; and determine a text description of a sample object to be detected in the sample image;

[0016] Inputting the text description of the sample object to be detected in the sample image into the CLIP model to obtain the text features corresponding to the sample image;

[0017] Input the sample image into the initial target detection model to obtain the fusion features corresponding to the sample image, the pixel-level classification results, and the classification results and position information of each sample object;

[0018] Inputting the fusion features corresponding to the sample image into a feature encoding network to obtain image features corresponding to the sample image;

[0019] The text features, the image features corresponding to the sample images, the pixel-level classification results, and the classification results and position information of each sample object are used, combined with the first target loss function, the second target loss function, and the third target loss function, to optimize the initial target detection model and obtain a trained target detection model.

[0020] In a possible implementation manner, after acquiring the sample image, the method further includes:

[0021] Perform data enhancement processing on the sample image to obtain an enhanced sample image.

[0022] In a possible implementation manner, determining a text description of the sample object to be detected in the sample image includes:

[0023] Determine the label category to which the sample object to be detected in the sample image belongs, and construct a text description of the sample object according to the label category.

[0024] In a possible implementation, the initial target detection model includes an initial backbone network layer, an initial neck network layer, an initial semantic branch network layer, an initial classification network layer, and an initial regression network layer; the sample image is input into the initial target detection model to obtain the fusion features corresponding to the sample image, the pixel-level classification results, and the classification results and position information of each sample object, including:

[0025] Inputting the sample image into the initial backbone network layer of the initial target detection model for feature extraction to obtain multi-scale features corresponding to the sample image;

[0026] Inputting the multi-scale features corresponding to the sample image into the initial neck network layer of the initial target detection model for feature fusion processing to obtain fusion features corresponding to the sample image;

[0027] Inputting the fusion features corresponding to the sample image into the initial semantic branch network layer of the initial object detection model to obtain the pixel-level classification result corresponding to the sample image;

[0028] The fusion features corresponding to the sample image are respectively input into the initial classification network layer and the initial regression network layer of the initial target detection model to obtain the classification results and position information of each sample object in the sample image.

[0029] In a possible implementation, the text features, the image features corresponding to the sample images, the pixel-level classification results, and the classification results and position information of each sample object are combined with the first target loss function, the second target loss function, and the third target loss function to optimize the initial target detection model and obtain the trained target detection model, including:

[0030] When optimizing and training the initial target detection model, the pixel-level classification results corresponding to the sample image are used to calculate the value of the first target loss function; and the classification results and position information of each sample object corresponding to the sample image are used to calculate the value of the second target loss function; and the text features and the image features corresponding to the sample image are used to calculate the value of the third target loss function, until the values ​​of each target loss function meet the preset conditions, then the updating of the model parameters is stopped, and the target detection model is obtained by training;

[0031] Among them, the values ​​of the first objective loss function, the second objective loss function and the third objective loss function are respectively used to constrain the updating of model parameters so that the target detection model can learn the image representation ability of the CLIP model as much as possible.

[0032] The present application also provides a target detection device, including:

[0033] A first acquisition unit is used to acquire a target image to be detected; the target image is an environment image of a scene where the edge device is located;

[0034] A prediction unit is used to input the target image into a pre-built target detection model, and predict the classification results and position information of each target object in the target image and the classification results at the pixel level; wherein the target detection model includes a backbone network layer, a neck network layer, a semantic branch network layer, a classification network layer, and a regression network layer; the target detection model is based on the CLIP model and feature encoding network pre-trained by contrasting language images, and is obtained by optimizing and training with pre-collected sample images; the target detection model learns the image representation ability of the CLIP model;

[0035] The fusion unit is used to fuse the classification results and position information of each target object and the classification results at the pixel level to obtain the target detection result corresponding to the target image.

[0036] In a possible implementation, the prediction unit includes:

[0037] A first input subunit is used to input the target image into a backbone network layer of a pre-built target detection model for feature extraction to obtain multi-scale features corresponding to the target image;

[0038] A second input subunit is used to input the multi-scale features into the neck network layer of the target detection model for feature fusion processing to obtain fusion features corresponding to the target image;

[0039] A third input subunit is used to input the fusion feature into the semantic branch network layer of the target detection model to obtain a pixel-level classification result corresponding to the target image;

[0040] The fourth input subunit is used to input the fusion features into the classification network layer and the regression network layer of the target detection model respectively to obtain the classification results and position information of each target object in the target image.

[0041] In a possible implementation manner, the device further includes:

[0042] A second acquisition unit is used to acquire a sample image; and determine a text description of a sample object to be detected in the sample image;

[0043] A first input unit, used for inputting a text description of a sample object to be detected in the sample image into a CLIP model to obtain a text feature corresponding to the sample image;

[0044] A second input unit is used to input the sample image into the initial target detection model to obtain the fusion features corresponding to the sample image, the pixel-level classification results, and the classification results and position information of each sample object;

[0045] A third input unit is used to input the fusion feature corresponding to the sample image into a feature encoding network to obtain the image feature corresponding to the sample image;

[0046] A training unit is used to utilize the text features, the image features corresponding to the sample images, the pixel-level classification results, and the classification results and position information of each sample object, combined with the first target loss function, the second target loss function, and the third target loss function, to optimize the training of the initial target detection model to obtain a trained target detection model.

[0047] In a possible implementation manner, the device further includes:

[0048] The data enhancement unit is used to perform data enhancement processing on the sample image to obtain an enhanced sample image.

[0049] In a possible implementation manner, the second obtaining unit is specifically configured to:

[0050] Determine the label category to which the sample object to be detected in the sample image belongs, and construct a text description of the sample object according to the label category.

[0051] In a possible implementation, the initial target detection model includes an initial backbone network layer, an initial neck network layer, an initial semantic branch network layer, an initial classification network layer, and an initial regression network layer; the second input unit is specifically used to:

[0052] A fourth input subunit, used for inputting the sample image into the initial backbone network layer of the initial target detection model for feature extraction to obtain multi-scale features corresponding to the sample image;

[0053] A fifth input subunit, configured to input the multi-scale features corresponding to the sample image into the initial neck network layer of the initial target detection model for feature fusion processing to obtain fusion features corresponding to the sample image;

[0054] A sixth input subunit, used to input the fusion feature corresponding to the sample image into the initial semantic branch network layer of the initial object detection model to obtain a pixel-level classification result corresponding to the sample image;

[0055] The seventh input subunit is used to input the fusion features corresponding to the sample image into the initial classification network layer and the initial regression network layer of the initial target detection model respectively, so as to obtain the classification results and position information of each sample object in the sample image.

[0056] In a possible implementation, the training unit is specifically used for:

[0057] When optimizing and training the initial target detection model, the pixel-level classification results corresponding to the sample image are used to calculate the value of the first target loss function; and the classification results and position information of each sample object corresponding to the sample image are used to calculate the value of the second target loss function; and the text features and the image features corresponding to the sample image are used to calculate the value of the third target loss function, until the values ​​of each target loss function meet the preset conditions, then the updating of the model parameters is stopped, and the target detection model is obtained by training;

[0058] Among them, the values ​​of the first objective loss function, the second objective loss function and the third objective loss function are respectively used to constrain the updating of model parameters so that the target detection model can learn the image representation ability of the CLIP model as much as possible.

[0059] The embodiment of the present application also provides a target detection device, including: a processor, a memory, and a system bus;

[0060] The processor and the memory are connected via the system bus;

[0061] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementations of the above-mentioned target detection method.

[0062] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementations of the above-mentioned target detection method.

[0063] An embodiment of the present application further provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes any one of the implementations of the above-mentioned target detection method.

[0064] The embodiments of the present application provide a target detection method, apparatus, storage medium and device. First, a target image to be detected is obtained; wherein the target image is an environmental image of a scene in which an edge device is located; then, the target image is input into a pre-constructed target detection model to predict the classification results and position information of each target object in the target image and the classification results at the pixel level; wherein the target detection model includes a backbone network layer, a neck network layer, a semantic branch network layer, a classification network layer and a regression network layer; the target detection model is based on a contrastive language-image pre-training (CLIP) model and a feature encoding network, and is obtained by optimizing and training using pre-collected sample images; the target detection model learns the image representation capability of the CLIP model, and then, the classification results and position information of each target object and the classification results at the pixel level are fused to obtain the detection results of the target objects in the target image.

[0065] It can be seen that since the present application first constructs a target detection model based on the CLIP model and the feature encoding network, and uses pre-collected sample images for optimization training, the model is able to learn the image representation ability of the CLIP model. Therefore, when the target detection model is used in the edge device to detect the target object in the target image, the accuracy and stability of the detection result can be effectively improved without increasing any computational complexity or delay, thereby improving the target detection effect of edge devices with limited computing resources (such as sweeping robots, learning machines or home robots, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 A schematic diagram of a target detection method provided in an embodiment of the present application;

[0068] Figure 2 An example diagram of the overall process of detecting a target object in a target image provided by an embodiment of the present application;

[0069] Figure 3 A schematic diagram of the structure of the target detection model provided in the embodiment of the present application;

[0070] Figure 4 A schematic diagram of the training process of the target detection model provided in the embodiment of the present application;

[0071] Figure 5 A schematic diagram of the composition of a target detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] Object detection is a basic technology in computer vision, which aims to detect and locate objects from images or video streams. It has many applications, including but not limited to self-driving cars, sweeping robots, video security, etc. At present, the existing object detection methods generally include the following three:

[0073] (1) Two-stage target detection algorithm. This method requires first automatically extracting the area to be detected that may contain the target object, and then further completing a series of detection or classification tasks on the area to be detected. For example, the two-stage target detection algorithm based on the region-based convolutional neural network (RCNN) algorithm first extracts the candidate box based on the image, and then performs secondary correction based on the candidate area to obtain the detection point result. Although the accuracy of the two-stage target detection algorithm is relatively high due to the screening effect of the candidate box, the calculation of the candidate area selection slows down the detection speed of the overall framework, which inevitably leads to higher latency and larger model parameters. These also make this detection method difficult to deploy on edge devices with limited computing resources (such as sweeping robots, learning machines, or home robots).

[0074] (2) An end-to-end visual transformer detection method based on image block matching. Due to the particularity of its structure, this method also has a strong dependence on computing resources and is not suitable for edge devices with limited computing resources.

[0075] (3) Single-stage target detection algorithm. Although this method can make up for the shortcomings of the above two detection algorithms in terms of model structure and reasoning logic, its post-processing of the prediction results depends on artificially pre-designed rules, which inevitably affects the detection effect of the model. For example, the single-stage target detection algorithm based on the YOLO algorithm can generate detection results by directly calculating the detection image to improve the detection speed of the algorithm, but it also inevitably has a certain impact on the detection accuracy.

[0076] It can be seen that the above-mentioned existing target detection methods cannot achieve ideal detection effects for edge devices with limited computing resources.

[0077] In recent years, with the continuous upgrading of computing resources, especially graphics processing units (GPUs), neural network processing units (NPUs), and tensor processing units (TPUs) with powerful parallel computing capabilities, deep learning methods based on neural networks have achieved increasingly excellent results in fields including computer vision and natural language processing. In particular, solutions for target detection and image segmentation tasks have also made new breakthroughs.

[0078] Among them, image segmentation tasks mainly include semantic segmentation tasks, instance segmentation tasks and panoramic segmentation tasks. The main goal of the semantic segmentation task is to classify each category in the current image at the pixel level to obtain the contour information of each category of objects, but do not distinguish between different individuals of the same category. When the task requirement becomes to distinguish different instances of the same category on the basis of semantic segmentation, then semantic segmentation will further evolve into an instance segmentation task. The panoramic segmentation task combines the instance segmentation task and the semantic segmentation task at the same time. In the panoramic segmentation task, not only the foreground target (things category, that is, countable categories, such as vehicles, tables, etc.) needs to be instance segmented, but also the background target (stuff category, that is, uncountable categories, such as walls, skies, etc.) needs to be semantically segmented.

[0079] It should be noted that the CLIP model is a basic large model widely used in the current computer vision field. It learns the mapping relationship between images and their text descriptions based on natural language, and has very good text and image representation capabilities. Although the CLIP model has very good representation capabilities and has models of multiple scales to choose from, it is still very difficult to borrow the powerful representation capabilities of CLIP on edge devices with limited computing resources, especially for low-power and low-computing edge devices.

[0080] Moreover, for edge devices with limited computing resources (such as sweeping robots, learning machines, or home robots, etc.), the image processing methods used are usually designed for a single task to achieve the optimal performance of the proprietary model, that is, only the task of detecting the target position of the target object in the image is completed, or only the task of classifying the target object at the pixel level in the image (i.e., the semantic segmentation task) is completed, and both tasks cannot be completed at the same time. In addition, due to limited computing resources, models with powerful representation capabilities such as CLIP cannot be deployed, resulting in the inability of edge devices to achieve ideal target detection effects, such as inaccurate detection results, and the inability to simultaneously complete the detection of the target position and the pixel-level classification of the target object.

[0081] To solve the above-mentioned defects, the present application provides a target detection method, first, obtaining a target image to be detected; wherein the target image is an environmental image of the scene in which the edge device is located, and then inputting the target image into a pre-built target detection model to predict the classification results and position information of each target object in the target image and the classification results at the pixel level; wherein the target detection model includes a backbone network layer, a neck network layer, a semantic branch network layer, a classification network layer, and a regression network layer; the target detection model is based on the CLIP model and the feature encoding network, and is optimized and trained using pre-collected sample images; the target detection model learns the image representation ability of the CLIP model, and then, the classification results and position information of each target object and the classification results at the pixel level are fused to obtain the detection results of the target objects in the target image.

[0082] It can be seen that since the present application is first based on the CLIP model and the feature encoding network, a target detection model (a small model consisting of a backbone network layer, a neck network layer, a semantic branch network layer, a classification network layer, and a regression network layer) is constructed by optimizing and training using pre-collected sample images, so that the small model can learn the image representation ability of the CLIP model. When the small model is used in the edge device to detect the target object in the target image, the accuracy and stability of the detection result can be effectively improved without increasing any calculation amount or delay, thereby improving the target detection effect of edge devices with limited computing resources (such as sweeping robots, learning machines, or household robots, etc.), that is, not only the accuracy of the detection result of the target object in the target image is improved, but also the detection of the target position of the target object and the pixel-level classification of the target object can be completed simultaneously.

[0083] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0084] First embodiment

[0085] See also Figure 1 , is a flow chart of a target detection method provided in this embodiment, which is applied to an edge device and may specifically include the following steps:

[0086] S101: Acquire a target image to be detected; wherein the target image is an environment image of a scene in which the edge device is located.

[0087] In this embodiment, any image that uses this embodiment to perform target detection is defined as a target image, and the object to be detected (or identified) contained in the target image is defined as a target object. The target image usually refers to the environmental image of the scene where the edge device is located, and the target object contained therein refers to any object that needs to be detected in a timely manner in the environment where the edge device is located. The specific content is not limited and can be set according to actual conditions. Taking the edge device as a sweeping robot as an example, the target object to be detected may refer to a puppy, a table, a door frame, etc. in the living room where the sweeping robot is located.

[0088] Furthermore, it should be noted that the present application does not limit the method for acquiring the target image. For example, an image of the environment in which the target image is located can be captured by a visual module (such as a camera) installed on an edge device as the target image. Furthermore, the present embodiment does not limit the type of target image. For example, the target image can be a color image composed of the three primary colors of red (R), green (G), and blue (B), or can be a grayscale image, or can be an image containing RGB color information and depth information (i.e., an RGBD image), etc. For example, the target image can be an environmental image of the living room (i.e., the real application scenario) in which the sweeping robot (and edge device) is located, captured by a camera installed on the sweeping robot (and edge device), such as Figure 2 shown.

[0089] S102: Input the target image into a pre-built target detection model to predict the classification results and position information of each target object in the target image and the classification results at the pixel level; wherein the target detection model includes a backbone network layer, a neck network layer, a semantic branch network layer, a classification network layer, and a regression network layer; the target detection model is based on the contrastive language image pre-trained CLIP model and feature encoding network, and is optimized and trained using pre-collected sample images; the target detection model learns the image representation ability of the CLIP model.

[0090] In this embodiment, after the target image to be detected is acquired through step S101, in order to effectively improve the accuracy and stability of target detection by the edge device and achieve an ideal detection effect, the target image can be further input into a pre-built target detection model (pre-deployed in the edge device) for processing, so as to predict the pixel-level classification result corresponding to the target image (that is, the classification result of each pixel in the target image), as well as the classification result of each target object to be detected in the target image (that is, the specific category information of the target object) and position information (such as the minimum circumscribed rectangular frame of the target object) to execute the subsequent step S103.

[0091] It should be noted that in order to improve the accuracy and stability of target detection by edge devices and achieve ideal detection effects, the present application uses a large number of collected sample images in advance, based on the CLIP model and feature encoding network, to optimize and train a small model consisting of a backbone network layer, a neck network layer, a semantic branch network layer, a classification network layer, and a regression network layer, and constructs a small model (i.e., a target detection model) that has learned the image representation capabilities of the CLIP model. The small model (i.e., the target detection model) is no longer constrained by memory resources or computing resources, so it can be deployed to edge devices for more accurate target detection processing, so as to improve the accuracy and stability of target detection by edge devices and achieve ideal detection effects. In addition, it should be noted that the present application does not limit the specific network composition structure of the target detection model, and it can be selected and set according to actual conditions. The present application adopts only a preferred implementation method, that is, the target detection model is composed of but not limited to a backbone network layer, a neck network layer, a semantic branch network layer, a classification network layer, and a regression network layer, such as Figure 3 shown.

[0092] Specifically, an optional implementation method is that the specific implementation process of "inputting the target image into a pre-built target detection model to predict the classification results and position information of each target object in the target image and the classification results at the pixel level" in step S102 may include the following steps S1021-S1024:

[0093] S1021: Input the target image into the backbone network layer of the pre-built target detection model for feature extraction to obtain multi-scale features corresponding to the target image.

[0094] In this implementation, after the target image to be detected (such as the environment image of the living room captured by the camera installed on the sweeping robot) is acquired in step S101, Figure 2 As shown, the target image can be further input into the backbone network layer of the target detection model for feature extraction to obtain multi-scale features (also a type of visual feature) corresponding to the target image to execute the subsequent step S1022.

[0095] Among them, this application does not limit the specific composition structure of the backbone network layer, and it can be selected according to actual conditions and experience. For example, according to the computing power characteristics of the deployed edge device, you can select any suitable full convolution backbone network (such as residual network (ResNet), GoogLeNet, etc.) as the backbone network layer to extract the multi-scale features of the target image, such as feature maps of multiple scales with sizes of 100*100, 50*50, 25*25, etc.

[0096] S1022: Input the multi-scale features into the neck network layer of the target detection model for feature fusion processing to obtain fusion features corresponding to the target image.

[0097] In this implementation, after obtaining the multi-scale features corresponding to the target image through step S1021, the multi-scale features can be further input into the neck network layer of the target detection model for feature fusion processing to obtain the fusion features (also a kind of visual features) corresponding to the target image for executing the subsequent step S1023.

[0098] Among them, the main function of the neck network layer is to further fuse the multi-scale features output by the backbone network layer so that the feature information at each scale can complement each other, and at the same time, any existing or future feature fusion method is used to fuse the multi-scale features to output a fusion feature (such as a fusion feature map) of a unified scale (such as 50*50). For example, multi-scale feature map fusion solutions such as Feature Pyramid Network (FPN) and Path Aggregation Network (PANet) can be used to fuse multi-scale features.

[0099] S1023: Input the fused features into the semantic branch network layer of the target detection model to obtain the pixel-level classification result corresponding to the target image.

[0100] In this implementation, after the fusion feature corresponding to the target image is obtained in step S1022, the fusion feature can be further input into the semantic branch network layer of the target detection model (which can be regarded as the semantic branch of the model) for image segmentation (specifically semantic segmentation) to obtain the pixel-level classification result corresponding to the target image (that is, the classification result of each pixel in the target image, such as Figure 2 The classification result of whether each pixel belongs to the door frame or other categories is used to execute the subsequent step S1024.

[0101] Among them, the present application does not limit the specific composition structure of the semantic branch network layer, which can be selected according to actual conditions and experience values. For example, any full convolutional neural network module (such as ResNet, etc.) can be used as the semantic branch network layer, and the fusion features output by the neck network layer can be used as input to output the pixel-level image segmentation result, that is, the pixel-level classification result corresponding to the target image.

[0102] S1024: Input the fused features into the classification network layer and the regression network layer of the target detection model respectively to obtain the classification results and position information of each target object in the target image.

[0103] In this implementation, after the fusion features corresponding to the target image are obtained in step S1022, the fusion features can be further input into the classification network layer and the regression network layer of the target detection model (the two can be regarded as the detection branches of the model) to perform image segmentation (specifically instance segmentation or panoptic segmentation) processing to obtain the classification results of each target object in the target image (that is, the specific category information of each target object in the target image, such as Figure 2 The classification results of specific categories such as door frames in the target image) and location information (i.e. the minimum bounding rectangle of each target object in the target image, such as Figure 2 The minimum circumscribed rectangular frame of the middle door frame, etc.) is used to execute the subsequent step S103.

[0104] Among them, the present application does not limit the specific composition structure of the classification network layer and the regression network layer, and can be selected according to actual conditions and experience values. For example, any full convolutional neural network module (such as ResNet, etc.) can be used as the classification network layer and the regression network layer, and the fusion features output by the neck network layer are used as input to output the instance-level classification results corresponding to the target image (that is, the specific category information of the target object) and the location information of the instance target (that is, the minimum circumscribed rectangular frame of the target object) respectively.

[0105] Next, this embodiment will introduce the training process of the target detection model in detail. Figure 4 As shown in , it is based on the CLIP model and an additional feature encoding network (the specific structure is not limited and can be set according to actual conditions and experience values), and is obtained by optimizing and training using pre-collected sample images, as shown in Figure 4 In the upper middle part of the figure, during the training process of the initial target detection model (which can be considered as a small model consisting of the initial target detection model including the initial backbone network layer, the initial neck network layer, the initial semantic branch network layer, the initial classification network layer, and the initial regression network layer), the sample image is output through the semantic branch (i.e., the initial semantic branch network layer) and the detection branch (i.e., the initial classification network layer and the initial regression network layer) of the initial target detection model, respectively, and the pixel-level classification results and the classification results and location information of the sample object are predicted, and supervised training is performed under the supervision of the corresponding labels. At the same time, the auxiliary learning branch (i.e., Figure 4 The additional feature encoding network shown in Figure 1 maps the visual features output by the initial target detection model (i.e., the fusion features of the sample image mentioned later) to the latent space to obtain the image features of the sample image. In the CLIP-assisted training process, the category labels of the sample objects contained in the sample image first form a series of text descriptions (such as Figure 4The CLIP model outputs the corresponding text features. With the powerful representation ability of the CLIP model, there is a certain mapping relationship between text features and image features in the latent space. Therefore, the present application uses the obtained text features to assist in training the visual features output by the initial target detection model, so that the visual features output by the initial target detection model and the text features and category ID (determined by the category label of the sample object) output by the CLIP model are mutually constrained. For example, Figure 4 In the example, after the visual features extracted by the initial target detection model of the sample image of the dog category are converted into image features through the feature encoding network, they will be constrained by the text features extracted by the CLIP model for the text description "a photo of dog" corresponding to the category. As a result, the trained initial target detection model (i.e., the target detection model) can not only learn the image representation ability of the CLIP model, but also grasp the implicit relationship between the category ID and the text features. This also lays the foundation for the zero-sample application of the target detection model, improves its generalization, and enables the edge devices that deploy the model to have stronger fault tolerance space and fault tolerance capabilities.

[0106] Specifically, an optional implementation method is that the training process of the target detection model may specifically include the following steps A1-A5:

[0107] Step A1: Acquire a sample image; and determine a text description of a sample object to be detected in the sample image.

[0108] In this implementation, in order to build a target detection model, a lot of preparatory work needs to be done in advance, that is, a large number of images need to be collected. For example, the camera installed on the sweeping robot can be used to capture the environmental images of the various scenes (such as the living room) in which it is located (including the sample objects to be detected) as sample images, or the environmental images of the vehicle surroundings containing the sample objects to be detected captured by the on-board camera can be used as sample images. Furthermore, the label categories of the sample objects to be detected in the sample images are manually or machine-annotated, and a text description of the sample objects is constructed based on these label categories to execute subsequent steps A2-A5, so as to realize the training of the target detection model.

[0109] Specifically, for different detection scenarios, a large number of images in the scene can be collected and labeled as sample images (used to form a labeled sample set). For example, for a target detection model deployed on a household sweeping robot, in order to build a target detection model, a large number of household scene images can be collected as sample images, and the categories of the sample objects to be detected are defined according to the needs of the actual detection task, and are represented by C = {0, 1, 2, ..., N}, where N represents the number of categories to which the sample objects to be detected belong; j∈C represents the category ID of the jth category to be detected, which can be specifically set according to the actual detection needs, such as common household furniture categories, obstacle categories, household pets, human bodies, etc., and is defined as D labeled ={(X1, Y1), ..., (X i , Y i ),...,(X n , Y n )}, i = 1, 2, 3, ..., n, where n represents the number of sample images in the sample set; X i ∈R E*H*W , represents the i-th sample image in the sample set, where E represents the number of channels of the collected image. Specifically, when E=1, it means that the collected sample image is a grayscale image, when E=3, it means that the collected sample image is a color RGB image, and when E=4, it means that the collected sample image is an RGBD image; Y i Represents the corresponding annotation information. This annotation information can be quickly obtained through existing or future annotation software, which mainly includes pixel-level category information and the minimum bounding rectangle of each instance sample (sample object). The sample image is an image collected by any image acquisition device. In particular, Y i ={L i =[l i1 , l i2 , ..., l im ], A i =[a i1 , a i2 , ..., a im ], B i =[b i1 , b i2 , ..., b im ]}, where m represents the sample image X i The number of instances (i.e., sample objects) contained in ij ∈C, represents the sample image X i The label of the jth instance (i.e., sample object) in Figure 4 The sample objects to be detected in the sample image are: base station, door frame); a ijb represents the pixel-level mask annotation of the jth instance (i.e., sample object); ij Represents the minimum bounding rectangle of the jth instance (i.e., sample object).

[0110] And, the text description corresponding to the category to which these N sample objects to be detected belong can be defined as T ij ={T1, T2, ..., T N}, and use T t Indicates the text description of the t-th category. The text can be in the form of a simple category name, or a prompt word such as a photo of {name}, or other forms of text description of the category. The specific content form is not limited and can be set according to actual conditions and experience.

[0111] In addition, when the number of sample images obtained is insufficient, in order to improve the training efficiency of the model, an optional implementation method is to perform data enhancement processing on the sample images to obtain enhanced sample images to expand the number of sample images.

[0112] In this implementation, when the sample image is X i When X is used, a series of data enhancement processes P can be applied to expand the feature space of the training sample image, that is, i ←P(X i ), so that the model has stronger generalization ability. In particular, different data enhancement processes can be adopted or designed according to different specific detection tasks. For example, for a sweeping robot scene with rapid perspective changes, a camera rotation data enhancement and a motion blur data enhancement can be applied to simulate the imaging in the actual scene to expand the number of sample images. Further, the text description of the sample object to be detected in the enhanced sample image can be used to perform subsequent steps A2-A5 to improve the training effect of the target detection model.

[0113] Step A2: Input the text description of the sample object to be detected in the sample image into the CLIP model to obtain the text features corresponding to the sample image.

[0114] In this implementation, a sample image (or an enhanced sample image, and in the subsequent model training process introduction, this application will use the sample image as input data as an example to explain the model training process in detail, and the training process using the enhanced sample image as input data can be referred to and will not be repeated) is obtained through step A1 and the text description of the sample object to be detected contained therein (such as Figure 4After the text description of the sample object to be detected in the sample image is input into the CLIP model, the text features corresponding to the sample image are obtained, such as Figure 4 The three boxes corresponding to the "text features" shown in the middle position of the lower middle part of the figure are used to perform the subsequent step A5.

[0115] Step A3: Input the sample image into the initial target detection model to obtain the fusion features corresponding to the sample image, the pixel-level classification results, and the classification results and position information of each sample object.

[0116] In this implementation, the specific composition structure of the initial target detection model is not limited, and can be but not limited to a small model consisting of an initial backbone network layer, an initial neck network layer, an initial semantic branch network layer, an initial classification network layer, and an initial regression network layer. In this way, after obtaining the sample image through step A1, further, first, the sample image can be input into the initial backbone network layer of the initial target detection model for feature extraction to obtain the multi-scale features (also a kind of visual features) corresponding to the sample image, and it is defined as F i .

[0117] Then, the multi-scale feature F corresponding to the sample image can be i Input the initial neck network layer of the initial target detection model for feature fusion processing to obtain the fusion feature corresponding to the sample image (also a kind of visual feature), and define it as Used to execute the subsequent step A4.

[0118] Next, the fusion feature corresponding to the sample image The initial semantic branch network layer of the initial target detection model (which can be regarded as the semantic branch of the model) is input for image segmentation (specifically semantic segmentation) to obtain the pixel-level classification result corresponding to the sample image (that is, the classification result of each pixel in the sample image, such as Figure 4 The classification result of each pixel in the sample image belongs to a dog, a door frame, or a base station, and is defined as Used to execute the subsequent step A5.

[0119] And then the fusion features corresponding to the sample image The initial classification network layer and the initial regression network layer of the initial target detection model are input respectively (the two can be regarded as the detection branches of the model), and the classification results of each sample object in the sample image are obtained (that is, the specific category information of each sample object in the sample image, such as Figure 4The classification results of the specific categories corresponding to the dog, door frame, and base station in the sample image) and the location information (that is, the minimum bounding rectangle of each sample object in the sample image, such as Figure 4 The minimum bounding rectangle of the dog, door frame base station in the sample image, etc.), and define the two as and Used to execute the subsequent step A5.

[0120] Step A4: Input the fusion features corresponding to the sample image into the feature encoding network to obtain the image features corresponding to the sample image.

[0121] In this implementation, the fusion features corresponding to the sample image are obtained through step A3 After that, the fusion features corresponding to the sample image can be further Input the additional feature encoding network (such as Figure 2 and Figure 4 As shown in the figure, the sample image is encoded to obtain image features corresponding to the sample image, which are used to perform subsequent step A5.

[0122] Among them, this application does not limit the specific composition structure of the additional feature encoding network, which can be selected according to actual conditions and experience. For example, any full convolutional neural network module (such as ResNet, etc.) can be used as the feature encoding network, and the depth or width of the network can be appropriately adjusted according to the computing power characteristics of the edge device where the model is deployed, and then the fusion feature corresponding to the sample image output by the initial neck network layer is used. As input, the corresponding image features are output, such as Figure 4 The three boxes corresponding to the "image features" shown in the upper right corner of the lower middle part of the figure are used to interact with the text features output by the CLIP model.

[0123] Specifically, on the one hand, the CLIP model is used for the input sample image X i The m instances (i.e., sample objects) in i categories (e.g. an image containing two people and a table, m=3, N i =2), the corresponding text features can be extracted as Where D represents the dimension of the latent space. On the other hand, the additional feature encoding network can convert the visual features of the sample image output by the initial target detection model (i.e., the fusion features corresponding to the sample image) into ) is mapped to the latent space to obtain image features and is defined as Taking cosine similarity as an example, by measuring text features in latent space and image features The calculation formula of the similarity relationship can be: Among them, for any value S in S i，j，k For example, i∈{0,1,...,H}, j∈{0,1,...}, k∈{0,1,2,...,N i}, represents the sample image X i The similarity relationship between the pixel at the coordinate (i, j) in the middle and the k-th category. By measuring the maximum value of S in the m dimension and N i By establishing the relationship between the real category labels, the alignment relationship between text features and image features can be completed.

[0124] Step A5: Utilize text features, image features corresponding to sample images, pixel-level classification results, and classification results and location information of each sample object, combined with the first target loss function, the second target loss function, and the third target loss function, to optimize and train the initial target detection model to obtain a trained target detection model.

[0125] In this implementation, the text features corresponding to the sample image are obtained through step A2 (which can be expressed as ), and obtain the pixel-level classification result corresponding to the sample image through step A3 The classification results and location information of each sample object (i.e. and ), and obtain the image features corresponding to the sample image through step A4 After that, the pixel-level classification results corresponding to the sample images can be used to further optimize the training of the initial target detection model. Calculate the value of the first objective loss function; and use the classification results and position information of each sample object corresponding to the sample image (i.e. and ), calculate the value of the second objective loss function; and use text features Image features corresponding to the sample image The value of the third objective loss function is calculated until the values ​​of each objective loss function meet the preset conditions, such as the weighted sum of the three reaches the minimum value and the change range is very small (basically unchanged), or reaches the preset maximum number of iterations (such as 10,000 times), then the model parameter update is stopped, the initial target detection model training is completed, and the trained target detection model is generated. Among them, the values ​​of the first objective loss function, the second objective loss function and the third objective loss function are used to constrain the update of the model parameters so that the target detection model can learn the image representation ability of the CLIP model as much as possible.

[0126] Among them, it should be noted that when the target detection model is trained in the present application, several training tasks are included: one is the detection task (i.e., the regression task of the detection box) and the classification result (i.e., the classification task of the specific category to which the sample object belongs) of the instance (i.e., the sample object) at the location, the second is the classification task (i.e., the image segmentation task) of the instance (i.e., the sample object) at the pixel level, and the third is the alignment task between the text features output by the CLIP model and the image features converted from the visual features output by the initial target detection model (a small model composed of an initial backbone network layer, an initial neck network layer, an initial semantic branch network layer, an initial classification network layer, and an initial regression network layer).

[0127] Among them, for the regression task of the detection box, the classic loss such as Intersection over Union Loss (IoU loss), Complete Intersection over Union Loss (CIoU loss), or the more novel Shape Intersection over Union (Shape-IoU) loss can be used as the target loss function for constraint calculation. The regression loss function is now expressed as L reg , then the sample image X input to the model i There is regression loss

[0128] For the classification task of the specific category to which the sample object belongs, any classification loss function such as Binary Cross Entropy Loss (BCE loss) and Cross-Entropy Loss (CE loss) can be used as the target loss function for constraint calculation. The classification loss function is now expressed as L cls , then the sample image X input to the model i There is classification loss

[0129] In this way, the comprehensive classification loss and regression loss can determine the calculation formula of the second objective loss function used when training the detection branch of the initial target detection model (that is, the initial classification network layer and the initial regression network layer) and can be expressed as:

[0130] For image segmentation tasks, Dice Loss, cross entropy loss, or focal loss can be used as the target loss function for constraint calculation. The image segmentation loss function is now expressed as L seg, it can be determined that when training the semantic branch of the initial target detection model (i.e., the initial semantic branch network layer), the calculation formula of the first objective loss function used can be expressed as:

[0131] For the alignment task, any existing or future similarity measurement function (such as cosine similarity, etc.) can be used as the similarity loss function to align in the latent space. Now the similarity loss function is represented as SIM, and the calculation formula of the third objective loss function used in training the initial target detection model can be expressed as:

[0132] An optional implementation method is to also transform the first objective loss function Second objective loss function The third objective loss function L clip The sum (or weighted sum) is performed to form a total loss function, which is defined as L to optimize the training of the initial target detection model. During the training process, the model parameters of the initial target detection model can be continuously updated according to the change of the function value of the total loss function L until the function value of L meets the requirements, such as reaching the minimum value and the change amplitude is very small (basically unchanged), and the model converges. Then stop updating the model parameters, complete the training of the initial L, and generate a trained initial target detection model. Among them, the specific calculation formula of L can be:

[0133] In this way, after obtaining the target detection model (such as a small model composed of a backbone network layer, a neck network layer, a semantic branch network layer, a classification network layer, and a regression network layer) with the help of the CLIP model and feature encoding network training, the image processing capability of the deployed edge device can be improved without the assistance of large models such as the CLIP model, while ensuring that the model does not lose accuracy and does not increase any computing resources. The classification results and location information of each target object in the target image as well as the pixel-level classification results can be quickly and accurately predicted.

[0134] It should also be noted that after the trained target detection model is formed through the above steps A1-A5, the trained target detection model can be further verified and updated using verification images to form a target detection model with better prediction effect. The specific verification process will not be repeated here.

[0135] S103: Fusing the classification results and position information of each target object and the classification results at the pixel level to obtain detection results of the target objects in the target image.

[0136] In this embodiment, after the classification results and position information of each target object in the target image and the classification results at the pixel level are predicted and obtained in step S102, the classification results and position information of each target object and the classification results at the pixel level can be further fused using existing or future information fusion methods to obtain more accurate detection results of each target object in the target image. For example, Figure 2 The target image on the left is processed by the target detection model deployed on the sweeping robot (and edge device), and the classification results and position information of each target object in the target image as well as the pixel-level classification results can be obtained. Then, this information is fused to obtain the detection results of each target object in the target image, as shown in the figure below. Figure 2 shown.

[0137] In summary, the present embodiment provides a target detection method, first, obtaining a target image to be detected; wherein the target image is an environmental image of a scene in which an edge device is located, and then inputting the target image into a pre-constructed target detection model to predict the classification results and position information of each target object in the target image and the classification results at the pixel level; wherein the target detection model includes a backbone network layer, a neck network layer, a semantic branch network layer, a classification network layer, and a regression network layer; the target detection model is based on the CLIP model and the feature encoding network, and is optimized and trained using pre-collected sample images; the target detection model learns the image representation capability of the CLIP model, and then, fuses the classification results and position information of each target object and the classification results at the pixel level to obtain the detection results of the target objects in the target image.

[0138] It can be seen that since the present application first constructs a target detection model based on the CLIP model and the feature encoding network, and uses pre-collected sample images for optimization training, the model is able to learn the image representation ability of the CLIP model. Therefore, when the target detection model is used in the edge device to detect the target object in the target image, the accuracy and stability of the detection result can be effectively improved without increasing any computational complexity or delay, thereby improving the target detection effect of edge devices with limited computing resources (such as sweeping robots, learning machines or home robots, etc.).

[0139] Second embodiment

[0140] This embodiment will introduce a target detection device. For related content, please refer to the above method embodiment.

[0141] See also Figure 5 , is a schematic diagram of the composition of a target detection device provided in this embodiment, the device 500 includes:

[0142] The first acquisition unit 501 is used to acquire a target image to be detected; the target image is an environment image of the scene where the edge device is located;

[0143] The prediction unit 502 is used to input the target image into a pre-built target detection model, and predict the classification results and position information of each target object in the target image and the classification results at the pixel level; wherein the target detection model includes a backbone network layer, a neck network layer, a semantic branch network layer, a classification network layer, and a regression network layer; the target detection model is based on the CLIP model and feature encoding network pre-trained by contrasting language images, and is obtained by optimizing and training using pre-collected sample images; the target detection model learns the image representation ability of the CLIP model;

[0144] The fusion unit 503 is used to fuse the classification results and position information of each target object and the classification results at the pixel level to obtain the target detection result corresponding to the target image.

[0145] In an implementation of this embodiment, the prediction unit 502 includes:

[0146] A first input subunit is used to input the target image into a backbone network layer of a pre-built target detection model for feature extraction to obtain multi-scale features corresponding to the target image;

[0147] A second input subunit is used to input the multi-scale features into the neck network layer of the target detection model for feature fusion processing to obtain fusion features corresponding to the target image;

[0148] A third input subunit is used to input the fusion feature into the semantic branch network layer of the target detection model to obtain a pixel-level classification result corresponding to the target image;

[0149] The fourth input subunit is used to input the fusion features into the classification network layer and the regression network layer of the target detection model respectively to obtain the classification results and position information of each target object in the target image.

[0150] In an implementation of this embodiment, the device further includes:

[0151] A second acquisition unit is used to acquire a sample image; and determine a text description of a sample object to be detected in the sample image;

[0152] A first input unit, used for inputting a text description of a sample object to be detected in the sample image into a CLIP model to obtain a text feature corresponding to the sample image;

[0153] A second input unit is used to input the sample image into the initial target detection model to obtain the fusion features corresponding to the sample image, the pixel-level classification results, and the classification results and position information of each sample object;

[0154] A third input unit is used to input the fusion feature corresponding to the sample image into a feature encoding network to obtain the image feature corresponding to the sample image;

[0155] A training unit is used to utilize the text features, the image features corresponding to the sample images, the pixel-level classification results, and the classification results and position information of each sample object, combined with the first target loss function, the second target loss function, and the third target loss function, to optimize the training of the initial target detection model to obtain a trained target detection model.

[0156] In an implementation of this embodiment, the device further includes:

[0157] The data enhancement unit is used to perform data enhancement processing on the sample image to obtain an enhanced sample image.

[0158] In an implementation of this embodiment, the second acquiring unit is specifically configured to:

[0159] Determine the label category to which the sample object to be detected in the sample image belongs, and construct a text description of the sample object according to the label category.

[0160] In an implementation of this embodiment, the initial target detection model includes an initial backbone network layer, an initial neck network layer, an initial semantic branch network layer, an initial classification network layer, and an initial regression network layer; the second input unit is specifically used to:

[0161] A fourth input subunit, used for inputting the sample image into the initial backbone network layer of the initial target detection model for feature extraction to obtain multi-scale features corresponding to the sample image;

[0162] A fifth input subunit, configured to input the multi-scale features corresponding to the sample image into the initial neck network layer of the initial target detection model for feature fusion processing to obtain fusion features corresponding to the sample image;

[0163] A sixth input subunit, used to input the fusion feature corresponding to the sample image into the initial semantic branch network layer of the initial object detection model to obtain a pixel-level classification result corresponding to the sample image;

[0164] The seventh input subunit is used to input the fusion features corresponding to the sample image into the initial classification network layer and the initial regression network layer of the initial target detection model respectively, so as to obtain the classification results and position information of each sample object in the sample image.

[0165] In an implementation of this embodiment, the training unit is specifically used to:

[0166] When optimizing and training the initial target detection model, the value of the first target loss function is calculated using the pixel-level classification results corresponding to the sample image; and the value of the second target loss function is calculated using the classification results and position information of each sample object corresponding to the sample image; and the value of the third target loss function is calculated using the text features and the image features corresponding to the sample image, until the values ​​of each target loss function meet the preset conditions, then the updating of the model parameters is stopped, and the target detection model is obtained by training;

[0167] Among them, the values ​​of the first objective loss function, the second objective loss function and the third objective loss function are respectively used to constrain the updating of model parameters so that the target detection model can learn the image representation ability of the CLIP model as much as possible.

[0168] Furthermore, an embodiment of the present application also provides a target detection device, including: a processor, a memory, and a system bus;

[0169] The processor and the memory are connected via the system bus;

[0170] The memory is used to store one or more programs, and the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned target detection method.

[0171] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned target detection method.

[0172] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned target detection method.

[0173] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0174] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0175] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0176] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target detection method, characterized in that: Applied to edge devices, including: Acquire a target image to be detected; the target image is an environmental image of the scene where the edge device is located; The target image is input into a pre-built target detection model to predict the classification results and position information of each target object in the target image and the classification results at the pixel level; wherein the target detection model includes a backbone network layer, a neck network layer, a semantic branch network layer, a classification network layer, and a regression network layer; the target detection model is based on the CLIP model and feature encoding network pre-trained by contrasting language images, and is obtained by optimizing and training using pre-collected sample images; the target detection model learns the image representation ability of the CLIP model; The classification results and position information of each target object and the classification results at the pixel level are fused to obtain the detection results of the target objects in the target image.

2. The method according to claim 1, characterized in that The step of inputting the target image into a pre-built target detection model and predicting the classification results and position information of each target object in the target image as well as the classification results at the pixel level includes: Inputting the target image into the backbone network layer of a pre-built target detection model for feature extraction to obtain multi-scale features corresponding to the target image; Inputting the multi-scale features into the neck network layer of the target detection model for feature fusion processing to obtain fusion features corresponding to the target image; Inputting the fused features into the semantic branch network layer of the target detection model to obtain a pixel-level classification result corresponding to the target image; The fused features are respectively input into the classification network layer and the regression network layer of the target detection model to obtain the classification results and position information of each target object in the target image.

3. The method according to claim 1, characterized in that The target detection model is constructed as follows: Acquire a sample image; and determine a text description of a sample object to be detected in the sample image; Inputting the text description of the sample object to be detected in the sample image into the CLIP model to obtain the text features corresponding to the sample image; Input the sample image into the initial target detection model to obtain the fusion features corresponding to the sample image, the pixel-level classification results, and the classification results and position information of each sample object; Inputting the fusion features corresponding to the sample image into a feature encoding network to obtain image features corresponding to the sample image; The text features, the image features corresponding to the sample images, the pixel-level classification results, and the classification results and position information of each sample object are used, combined with the first target loss function, the second target loss function, and the third target loss function, to optimize the initial target detection model and obtain a trained target detection model.

4. The method according to claim 3, characterized in that After acquiring the sample image, the method further includes: Perform data enhancement processing on the sample image to obtain an enhanced sample image.

5. The method according to claim 3, characterized in that: The determining of a text description of a sample object to be detected in the sample image includes: Determine the label category to which the sample object to be detected in the sample image belongs, and construct a text description of the sample object according to the label category.

6. The method according to claim 3, characterized in that The initial target detection model includes an initial backbone network layer, an initial neck network layer, an initial semantic branch network layer, an initial classification network layer, and an initial regression network layer; the sample image is input into the initial target detection model to obtain the fusion features corresponding to the sample image, the classification results at the pixel level, and the classification results and position information of each sample object, including: Inputting the sample image into the initial backbone network layer of the initial target detection model for feature extraction to obtain multi-scale features corresponding to the sample image; Inputting the multi-scale features corresponding to the sample image into the initial neck network layer of the initial target detection model for feature fusion processing to obtain fusion features corresponding to the sample image; Inputting the fusion features corresponding to the sample image into the initial semantic branch network layer of the initial object detection model to obtain the pixel-level classification result corresponding to the sample image; The fusion features corresponding to the sample image are respectively input into the initial classification network layer and the initial regression network layer of the initial target detection model to obtain the classification results and position information of each sample object in the sample image.

7. The method according to claim 3, characterized in that The method uses the text features, the image features corresponding to the sample images, the pixel-level classification results, and the classification results and position information of each sample object, combined with the first target loss function, the second target loss function, and the third target loss function to optimize the initial target detection model and obtain the trained target detection model, including: When optimizing and training the initial target detection model, the pixel-level classification results corresponding to the sample image are used to calculate the value of the first target loss function; and the classification results and position information of each sample object corresponding to the sample image are used to calculate the value of the second target loss function; and the text features and the image features corresponding to the sample image are used to calculate the value of the third target loss function, until the values ​​of each target loss function meet the preset conditions, then the updating of the model parameters is stopped, and the target detection model is obtained by training; Among them, the values ​​of the first objective loss function, the second objective loss function and the third objective loss function are respectively used to constrain the updating of model parameters so that the target detection model can learn the image representation ability of the CLIP model as much as possible.

8. A target detection device, characterized in that: Applied to edge devices, including: A first acquisition unit is used to acquire a target image to be detected; the target image is an environment image of a scene where the edge device is located; A prediction unit is used to input the target image into a pre-built target detection model, and predict the classification results and position information of each target object in the target image and the classification results at the pixel level; wherein the target detection model includes a backbone network layer, a neck network layer, a semantic branch network layer, a classification network layer, and a regression network layer; the target detection model is based on the CLIP model and feature encoding network pre-trained by contrasting language images, and is obtained by optimizing and training with pre-collected sample images; the target detection model learns the image representation ability of the CLIP model; The fusion unit is used to fuse the classification results and position information of each target object and the classification results at the pixel level to obtain the target detection result corresponding to the target image.

9. A target detection device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 7.

Images