Target detection method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a target detection method, device, electronic device and storage medium, which relate to the field of computer vision technology. A first visual cue feature of each target object is obtained based on a detection frame of a target object marked with a preset category and a first sample image of the category of the target object. The first visual cue feature of each target object is then classified, and a second visual cue feature of each category is determined. For each category, a third visual cue feature of the category is determined based on the second visual cue feature of the category and the first visual cue feature of all target objects of the category. After extracting multi-scale features of the image to be detected, a detection frame of each object to be detected is determined based on the multi-scale features of the image to be detected. Finally, the features of the image content in each detection frame are matched with the third visual cue features of each category to determine the category of each object to be detected, thereby improving the efficiency of target detection.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target detection method, device, electronic device and storage medium. Background Art

[0002] Object detection is one of the core contents in the field of computer vision technology. It aims to identify and locate target objects in images or videos. Therefore, the task of object detection can be divided into two parts: object classification and object localization. Object classification is to determine the category of the object in the image, and object localization is to determine the location of the object in the image.

[0003] In the related art, each target detection solution can only detect a limited number of categories of objects, such as a pet detection solution can only detect pet cats and pet dogs, and a vehicle detection solution can only detect vehicles. Therefore, when a user has a target detection requirement, he needs to indicate the category he needs to detect (hereinafter referred to as the required category) to the target detection solution provider (hereinafter referred to as the provider). The provider will formulate a corresponding target detection solution based on the required category and provide it to the user.

[0004] However, as time goes by, the demand category will also change. At this time, the user needs to re-indicate the latest demand category to the provider, and the provider will re-formulate the corresponding target detection plan based on the latest demand category and provide it to the user. However, it often takes a long time to re-formulate the corresponding target detection plan, which makes it impossible for users to perform target detection in time according to their own needs, which also leads to low target detection efficiency. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a target detection method, device, electronic device and storage medium to improve the efficiency of target detection. The specific technical solution is as follows:

[0006] In a first aspect of an embodiment of the present application, a target detection method is provided, the method comprising:

[0007] Acquire a plurality of first sample images containing target objects of a plurality of preset categories, wherein the first sample images are marked with a detection frame where each target object is located and the category of the target object;

[0008] For each first sample image, extract a first multi-scale feature of the first sample image, extract a feature of the image content in a detection frame where each target object in the first sample image is located as a first content feature of the target object, and extract a position feature of the detection frame where each target object in the first sample image is located as a first position feature of the target object;

[0009] respectively splicing the sub-features of each scale in the first multi-scale features of each first sample image in the spatial dimension to obtain a spliced ​​global image feature of each first sample image;

[0010] For each target object, an attention calculation is performed using the spliced ​​global image feature of the first sample image to which the target object belongs as a value vector and a key vector, the first content feature of the target object as a query content vector, and the first position feature of the target object as a query position vector to obtain a first visual cue feature of the target object;

[0011] Respectively counting the first visual cue features of all target objects of each category to obtain the second visual cue features of each category, wherein the similarity between the second visual cue features and the first visual cue features of any target object of the category is greater than a preset similarity threshold;

[0012] For each of the preset categories, weighted summation is performed on the first visual cue features of all target objects in the category to obtain a third visual cue feature of the category, wherein the weight of each first visual cue feature in the weighted summation process is positively correlated with the similarity of the same category, and the similarity of the same category is the similarity between the first visual cue feature and the second visual cue feature of the category;

[0013] Acquire an image to be detected, and extract a second multi-scale feature of the image to be detected;

[0014] Determining, according to the second multi-scale features, a detection frame where each object to be detected in the image to be detected is located;

[0015] Respectively extracting features of the image content in the detection frame where each of the objects to be detected in the image to be detected is located as second content features of the objects to be detected;

[0016] For each object to be detected, a third visual cue feature matching the second content feature of the object to be detected is determined, and the category to which the determined third visual cue feature belongs is determined as the category of the object to be detected.

[0017] In a possible implementation, performing weighted summation on the first visual cue features of all target objects of the category to obtain the third visual cue feature of the category includes:

[0018] Performing weighted summation on the first visual cue features of all target objects of the category to obtain a fourth visual cue feature of the category, wherein the weight of each first visual cue feature in the weighted summation process is positively correlated with the similarity of the same category;

[0019] The fourth visual cue feature is used as a new second visual cue feature, and the step of performing weighted summation of the first visual cue features of all target objects of the category to obtain the fourth visual cue feature of the category is returned to, until a preset first end condition is met, and the latest fourth visual cue feature is determined as the third visual cue feature.

[0020] In a possible implementation, respectively counting the first visual cue features of all target objects of each category to obtain the second visual cue features of each category includes:

[0021] Classifying the first visual cue features of all target objects to obtain the first visual cue features of all target objects in each category;

[0022] Calculating the mean of the first visual cue features of all target objects of each category to obtain the second visual cue features of each category;

[0023] The method of performing attention calculation using the spliced ​​global image feature of the first sample image to which the target object belongs as a value vector and a key vector, the first content feature of the target object as a query content vector, and the first position feature of the target object as a query position vector to obtain a first visual cue feature of the target object includes:

[0024] Based on the deformable attention mechanism, attention calculation is performed with the spliced ​​global image features of the first sample image to which the target object belongs as the value vector and the key vector, the first content feature of the target object as the query content vector, and the first position feature of the target object as the query position vector to obtain the first visual cue feature of the target object.

[0025] In a possible implementation, extracting a first multi-scale feature of the first sample image includes:

[0026] Inputting the first sample image into a backbone feature extraction module in a first open object detection model, obtaining a multi-scale feature output by the backbone feature extraction module as a first multi-scale feature of the first sample image;

[0027] The extracting the second multi-scale feature of the image to be detected includes:

[0028] Inputting the image to be detected into a backbone feature extraction module in the first open object detection model, obtaining a multi-scale feature output by the backbone feature extraction module as a second multi-scale feature of the image to be detected;

[0029] The step of determining, according to the second multi-scale feature, a detection frame where each object to be detected in the image to be detected is located includes:

[0030] Inputting the second multi-scale feature into the detection frame module of the first open object detection model, and obtaining the detection frame output by the detection frame module where each of the objects to be detected in the image to be detected is located;

[0031] The step of determining, for each object to be detected, a third visual cue feature that matches the second content feature of the object to be detected, and determining the category to which the determined third visual cue feature belongs as the category of the object to be detected includes:

[0032] For each object to be detected, the second content feature of the object to be detected is input into the classification module of the open target detection model, so that the classification module determines a third visual cue feature that matches the second content feature of the object to be detected, and outputs the category to which the determined third visual cue feature belongs as the category of the object to be detected.

[0033] In a possible implementation, the first open object detection model is pre-trained in the following manner, including:

[0034] Inputting the second sample image into a backbone feature extraction module in a second open object detection model to obtain a multi-scale feature output by the backbone feature extraction module as a predicted multi-scale feature;

[0035] According to the difference between the predicted multi-scale features and the true value features annotated by the second sample image, adjusting the backbone feature extraction module in the second open object detection model to obtain a first open object detection model;

[0036] Alternatively, the second sample image is input into a backbone feature extraction module in a second open object detection model to obtain a multi-scale feature output by the backbone feature extraction module as a predicted multi-scale feature;

[0037] Inputting the predicted multi-scale features into a pre-trained detection module to obtain a detection result as a predicted detection result;

[0038] According to the difference between the detection result and the true value detection result annotated by the second sample image, the backbone feature extraction module in the second open object detection model is adjusted to obtain the first open object detection model.

[0039] In a possible implementation, for each of the preset categories, weighted summing of the first visual cue features of all target objects in the category to obtain the third visual cue feature of the category includes:

[0040] For each of the preset categories, attention calculation is performed based on the attention mechanism with the first visual cue features of all target objects of the category as the value vector and the key vector and the second visual cue features of the category as the query vector to obtain the third visual cue features of the category.

[0041] In a possible implementation, performing weighted summation on the first visual cue features of all target objects of the category to obtain the third visual cue feature of the category includes:

[0042] Taking the first visual cue features of all target objects of the category as the value vector and the key vector and the second visual cue features of the category as the query vector, performing attention calculation based on the deformable attention mechanism to obtain the weights of the first visual cue features of all target objects of the category;

[0043] Using the weights of the first visual cue features of all target objects of the category, weighted summation is performed on the first visual cue features of all target objects of the category to obtain a fourth visual cue feature of the category;

[0044] The fourth visual cue feature of the category is used as a new second visual cue feature, and the step of performing attention calculation based on a deformable attention mechanism with the first visual cue features of all target objects of the category as value vectors and key vectors and the second visual cue features of the category as query vectors is returned to obtain the weights of the first visual cue features of all target objects of the category, until a preset first end condition is met, and the latest fourth visual cue feature is determined as the third visual cue feature.

[0045] In a possible implementation, the step of performing attention calculation using the spliced ​​global image feature of the first sample image to which the target object belongs as a value vector and a key vector, the first content feature of the target object as a query content vector, and the first position feature of the target object as a query position vector to obtain the first visual cue feature of the target object includes:

[0046] fusing the first content feature of the target object and the first position object to obtain a query vector of the target object;

[0047] Taking the spliced ​​global image feature of the first sample image to which the target object belongs as a value vector and a key vector, performing attention calculation based on a deformable attention mechanism using the value vector, the key vector and the query vector to obtain a first weight of a first content feature of the target object;

[0048] weighting the first content feature of the target object by using the first weight to obtain a second content feature of the target object;

[0049] The second content feature of the target object is used as the new first content feature of the target object, and the first content feature of the target object and the first position object are fused to obtain the query vector of the target object. The spliced ​​global image feature of the first sample image to which the target object belongs is used as the value vector and the key vector, and the value vector, the key vector and the query vector are used to perform attention calculation based on a deformable attention mechanism to obtain a first weight of the first content feature of the target object, until a preset second end condition is met, and the newly obtained second content feature is used as the first visual cue feature of the target object.

[0050] According to a second aspect of the present application, a target detection device is provided, the device comprising:

[0051] A first sample image acquisition module, used to acquire a plurality of first sample images containing target objects of a plurality of preset categories, wherein the first sample images are marked with a detection frame where each target object is located and the category of the target object;

[0052] a first feature extraction module, configured to extract, for each first sample image, a first multi-scale feature of the first sample image, extract a feature of an image content in a detection frame where each target object in the first sample image is located as a first content feature of the target object, and extract a position feature of a detection frame where each target object in the first sample image is located as a first position feature of the target object;

[0053] A feature stitching module, used for stitching sub-features of each scale in the first multi-scale features of each first sample image in the spatial dimension to obtain a stitched global image feature of each first sample image;

[0054] A first attention calculation module is used to perform attention calculation for each target object, using the spliced ​​global image feature of the first sample image to which the target object belongs as a value vector and a key vector, the first content feature of the target object as a query content vector, and the first position feature of the target object as a query position vector, to obtain a first visual cue feature of the target object;

[0055] A feature statistics module, used for respectively counting the first visual cue features of all target objects of each category to obtain the second visual cue features of each category, wherein the similarity between the second visual cue features and the first visual cue features of any target object of the category is greater than a preset similarity threshold;

[0056] A feature summing module is used for performing weighted summation of the first visual cue features of all target objects in each preset category to obtain a third visual cue feature of the category, wherein the weight of each first visual cue feature in the weighted summation process is positively correlated with the similarity of the same category, and the similarity of the same category is the similarity between the first visual cue feature and the second visual cue feature of the category;

[0057] An image acquisition module to be detected, used to acquire the image to be detected and extract the second multi-scale features of the image to be detected;

[0058] a detection frame determination module, configured to determine a detection frame where each object to be detected in the image to be detected is located according to the second multi-scale feature;

[0059] A second feature extraction module is used to respectively extract features of the image content in the detection frame where each of the objects to be detected in the image to be detected is located as second content features of the objects to be detected;

[0060] The category determination module is used to determine, for each object to be detected, a third visual cue feature that matches the second content feature of the object to be detected, and determine the category to which the determined third visual cue feature belongs as the category of the object to be detected.

[0061] According to a third aspect of the embodiments of the present application, an electronic device is provided, including:

[0062] Memory, used to store computer programs;

[0063] The processor is used to implement any of the above-mentioned target detection methods when executing the program stored in the memory.

[0064] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned target detection methods is implemented.

[0065] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the target detection methods described above.

[0066] The embodiments of the present application provide a target detection method, device, electronic device and storage medium. For sample images containing target objects of preset categories, the spliced ​​global image features, the first content features of each target object and the position features of the detection frame where each target object is located in each sample image are obtained. For each target object, attention calculation is performed using the spliced ​​global image features as the value vector and the key vector, the first content feature as the query content vector, and the first position feature as the query position vector. Since both the query content vector and the query position vector can represent the target object, the query content vector and the query position vector can be used to determine the features related to the target object in the spliced ​​global image features as the key features (i.e., the first visual cue features). Moreover, since the spliced ​​global image features are obtained by splicing features of different scales of the first sample image, the spliced ​​global image features contain rich feature information. Regardless of whether the target object is a large-sized object or a small-sized object, the query content vector and the query position vector can be used in this embodiment to relatively accurately determine the features related to the target object as the first visual cue features. Since the first visual cue feature is a feature related to the target object, and there are certain commonalities and certain individual differences between different target objects of the same category, the first visual cue feature can be regarded as consisting of two parts: the common features of the category to which the target object belongs, and the individual difference features of the target object itself that are different from target objects of the same category. Since it can be considered that the common features of different target objects of the same category should be the same, it can be considered that the similar parts of the first visual cue features of different target objects of the same category are the common features. The second visual cue features statistically obtained in this application and similar to the first visual cue features of target objects of the same category can be approximately regarded as the similar parts, that is, the second visual cue features can be approximately regarded as the common features. The third visual cue feature is obtained by weighted summation according to the degree of similarity with the second visual cue feature, so it can be regarded as retaining the common features in the first visual cue features in the weighted summation process and discarding the individual difference features therein, so that the third visual cue feature of a category can be regarded as the common feature of all target objects in the category, that is, the third visual cue feature of a category can be regarded as the feature possessed by all objects in the category. In other words, it can be considered that if the feature of an object matches the third visual cue feature of a category, then the object can be considered to be an object of the category. Conversely, if the feature of an object does not match the third visual cue feature of a category, then the object can be considered not to be an object of the category. Therefore, the present application can subsequently use the third visual cue feature to classify the objects to be detected in the image to be detected, thereby realizing the detection of the image to be detected.Since the solution of the present application can automatically extract the common features of target objects of each category to be detected and then realize detection, the target detection method provided by the present application can automatically adapt to different detection needs. The user only needs to provide a first sample image marked with the category and detection frame of the target object, without having to contact the provider to reformulate the target detection solution, thereby effectively improving the target detection efficiency.

[0067] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0069] Figure 1 A schematic diagram of a target detection method provided in an embodiment of the present application;

[0070] Figure 2 An example diagram of splicing global image features obtained by splicing first multi-scale features provided in an embodiment of the present application;

[0071] Figure 3 A schematic diagram of calculating a first visual cue feature based on a deformable cross-attention mechanism provided in an embodiment of the present application;

[0072] Figure 4 A first flow chart for determining a third visual cue feature provided in an embodiment of the present application;

[0073] Figure 5 A second flow chart for determining a third visual cue feature provided in an embodiment of the present application;

[0074] Figure 6 A schematic diagram of calculating a third visual cue feature based on a deformable cross-attention mechanism provided in an embodiment of the present application;

[0075] Figure 7 A framework diagram of an open target detection model provided in an embodiment of the present application to implement target detection;

[0076] Figure 8 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application;

[0077] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0079] First, the professional terms in the embodiments of the present application are explained:

[0080] Receptive field: In the embodiment of the present application, it refers to the receptive field of neurons in the convolutional layer of the deep learning network, specifically an area in the input image. The neurons in the convolutional layer are only sensitive to the pixel values ​​in this area, and the size and position of this area will affect the output of the neurons.

[0081] Attention mechanism: It is a mechanism that allows focusing on the key parts of information and ignoring irrelevant parts during information processing, thereby improving processing efficiency and accuracy. In this article, the attention mechanism specifically refers to increasing the influence of key sub-features in the feature on the detection results and reducing the influence of uncritical sub-features on the detection results during feature-based detection. The attention mechanism is implemented based on value vectors, key vectors, and query vectors.

[0082] Value vector: used to represent the information to be processed. In this article, the value vector specifically refers to the features used to represent the image or the objects in the image.

[0083] Key vector: In the attention mechanism involved in this application, the key vector is equal to the value vector.

[0084] Query vector: used to represent key information, that is, to describe what kind of information is key information. In the process of the attention mechanism, by interacting the query vector with the value vector and the key vector, it is possible to determine which information in the information to be processed is key information and which is not key information.

[0085] Deformable Attention Mechanism: When processing complex spatial or sequential data, traditional attention mechanisms may not be able to adapt well to changes and irregularities in the data. For example, when processing images of objects with different shapes, traditional attention mechanisms may not be able to flexibly focus on the key parts of the objects. Deformable attention allows the attention mechanism to dynamically adjust the location or features of attention based on the content of the input data, thereby better capturing important information in the data.

[0086] Cross-attention mechanism: is an attention mechanism used to determine key information from multiple information. In this article, it specifically refers to the mechanism used to determine features that are helpful for detection from the features of multiple objects of the same category. For example, it is used to determine features that are helpful for vehicle detection from the features of multiple vehicles. The determined features can be regarded as common features used to characterize vehicles.

[0087] In related technologies, the target detection model used by users is usually developed by the provider based on user needs. The target detection model can often only detect objects of the category required by the user. If the user needs change, the user needs to re-state the needs to the provider, and the provider will develop a new target detection plan based on the new user needs.

[0088] Taking the application scenario of food hygiene inspection in a restaurant kitchen as an example, assuming that the user needs to detect whether there are rats in the restaurant kitchen, the user contacts the provider and indicates his or her need to detect rats. The provider develops a target detection model for detecting rats based on the user's needs. The user deploys a camera in the restaurant kitchen to capture images of the kitchen and inputs the images into the target detection model. Based on the detection results of the target detection model, it can be determined whether there are rats in the restaurant kitchen.

[0089] After a period of time, the user found that rats had not been detected in the restaurant kitchen for a long time, so there was no need to continue to detect whether there were rats in the restaurant kitchen. The user also found that some staff in the kitchen did not wear chef hats as required, resulting in poor food hygiene in the restaurant. Therefore, the user's needs changed from detecting rats to whether the staff wore chef hats. At this time, the user needs to contact the provider again and indicate that he has the need to detect whether the staff wear chef hats. The provider develops a target detection model for detecting chef hats and staff according to the user's new needs. The user inputs the kitchen image into the new target detection model. If the target detection model detects the staff but not the chef hat, or the distance between the detected staff and the chef hat is too far, it can be determined that the staff is not wearing a chef hat. If the chef hat and the staff are detected at the same time, and the distance between the chef hat and the staff is close, it can be determined that the staff is wearing a chef hat. Farther in this article means that the distance is greater than the preset distance threshold, and closer means that the distance is less than the preset distance threshold. The preset distance threshold can be set according to the user's actual needs or experience.

[0090] However, while the provider is redefining the target detection model according to the user's new needs, the user is unable to perform target detection in accordance with his or her new needs in a timely manner, resulting in low target detection efficiency.

[0091] In order to improve the efficiency of target detection, a first aspect of the embodiments of the present application provides a target detection method. The target detection method provided by the embodiments of the present application is described in detail below through specific embodiments.

[0092] The target detection method provided in the embodiment of the present application can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, or a mobile terminal such as a laptop computer, a mobile phone, or a tablet.

[0093] See also Figure 1 , Figure 1 The following is a flow chart of a target detection method provided in an embodiment of the present application. The method comprises the following steps:

[0094] Step S1, obtaining a plurality of first sample images containing target objects of a plurality of preset categories, wherein the first sample images are marked with a detection frame where each target object is located and the category of the target object;

[0095] Step S2, for each first sample image, extracting a first multi-scale feature of the first sample image, extracting a feature of the image content in a detection frame where each target object in the first sample image is located as a first content feature of the target object, and extracting a position feature of the detection frame where each target object in the first sample image is located as a first position feature of the target object;

[0096] Step S3, respectively splicing the sub-features of each scale in the first multi-scale features of each first sample image in the spatial dimension to obtain a spliced ​​global image feature of each first sample image;

[0097] Step S4, for each target object, using the spliced ​​global image feature of the first sample image to which the target object belongs as the value vector and the key vector, the first content feature of the target object as the query content vector, and the first position feature of the target object as the query position vector to perform attention calculation to obtain a first visual cue feature of the target object;

[0098] Step S5, respectively counting the first visual cue features of all target objects in each category to obtain the second visual cue features of each category;

[0099] wherein the similarity between the second visual cue feature and the first visual cue feature of any target object of the category is greater than a preset similarity threshold;

[0100] Step S6, for each preset category, performing weighted summation on the first visual cue features of all target objects in the category to obtain a third visual cue feature of the category;

[0101] The weight of each first visual cue feature in the weighted summation process is positively correlated with the similarity of the same category, and the similarity of the same category is the similarity between the first visual cue feature and the second visual cue feature of the category;

[0102] Step S7, obtaining an image to be detected, and extracting a second multi-scale feature of the image to be detected;

[0103] Step S8, determining a detection frame where each object to be detected in the image to be detected is located according to the second multi-scale feature;

[0104] Step S9, respectively extracting features of the image content in the detection frame where each object to be detected is located in the image to be detected as a second content feature of the object to be detected;

[0105] Step S10: for each object to be detected, determine a third visual cue feature that matches the second content feature of the object to be detected, and determine the category to which the determined third visual cue feature belongs as the category of the object to be detected.

[0106] Using the method of the embodiment of the present application, for sample images containing target objects of preset categories, the stitched global image features in each sample image, the first content features of each target object, and the position features of the detection frame where each target object is located are obtained. For each target object, attention calculation is performed using the stitched global image features as the value vector and the key vector, the first content feature as the query content vector, and the first position feature as the query position vector. Since both the query content vector and the query position vector can represent the target object, the query content vector and the query position vector can be used to determine the features related to the target object in the stitched global image features as the key features (i.e., the first visual cue features). Moreover, since the stitched global image features are obtained by stitching together features of different scales of the first sample image, the stitched global image features contain rich feature information. Regardless of whether the target object is a large-sized object or a small-sized object, in this embodiment, the query content vector and the query position vector can be used to relatively accurately determine the features related to the target object as the first visual cue features. Since the first visual cue feature is a feature related to the target object, and there are certain commonalities and certain individual differences between different target objects of the same category, the first visual cue feature can be regarded as consisting of two parts: the common features of the category to which the target object belongs, and the individual difference features of the target object itself that are different from target objects of the same category. Since it can be considered that the common features of different target objects of the same category should be the same, it can be considered that the similar parts of the first visual cue features of different target objects of the same category are the common features. The second visual cue features statistically obtained in this application and similar to the first visual cue features of target objects of the same category can be approximately regarded as the similar parts, that is, the second visual cue features can be approximately regarded as the common features. The third visual cue feature is obtained by weighted summation according to the degree of similarity with the second visual cue feature, so it can be regarded as retaining the common features in the first visual cue features in the weighted summation process and discarding the individual difference features therein, so that the third visual cue feature of a category can be regarded as the common feature of all target objects in the category, that is, the third visual cue feature of a category can be regarded as the feature possessed by all objects in the category. In other words, it can be considered that if the feature of an object matches the third visual cue feature of a category, then the object can be considered to be an object of the category. Conversely, if the feature of an object does not match the third visual cue feature of a category, then the object can be considered not to be an object of the category. Therefore, the present application can subsequently use the third visual cue feature to classify the objects to be detected in the image to be detected, thereby realizing the detection of the image to be detected.Since the solution of the present application can automatically extract the common features of target objects of each category to be detected and then realize detection, the target detection method provided by the present application can automatically adapt to different detection needs. The user only needs to provide a first sample image marked with the category and detection frame of the target object, without having to contact the provider to reformulate the target detection solution, thereby effectively improving the target detection efficiency.

[0107] Exemplarily, still taking the aforementioned restaurant kitchen scene as an example, using the target detection method provided in the embodiment of the present application, the user can first obtain a sample image containing a mouse, and mark the detection frame of the mouse in the sample image, and then input the sample image marked with the mouse detection frame into the open target detection model in sequence. The model extracts the multi-scale features of the sample image, the features of the image content in the detection frame where the mouse is located in the sample image, and the position features of each detection frame, and performs attention learning based on the attention mechanism to learn the visual cue features of the mouse. In this way, the open target detection model can be used to detect the mouse.

[0108] When the user has a need to detect a chef hat, the user only needs to obtain a sample image containing a chef hat, and mark the detection frame of the chef hat in the sample image, and then input the sample image marked with the detection frame of the chef hat into the open target detection model in sequence. The model extracts the multi-scale features of the sample image, the features of the image content in the detection frame where the chef hat is located in the sample image, and the position features of each detection frame, and performs attention learning based on the attention mechanism to learn the visual cue features of the chef hat. In this way, the open target detection model actually learns the visual cue features of the chef hat, and then the open target detection model can be used to detect the chef hat. When the user has a need to detect a target object of any new category, he only needs to obtain a sample image containing a new category, mark the detection frame of the target object of the new category in the sample image, and then input the sample image marked with the detection frame of the target object of the new category into the open target detection model for attention calculation, and then the open target detection model can be used to detect the target object of the new category.

[0109] Based on the above, the target detection method provided in the embodiment of the present application actually includes a process of determining a third visual cue feature of a preset category and a process of actually performing target detection. The above steps S1 to S6 are the process of determining the third visual cue feature of the preset category, and the above steps S7 to S10 are the process of actually performing target detection. The above steps S1 to S10 are described in detail below.

[0110] First, the process of determining the third visual cue feature of each preset category is described:

[0111] In the above step S1, the preset category is the category of the target object that the user needs to detect, which can be one category or multiple categories. The first sample image includes target objects of at least one category in the preset category, all the first sample images include target objects of all preset categories, and the number of target objects of each preset category in all the first sample images is multiple. Exemplarily, the preset categories are birds and snakes, then there may be birds in a part of the first sample images and snakes in another part of the first sample images; or there may be birds and snakes in each first sample image; or there may be birds in a part of the first sample images, snakes in a part of the first sample images, and birds and snakes in another part of the first sample images. It can be understood that the number of target objects of each preset category in each first sample image can be 0, one, or multiple, and the embodiment of the present application is not limited to this, but there is at least one target object in each first sample image.

[0112] The first sample image may be a first sample image output by other target detection models and annotated with detection frames where each target object is located and the category of the target object in each detection frame. It may also be an image obtained by a user after acquiring an image containing target objects of preset categories and annotating the detection frames where each target object is located in the image and the category of the target object in each detection frame. The embodiments of the present application are not limited to this.

[0113] In the above step S2, multi-scale features refer to features of images extracted at different scales (or resolutions), which can be obtained at different resolutions or using receptive fields of different sizes. Extracting the first multi-scale features of the first sample image can be achieved using a pre-trained deep learning model or other feature extraction methods, such as a multi-scale convolution kernel, which is not limited in the embodiments of the present application.

[0114] In the target detection task, since the target objects may have different sizes in the image and occupy different image areas, the use of multi-scale features can better adapt to the detection of targets of different sizes. For example, in traffic detection images, large-scale features can be used to determine the overall shape and position of large vehicles, and small signs on the vehicle (such as vehicle brand logos) can be finely identified through small-scale features.

[0115] The first content feature is a feature used to represent details such as texture and edge of an object, and the first position feature is a feature used to represent the position of the target object in the first sample image. The features of the image content in the detection frame where each target object of the first sample image is located can be extracted using a pre-trained deep learning model, or the features of the target object in the detection frame can be described by calculating and counting the gradient direction histogram of the detection frame area; the first position feature can be the position of the detection frame where each target object in the first sample image is located extracted as the first position feature, or it can be obtained by inputting the first sample image into a pre-trained deep learning model, or it can be obtained by encoding the detection frame, and the embodiments of the present application are not limited to this.

[0116] There may be target objects of different sizes in the same first sample image. In order to accurately identify all target objects in the first sample image, the sub-features of each scale may be spliced ​​in the spatial dimension in the manner of step S3 to obtain the spliced ​​global image features of the first sample image.

[0117] Splicing sub-features of each scale in the spatial dimension means tiling the sub-features of each scale along the spatial dimension and then splicing sub-features of multiple scales. For example, Figure 2 As shown is an example diagram of splicing the first multi-scale features to obtain a spliced ​​global image feature provided by an embodiment of the present application. Assuming that the sub-features of each scale in the first multi-scale feature 201 of the first sample image are 4 10×10×3 features, the 4 10×10×3 features can be first tiled along the spatial dimension to obtain a tiled feature 202, and finally spliced ​​in 2×2 manner to obtain a 20×20×3 spliced ​​global image feature 203.

[0118] In order to facilitate the attention calculation in step S4, the first content feature and the first position feature mentioned above can be represented in the form of vectors, namely, the query content vector and the query position vector. In a possible implementation, the query content vector and the query position vector can also be obtained by inputting the detection box into a linear layer output after sin-cos encoding.

[0119] In the above step S4, for each target object, the spliced ​​global image features of the first sample image to which the target object belongs are used as the value vector and key vector of the deformable cross-attention mechanism, the first content feature of the target object is used as the query content vector, and the first position feature of the target object is used as the query position vector for attention calculation. The attention calculation can be based on a common attention mechanism or based on a deformable cross-attention mechanism.

[0120] When using the ordinary attention mechanism for attention calculation, for each target object, the spliced ​​global image features of the first sample image to which the target object belongs are used as the value vector and the key vector, and the first content feature of the target object is used as the query vector to perform an attention calculation to obtain the first weight of the first content feature of the target object. The first content feature of the target object is weighted using the first weight to obtain the second content feature of the target object, and the second content feature is used as the first visual cue feature of the target object.

[0121] However, when using an ordinary attention mechanism for attention calculation, the attention mechanism can only simply calculate the attention weight based on a fixed similarity calculation method, and cannot dynamically adjust the weight of each first content feature according to the change of the first content feature. In the present application, since the user's demand category may change, and the size and position of target objects of different categories in the image are not fixed, if a deformable cross-attention mechanism is used for attention calculation, the attention mechanism can generate an offset according to the actual situation such as the shape and position of the target object in the image, and use the offset to dynamically adjust the matching position of the key vector and the query vector, thereby dynamically adjusting the area that needs attention. Therefore, in one possible implementation, a deformable cross-attention mechanism can be used for attention calculation.

[0122] When the deformable cross-attention mechanism is used for attention calculation, for each target object, the spliced ​​global image features of the first sample image to which the target object belongs are used as the value vector and key vector of the deformable cross-attention mechanism, the first content feature and the first position feature of the target object are fused as the query vector for attention calculation, and the first weight of the first content feature of the target object is obtained. The first content feature of the target object is weighted by the first weight to obtain the second content feature of the target object, and then the second content feature of the target object is used as the new first content feature of the target object. The step of returning the spliced ​​global image features of the first sample image to which the target object belongs as the value vector and key vector of the deformable cross-attention mechanism, and the first content feature and the first position feature of the target object are fused as the query vector for attention calculation, until the preset end condition is met, and the latest second content feature is used as the first visual cue feature of the target object.

[0123] Among them, fusing the first content feature and the first position feature of the target object may refer to summing the first content feature and the first position feature of the target object, or may refer to splicing the first content feature and the first position feature of the target object, which is not limited in this application.

[0124] like Figure 3The figure shows a schematic diagram of calculating the first visual cue feature based on the deformable cross-attention mechanism provided by an embodiment of the present application. In this embodiment, for each target object, the first multi-scale feature 201 of the first sample image to which the target object belongs is tiled and spliced ​​to obtain a spliced ​​global image feature as the value vector and key vector of the deformable cross-attention mechanism; the detection frame 204 of the target object is encoded to obtain the first position feature and the first content feature of the target object, the first content feature of the target object is used as the query content vector 206, the first position feature of the target object is used as the query position vector 205, the query position vector 205 and the query content vector 206 are summed to obtain the query vector, and the first attention calculation is performed for each weight (corresponding to Figure 3 The deformable cross attention mechanism 1 in the above example is used to weight the first content feature based on the calculated weight (i.e., the first weight in the previous text) to obtain the second content feature as the new query content vector; then, the query position vector and the query content vector are summed up again to obtain a new query vector, and the spliced ​​global image feature of the first sample image to which the target object belongs is used as the value vector and key vector of the deformable cross attention mechanism. The value vector, the key vector, and the new query vector are used to perform a second attention calculation to obtain the weights (corresponding to Figure 3 The deformable cross attention 2 in the above method is used to weight the second content feature based on the calculated weight to obtain the third content feature of the target image as the new query content vector; then the query position vector and the query content vector are summed again to obtain a new query vector, and the spliced ​​global image feature of the first sample image to which the target object belongs is used as the value vector and key vector of the deformable cross attention mechanism. The value vector, the key vector and the new query vector are used to perform the third attention calculation to obtain the weights, and the third content feature is weighted based on the calculated weight to obtain the fourth content feature of the target image as the new query content vector; ... and so on, after the Nth attention calculation (corresponding to Figure 3 After deformable cross attention N) in the image, the (N+1)th content feature is obtained, and the (N+1)th content feature is used as the first visual cue feature 208 of the target object.

[0125] It is understandable that the attention calculation is implemented by a related software module (hereinafter referred to as a calculation module), and each attention calculation can be performed in the same calculation module or in different calculation modules, for example Figure 2The calculation modules 1 to N in the formula may be the same calculation module or N different calculation modules. In this embodiment, N may be any integer greater than or equal to 1. For example, N may be 2, and the third content feature obtained by the second attention calculation is used as the first visual cue feature of the target object; for another example, if N is 4, the fifth content feature obtained by the fourth attention calculation is used as the first visual cue feature of the target object. The value of N may be set in advance based on actual experience, or after the (N+1)th content feature is obtained by the Nth attention calculation, it is determined that the difference between the Nth content feature and the (N+1)th content feature meets the preset convergence condition, and the (N+1)th content feature is used as the first visual cue feature of the target object. In other words, the preset end condition in the above text may be that all calculation modules have completed the attention calculation, or that the difference between the Nth content feature and the (N+1)th content feature meets the preset convergence condition, or that the number of attention calculations reaches the preset number threshold, which is all possible and is not limited in this application.

[0126] According to the embodiment of the present application, since the query content vector can reflect more detailed features of the target object, the spliced ​​global image features can provide more macroscopic classification clues. The attention of these two features is calculated based on the deformable cross-attention mechanism, so that the first visual cue feature contains both details and overall information, thereby improving the representation ability of the first visual cue feature for the target object, which helps to improve the accuracy of target detection. In addition, the first visual cue feature of each target object is calculated by the deformable cross-attention mechanism, which can effectively handle target objects of different sizes and shapes. For small target objects, the focus area can be adjusted to capture the features of the target object more finely to avoid losing details due to too large a receptive field; for large target objects or irregularly shaped target objects, it can adaptively fit the shape of the target object, thereby improving the accuracy of target detection.

[0127] After the first visual features of all target objects in each first sample image are calculated through the above steps S1 to S4, since there may be multiple preset categories, and the presentation forms of target objects of the same category in different first sample images may be different, for example, in the first sample image 1, the puppy's mouth is open and its eyes are closed, and in the first sample image 2, the puppy's mouth is closed and its eyes are open, which will result in different first visual cue features of target objects of the same category in different first sample images. Therefore, the first visual cue features of all target objects of each category can be counted, and then the first visual cue features of target objects of each category can be used to determine the commonality of the first visual cue features of all target objects of that category.

[0128] Among them, counting the first visual cue features of all target objects in each category can be performed by clustering the first visual cue features of all target objects in all first sample images to obtain the first visual cue features of target objects in each category, or can be performed by classifying the first visual cue features of all target objects in all first sample images according to the categories of the target objects to obtain the first visual cue features of target objects in each category.

[0129] It can be understood that the commonality between the first visual cue features of all target objects of the same category can be measured by the similarity between the first visual cue features of all target objects of the category. Taking the image features of cats as an example, if the cats in two images have similar body postures, similar hair color and texture, and similar shapes and layouts of facial features, then the similarity between the image features of the cats in the two images is relatively high.

[0130] In order to facilitate the acquisition of the commonality between the first visual cue features of all target objects of the same category, in a possible implementation, the second visual cue feature of the category can be determined based on the first visual cue features of all target objects of the same category, and the similarity between the second visual cue feature of the category and the first visual cue feature of any target object of the category is greater than a preset similarity threshold, that is, the second visual cue feature of the category is used to represent the commonality between the first visual cue features of all target objects of the category. The preset similarity threshold is set in advance based on experience, and the embodiments of the present application do not limit this.

[0131] When the first visual cue features of all target objects of the same category have commonalities, the means of the first visual cue features will tend to converge to the vicinity of the feature value corresponding to the commonality. Therefore, the mean of the first visual cue features of all target objects of the same category can be regarded as a quantitative method of the commonalities of the first visual cue features of all target objects of the category. Therefore, in a possible implementation, the mean of the first visual cue features of all target objects of each category can be calculated as the second visual cue feature of each category.

[0132] By using the embodiment of the present application, since the calculation amount of the mean is small and the mean can balance the differences between the various first visual cue features and has a certain noise resistance, the method of calculating the mean can quickly obtain a stable feature that can represent commonality.

[0133] However, the mean value can only reflect the commonality of the first visual cue features of each target object of the same category, and it cannot reflect the importance of different target objects of the same category to the category. In order to facilitate the machine to better understand the difference and connection between the first visual cue features of each target object of the same category and the second visual cue features of the category, in the above step S6, for each preset category, the first visual cue features of all target objects of the category are weighted summed to obtain the third visual cue features of the category.

[0134] For each preset category, the higher the similarity between the first visual cue feature of the target object and the second cue feature of the category, the more it can be considered that the first visual cue feature of the target object can reflect the common features of most target objects of this category. In order to make the common information occupy a more dominant position in the final summation result, so that the third visual cue feature after weighted summation can better reflect the common features of this category of target objects, in the process of weighted summation, the weight value of the first visual cue feature of the target object should be the larger, that is, in the process of weighted summation, the weight of each first visual cue feature is positively correlated with the similarity of the same category, and the similarity of the same category is the similarity between the first visual cue feature and the second visual cue feature of the category.

[0135] For each preset category, weighted summing of the first visual cue features of all target objects in the category can quickly fuse the common information represented by the second visual cue features and the difference information of each first visual cue feature.

[0136] However, if a first visual cue feature is an outlier, the first visual cue feature will still participate in the weighted summation, thereby affecting the accuracy of the third visual cue feature.

[0137] In order to reduce the influence of outliers on the accuracy of the third visual cue feature, in a possible implementation, the third visual cue feature may be determined by multiple weighted summations. Figure 4 The first flow chart of determining the third visual cue feature provided by the embodiment of the present application is shown in FIG. Figure 4 The third visual prompt features of each category determined in step S6 may specifically include:

[0138] Step S61, performing weighted summation on the first visual cue features of all target objects of the same category to obtain the fourth visual cue features of the category;

[0139] Among them, the weight of each first visual cue feature in the weighted summation process is positively correlated with the similarity of the same category;

[0140] Step S62, taking the fourth visual cue feature as a new second visual cue feature, returning to step S61, until a preset first end condition is met, and determining the latest fourth visual cue feature as the third visual cue feature.

[0141] By adopting the method of the embodiment of the present application, during multiple calculation processes, the influence of outliers can be gradually reduced by adjusting the weights, thereby reducing the influence of outliers on the accuracy of the third visual cue feature.

[0142] Based on the above, it can be known that the process of attention calculation is actually a process of determining weights and performing weighted summation. Based on this, in a possible implementation, the attention mechanism can also be used to determine the third visual cue features of each category. Therefore, in another possible implementation, the aforementioned step S6 can also be:

[0143] For each preset category, attention calculation is performed based on the attention mechanism with the first visual cue features of all target objects in the category as the value vector and the key vector and the second visual cue features of the category as the query vector to obtain the third visual cue features of the category.

[0144] By adopting the method of the embodiment of the present application, the third visual cue feature of each category can be quickly determined by performing attention calculation on the first visual cue feature and the second visual cue feature of all target objects of each preset category.

[0145] However, when using ordinary attention mechanisms for attention calculation, the attention mechanism can only simply calculate the attention weight based on a fixed similarity calculation method, and cannot dynamically adjust the weight according to the changes in the first visual cue features, and thus cannot pay attention to the characteristics of each first visual cue feature of the same category.

[0146] Based on this, in a possible implementation, a deformable cross-attention mechanism can also be used to determine the third visual cue features of each category.

[0147] like Figure 5 The second flow chart for determining the third visual cue feature provided by the embodiment of the present application is shown. The above step S6 may specifically include:

[0148] Step S601, using the first visual cue features of all target objects of the same category as the value vector and the key vector, and the second visual cue features of the category as the query vector, performing attention calculation based on a deformable attention mechanism to obtain the weights of the first visual cue features of all target objects of the category;

[0149] Step S602, using the weights of the first visual cue features of all target objects in the category to perform weighted summation on the first visual cue features of all target objects in the category, to obtain a fourth visual cue feature of the category;

[0150] Step S603: The fourth visual cue feature of the category is used as a new second visual cue feature, and the process returns to step S601 until a preset first end condition is met, and the latest fourth visual cue feature is determined as the third visual cue feature.

[0151] For example, Figure 6 Shown is a schematic diagram of calculating the third visual cue feature based on the deformable cross-attention mechanism provided in an embodiment of the present application. For each preset category, the average value of the first visual cue features 601 of all target objects in the category is calculated as the second visual cue feature 602 of the category, and then the second visual cue feature 602 of the category is used as the query vector, and the first visual cue features 601 of all target objects in the category are used as the value vector and the key vector. Attention calculation is performed through the above steps S601 and S602 to obtain the weights of the first visual cue features of all target objects in the category, and then the weights are used to weighted sum the first visual cue features of all target objects in the category to obtain the fourth visual cue feature of the category as the new second visual cue feature. Through the above step S603, attention calculation is performed on the query vector, the value vector and the key vector based on the deformable cross attention mechanism to obtain the weights of the first visual cue features of all target objects in the category, and then the weights are used to weighted sum the first visual cue features of all target objects in the category to obtain the fourth visual cue feature of the category as the new second visual cue feature... and so on, after M times of attention calculation, the third visual cue feature 603 of the category is obtained.

[0152] Similar to the attention calculation in step S4 above, each attention calculation can be performed in the same calculation module or in different calculation modules, and M can be any integer greater than or equal to 1, for example, M can be 2, and the fifth visual cue feature obtained by the second attention calculation is used as the third visual cue feature of the category. The value of M can be set in advance based on actual experience, or after the (M+3)th visual cue feature is obtained by the Mth attention calculation, it is determined that the difference between the (M+2)th visual cue feature and the (M+3)th visual cue feature meets the preset convergence condition, and the (M+3)th visual cue feature is used as the third visual cue feature of the category. In other words, the preset end condition can be that all calculation modules have completed the attention calculation, or that the difference between the (M+2)th visual cue feature and the (M+3)th visual cue feature meets the preset convergence condition, or that the number of attention calculations reaches a preset number threshold, all of which are possible, and the present application does not limit this.

[0153] By using the embodiments of the present application, the second visual cue feature can reflect the commonality of the first visual cue feature of all target objects of the category, and the first visual cue feature of all target objects of the category can reflect the diversity of the first visual cue features of all target objects of the category. In this way, in the process of machine learning, it is possible to grasp the common rules and adapt to the individual differences of different samples.

[0154] The following is an explanation of the actual target detection process:

[0155] First, in step S7, an image to be detected is obtained, and a second multi-scale feature of the image to be detected is extracted. The method of extracting the second multi-scale feature of the image to be detected in step S7 is the same as the method of extracting the first multi-scale feature of the first sample image in step S2, and will not be repeated here.

[0156] Then in step S8, based on the second multi-scale features, the detection frame of each object to be detected in the image to be detected is determined. Specifically, a convolution operation can be used to regress the multi-scale features to predict the detection frame of each object in the image to be detected.

[0157] Then, in step S9, the features of the image content in the detection frame where each object to be detected is located in the image to be detected are extracted as the second content features of the object to be detected. The method of extracting the second content features of each object to be detected in step S9 is the same as the method of extracting the first content features of the target object in step S2 above, and will not be repeated here.

[0158] Since the third visual cue features of each preset category are determined through the above steps S1 to S6, the second content features of each object to be detected are matched with the third visual cue features of each category in step S10 to determine the category of each object to be detected.

[0159] In step S10, the third visual cue feature that matches the second content feature of the object to be detected may refer to the third visual cue feature that has the greatest similarity to the second content feature of the object to be detected. For example, assuming that there are three preset categories, which are respectively recorded as preset categories A, B, and C, wherein the similarity between the third visual cue feature of preset category A and the second content feature is 85%, the similarity between the third visual cue feature of preset category B and the second content feature is 90%, and the similarity between the third visual cue feature of preset category C and the second content feature is 95%. In this example, the third visual cue feature that matches the second content feature is the third visual cue feature of preset category C. In another possible embodiment, the third visual cue feature that matches the second content feature of the object to be detected may also refer to: the third visual cue feature whose similarity to the second content feature of the object to be detected is greater than or equal to a preset similarity threshold. Still taking the above example, assuming that the preset similarity threshold is 90%, the third visual cue feature that matches the second content feature is the third visual cue feature of preset category B and the third visual cue feature of preset category C.

[0160] Furthermore, in step S10, if there is only one third visual cue feature that matches, the category of the object to be detected is determined to be the category to which the third visual cue feature belongs. For example, in the aforementioned example, the third visual cue feature that matches the second content feature only has the third visual cue feature of the preset category C, and the category of the object to be detected is determined to be the preset category C. If there are multiple third visual cue features that match, one category can be further selected from the categories to which the multiple third visual cue features belong through other classification methods, and the category of the object to be detected is determined to be the selected category. For example, in the aforementioned example, the third visual cue feature that matches the second content feature has the third visual cue feature of the preset category B and the third visual cue feature of the preset category C. Assuming that the preset category B is a vehicle and the preset category C is a tree, it can be further determined whether the object to be detected is on the road. If the object to be detected is on the road, the category of the object to be detected is determined to be a vehicle, and if the object to be detected is not on the road, the category of the object to be detected is determined to be a tree.

[0161] It can be understood that the third visual cue feature that matches the second content feature of the aforementioned object to be detected refers to: an example of a third visual cue feature whose similarity with the second content feature of the object to be detected is greater than or equal to a preset similarity threshold. There may be a situation where the similarity between the third visual cue features of all preset categories and the second content features is less than the preset similarity threshold. In this case, the position of the object to be detected in the image to be detected can be output, as well as a message to prompt the user that there is an unclassifiable object at that position. The category of the object to be detected can also be determined as background to indicate that the object to be detected is not an object of any preset category.

[0162] Still taking the above-mentioned restaurant kitchen scene as an example, assuming that the preset similarity threshold is 95%, if the third visual cue feature of the mouse is determined through the above steps S1 to S6, when performing target detection on the image 1 to be detected, the second multi-scale feature of the image 1 to be detected is first extracted, and then the detection frame of the object to be detected in the detection image 1 is obtained based on the second multi-scale feature regression prediction. Assuming that there are two detection frames determined: detection frame 1 and detection frame 2, the second content feature of the image in detection frame 1 and the second content feature of the image in detection frame 2 are extracted respectively, and then the similarity between the second content feature of the image in detection frame 1 and the third visual cue feature of the mouse is calculated to be 50%, and the similarity between the second content feature of the image in detection frame 2 and the third visual cue feature of the mouse is calculated to be 97%, then it is considered that there is a mouse in detection frame 2.

[0163] In a possible implementation, the above steps S2 to S6 may be implemented using different models or the same model. When the above steps S2 to S6 are implemented using the same model, steps S7 to S10 are implemented using the trained model, which is hereinafter referred to as the first open object detection model.

[0164] The first open target detection model includes a backbone feature extraction module, a detection frame module and a classification module. The first sample image is input into the backbone feature extraction module in the first open target detection model to obtain the multi-scale features output by the backbone feature extraction module as the second multi-scale features of the image to be detected; the second multi-scale features are input into the detection frame module of the first open target detection model to obtain the detection frame where each object to be detected in the image to be detected is output by the detection frame module; for each object to be detected, the second content feature of the object to be detected is input into the classification module of the open target detection model so that the classification module determines the third visual cue feature that matches the second content feature of the object to be detected, and outputs the category to which the determined third visual cue feature belongs as the category of the object to be detected.

[0165] By adopting the method of the embodiment of the present application, the category of each object to be detected can be quickly determined using a pre-trained first open target detection model, thereby improving the efficiency of target detection.

[0166] In a possible implementation, the first open object detection model may be trained in the following manner:

[0167] Inputting the second sample image into a backbone feature extraction module in a second open object detection model, obtaining a multi-scale feature output by the backbone feature extraction module as a predicted multi-scale feature;

[0168] According to the difference between the predicted multi-scale features and the true value features annotated by the second sample image, adjusting the backbone feature extraction module in the second open object detection model to obtain the first open object detection model;

[0169] In another possible implementation, the first open object detection model may also be trained in the following manner:

[0170] Inputting the second sample image into a backbone feature extraction module in a second open object detection model, obtaining a multi-scale feature output by the backbone feature extraction module as a predicted multi-scale feature;

[0171] Input the predicted multi-scale features into the pre-trained detection module to obtain the detection result as the predicted detection result;

[0172] According to the difference between the detection result and the true value detection result annotated by the second sample image, the backbone feature extraction module in the second open object detection model is adjusted to obtain the first open object detection model.

[0173] By adopting the method of the embodiment of the present application, the backbone feature extraction module of the second open target detection model is adjusted based on the difference between the multi-scale features output by the second open target detection model and the true value features annotated by the second sample image, or based on the difference between the detection results output by the second open target detection model and the true value detection results annotated by the second sample image, thereby improving the accuracy of target detection by the open target detection model.

[0174] In order to enable the open target detection model to accurately extract the visual cue features of target objects of a preset category, in a possible implementation, the open target detection model also includes a visual cue feature extraction module and a visual cue feature fusion module, the visual cue feature extraction module is used to implement the above step S4, and the visual cue feature fusion module is used to implement the above steps S5 to S6.

[0175] The framework diagram of the open target detection model to achieve target detection is as follows Figure 7 As shown. First, the detection frame 701 marked with the target object and the sample image 702 of the target object category are input into the backbone feature extraction network 703 of the open target detection model, and the multi-scale features 704 of the sample image, the features of the image content in each detection frame, and the position features of each detection frame are output. Then, the multi-scale features of the sample image, the features of the image content in each detection frame, and the position features of each detection frame are input into the visual cue extraction module 705, and the visual cue features of each target object are output. The visual cue features of each target object are input into the visual cue feature fusion module 706 to classify according to the category of the target object, determine the second visual cue features of each category, and perform attention calculation to obtain the third visual cue features of each category.

[0176] During actual detection, the image to be tested 707 is input into the backbone feature extraction network 703 of the open target detection model to output the multi-scale features 709 of the image to be tested, and then regression prediction is performed in the regression unit 711 based on the multi-scale features 709 of the image to be tested to obtain the detection frames where the objects to be detected in the image to be tested 707 are located, and then the features of the image content in each detection frame are extracted, and the similarity scores between the features of the image content in each detection frame and the third visual cue features of each category are calculated through the classification module 710, so as to determine the category of the object to be detected in the image to be tested.

[0177] Corresponding to the first aspect, the second aspect of the embodiment of the present application provides a target detection device, such as Figure 8 As shown, the device comprises:

[0178] The first sample image acquisition module 801 is used to acquire a plurality of first sample images containing target objects of a plurality of preset categories, wherein the first sample images are marked with a detection frame where each target object is located and the category of the target object;

[0179] A first feature extraction module 802 is used to extract, for each first sample image, a first multi-scale feature of the first sample image, and extract features of the image content in a detection frame where each target object in the first sample image is located as a first content feature of the target object, and extract position features of the detection frame where each target object in the first sample image is located as a first position feature of the target object;

[0180] A feature stitching module 803 is used to stitch the sub-features of each scale in the first multi-scale features of each first sample image in the spatial dimension to obtain a stitched global image feature of each first sample image;

[0181] A first attention calculation module 804 is used to perform attention calculation for each target object by using the spliced ​​global image feature of the first sample image to which the target object belongs as a value vector and a key vector, the first content feature of the target object as a query content vector, and the first position feature of the target object as a query position vector to obtain a first visual cue feature of the target object;

[0182] A feature statistics module 805 is used to respectively count the first visual cue features of all target objects of each category to obtain the second visual cue features of each category, wherein the similarity between the second visual cue features and the first visual cue features of any target object of the category is greater than a preset similarity threshold;

[0183] A feature summing module 806 is used to perform weighted summation of the first visual cue features of all target objects in each preset category to obtain a third visual cue feature of the category, wherein the weight of each first visual cue feature in the weighted summation process is positively correlated with the similarity of the same category, and the similarity of the same category is the similarity between the first visual cue feature and the second visual cue feature of the category;

[0184] The image acquisition module 807 is used to acquire the image to be detected and extract the second multi-scale features of the image to be detected;

[0185] A detection frame determination module 808, configured to determine a detection frame where each object to be detected in the image to be detected is located according to the second multi-scale feature;

[0186] The second feature extraction module 809 is used to extract features of the image content in the detection frame where each object to be detected is located in the image to be detected, as a second content feature of the object to be detected;

[0187] The category determination module 810 is used to determine, for each object to be detected, a third visual cue feature that matches the second content feature of the object to be detected, and determine the category to which the determined third visual cue feature belongs as the category of the object to be detected.

[0188] In a third aspect of the embodiments of the present application, an electronic device is provided, such as Fig. 9 As shown, including:

[0189] Memory 901, used for storing computer programs;

[0190] The processor 902 is used to execute the program stored in the memory 901, and implements the following steps:

[0191] Acquire a plurality of first sample images containing target objects of a plurality of preset categories, wherein the first sample images are marked with a detection frame where each target object is located and the category of the target object;

[0192] For each first sample image, extract a first multi-scale feature of the first sample image, extract a feature of the image content in a detection frame where each target object in the first sample image is located as a first content feature of the target object, and extract a position feature of the detection frame where each target object in the first sample image is located as a first position feature of the target object;

[0193] respectively splicing the sub-features of each scale in the first multi-scale features of each first sample image in the spatial dimension to obtain a spliced ​​global image feature of each first sample image;

[0194] For each target object, the first visual cue feature of the target object is obtained by performing attention calculation using the spliced ​​global image feature of the first sample image to which the target object belongs as the value vector and the key vector, the first content feature of the target object as the query content vector, and the first position feature of the target object as the query position vector;

[0195] The first visual cue features of all target objects in each category are counted respectively to obtain the second visual cue features of each category, wherein the similarity between the second visual cue features and the first visual cue features of any target object in the category is greater than a preset similarity threshold;

[0196] For each preset category, weighted summation is performed on the first visual cue features of all target objects in the category to obtain the third visual cue feature of the category, wherein the weight of each first visual cue feature in the weighted summation process is positively correlated with the similarity of the same category, and the similarity of the same category is the similarity between the first visual cue feature and the second visual cue feature of the category;

[0197] Acquire an image to be detected, and extract a second multi-scale feature of the image to be detected;

[0198] Determine, according to the second multi-scale feature, a detection frame where each object to be detected in the image to be detected is located;

[0199] Extracting features of the image content in the detection frame where each object to be detected is located in the image to be detected as a second content feature of the object to be detected;

[0200] For each object to be detected, a third visual cue feature matching the second content feature of the object to be detected is determined, and the category to which the determined third visual cue feature belongs is determined as the category of the object to be detected.

[0201] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0202] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0203] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above target detection methods are implemented.

[0204] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any target detection method in the above embodiments.

[0205] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0206] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0207] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0208] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A target detection method, characterized in that: The method comprises: Acquire a plurality of first sample images containing target objects of a plurality of preset categories, wherein the first sample images are marked with a detection frame where each target object is located and the category of the target object; For each first sample image, extract a first multi-scale feature of the first sample image, extract a feature of the image content in a detection frame where each target object in the first sample image is located as a first content feature of the target object, and extract a position feature of the detection frame where each target object in the first sample image is located as a first position feature of the target object; respectively splicing the sub-features of each scale in the first multi-scale features of each first sample image in the spatial dimension to obtain a spliced ​​global image feature of each first sample image; For each target object, an attention calculation is performed using the spliced ​​global image feature of the first sample image to which the target object belongs as a value vector and a key vector, the first content feature of the target object as a query content vector, and the first position feature of the target object as a query position vector to obtain a first visual cue feature of the target object; Respectively counting the first visual cue features of all target objects of each category to obtain the second visual cue features of each category, wherein the similarity between the second visual cue features and the first visual cue features of any target object of the category is greater than a preset similarity threshold; For each of the preset categories, weighted summation is performed on the first visual cue features of all target objects in the category to obtain a third visual cue feature of the category, wherein the weight of each first visual cue feature in the weighted summation process is positively correlated with the similarity of the same category, and the similarity of the same category is the similarity between the first visual cue feature and the second visual cue feature of the category; Acquire an image to be detected, and extract a second multi-scale feature of the image to be detected; Determining, according to the second multi-scale features, a detection frame where each object to be detected in the image to be detected is located; Respectively extracting features of the image content in the detection frame where each of the objects to be detected in the image to be detected is located as second content features of the objects to be detected; For each object to be detected, a third visual cue feature matching the second content feature of the object to be detected is determined, and the category to which the determined third visual cue feature belongs is determined as the category of the object to be detected.

2. The method according to claim 1, characterized in that The weighted summing of the first visual cue features of all target objects of the category to obtain the third visual cue feature of the category includes: Performing weighted summation on the first visual cue features of all target objects of the category to obtain a fourth visual cue feature of the category, wherein the weight of each first visual cue feature in the weighted summation process is positively correlated with the similarity of the same category; The fourth visual cue feature is used as a new second visual cue feature, and the step of performing weighted summation of the first visual cue features of all target objects of the category to obtain the fourth visual cue feature of the category is returned to, until a preset first end condition is met, and the latest fourth visual cue feature is determined as the third visual cue feature.

3. The method according to claim 1, characterized in that The respectively counting of the first visual cue features of all target objects of each category to obtain the second visual cue features of each category includes: Classifying the first visual cue features of all target objects to obtain the first visual cue features of all target objects in each category; Calculating the mean of the first visual cue features of all target objects of each category to obtain the second visual cue features of each category; The method of performing attention calculation using the spliced ​​global image feature of the first sample image to which the target object belongs as a value vector and a key vector, the first content feature of the target object as a query content vector, and the first position feature of the target object as a query position vector to obtain a first visual cue feature of the target object includes: Based on the deformable attention mechanism, attention calculation is performed with the spliced ​​global image features of the first sample image to which the target object belongs as the value vector and the key vector, the first content feature of the target object as the query content vector, and the first position feature of the target object as the query position vector to obtain the first visual cue feature of the target object.

4. The method according to claim 1, characterized in that: The extracting the first multi-scale feature of the first sample image includes: Inputting the first sample image into a backbone feature extraction module in a first open object detection model, obtaining a multi-scale feature output by the backbone feature extraction module as a first multi-scale feature of the first sample image; The extracting the second multi-scale feature of the image to be detected includes: Inputting the image to be detected into a backbone feature extraction module in the first open object detection model, obtaining a multi-scale feature output by the backbone feature extraction module as a second multi-scale feature of the image to be detected; The step of determining, according to the second multi-scale feature, a detection frame where each object to be detected in the image to be detected is located includes: Inputting the second multi-scale feature into the detection frame module of the first open object detection model, and obtaining the detection frame output by the detection frame module where each of the objects to be detected in the image to be detected is located; The step of determining, for each object to be detected, a third visual cue feature that matches the second content feature of the object to be detected, and determining the category to which the determined third visual cue feature belongs as the category of the object to be detected includes: For each object to be detected, the second content feature of the object to be detected is input into the classification module of the open target detection model, so that the classification module determines a third visual cue feature that matches the second content feature of the object to be detected, and outputs the category to which the determined third visual cue feature belongs as the category of the object to be detected.

5. The method according to claim 4, characterized in that The first open target detection model is pre-trained in the following manner, including: Inputting the second sample image into a backbone feature extraction module in a second open object detection model to obtain a multi-scale feature output by the backbone feature extraction module as a predicted multi-scale feature; According to the difference between the predicted multi-scale features and the true value features annotated by the second sample image, adjusting the backbone feature extraction module in the second open object detection model to obtain a first open object detection model; Alternatively, the second sample image is input into a backbone feature extraction module in a second open object detection model to obtain a multi-scale feature output by the backbone feature extraction module as a predicted multi-scale feature; Inputting the predicted multi-scale features into a pre-trained detection module to obtain a detection result as a predicted detection result; According to the difference between the detection result and the true value detection result annotated by the second sample image, the backbone feature extraction module in the second open object detection model is adjusted to obtain the first open object detection model.

6. The method according to claim 1, characterized in that For each of the preset categories, weighted summing of the first visual cue features of all target objects in the category is performed to obtain the third visual cue feature of the category, including: For each of the preset categories, attention calculation is performed based on the attention mechanism with the first visual cue features of all target objects of the category as the value vector and the key vector and the second visual cue features of the category as the query vector to obtain the third visual cue features of the category.

7. The method according to claim 2, characterized in that: The weighted summing of the first visual cue features of all target objects of the category to obtain the third visual cue feature of the category includes: Taking the first visual cue features of all target objects of the category as the value vector and the key vector and the second visual cue features of the category as the query vector, performing attention calculation based on the deformable attention mechanism to obtain the weights of the first visual cue features of all target objects of the category; Using the weights of the first visual cue features of all target objects of the category, weighted summation is performed on the first visual cue features of all target objects of the category to obtain a fourth visual cue feature of the category; The fourth visual cue feature of the category is used as a new second visual cue feature, and the step of performing attention calculation based on a deformable attention mechanism with the first visual cue features of all target objects of the category as value vectors and key vectors and the second visual cue features of the category as query vectors is returned to obtain the weights of the first visual cue features of all target objects of the category, until a preset first end condition is met, and the latest fourth visual cue feature is determined as the third visual cue feature.

8. The method according to claim 1, characterized in that The method of performing attention calculation using the spliced ​​global image feature of the first sample image to which the target object belongs as a value vector and a key vector, the first content feature of the target object as a query content vector, and the first position feature of the target object as a query position vector to obtain a first visual cue feature of the target object includes: fusing the first content feature of the target object and the first position object to obtain a query vector of the target object; Taking the spliced ​​global image feature of the first sample image to which the target object belongs as a value vector and a key vector, performing attention calculation based on a deformable attention mechanism using the value vector, the key vector and the query vector to obtain a first weight of a first content feature of the target object; weighting the first content feature of the target object by using the first weight to obtain a second content feature of the target object; The second content feature of the target object is used as the new first content feature of the target object, and the first content feature of the target object and the first position object are fused to obtain the query vector of the target object. The spliced ​​global image feature of the first sample image to which the target object belongs is used as the value vector and the key vector, and the value vector, the key vector and the query vector are used to perform attention calculation based on a deformable attention mechanism to obtain a first weight of the first content feature of the target object, until a preset second end condition is met, and the newly obtained second content feature is used as the first visual cue feature of the target object.

9. A target detection device, characterized in that: The device comprises: A first sample image acquisition module, used to acquire a plurality of first sample images containing target objects of a plurality of preset categories, wherein the first sample images are marked with a detection frame where each target object is located and the category of the target object; a first feature extraction module, configured to extract, for each first sample image, a first multi-scale feature of the first sample image, extract a feature of an image content in a detection frame where each target object in the first sample image is located as a first content feature of the target object, and extract a position feature of a detection frame where each target object in the first sample image is located as a first position feature of the target object; A feature stitching module, used for stitching sub-features of each scale in the first multi-scale features of each first sample image in the spatial dimension to obtain a stitched global image feature of each first sample image; A first attention calculation module is used to perform attention calculation for each target object, using the spliced ​​global image feature of the first sample image to which the target object belongs as a value vector and a key vector, the first content feature of the target object as a query content vector, and the first position feature of the target object as a query position vector, to obtain a first visual cue feature of the target object; A feature statistics module, used for respectively counting the first visual cue features of all target objects of each category to obtain the second visual cue features of each category, wherein the similarity between the second visual cue features and the first visual cue features of any target object of the category is greater than a preset similarity threshold; A feature summing module is used for performing weighted summation of the first visual cue features of all target objects in each preset category to obtain a third visual cue feature of the category, wherein the weight of each first visual cue feature in the weighted summation process is positively correlated with the similarity of the same category, and the similarity of the same category is the similarity between the first visual cue feature and the second visual cue feature of the category; An image acquisition module to be detected, used to acquire the image to be detected and extract the second multi-scale features of the image to be detected; a detection frame determination module, configured to determine a detection frame where each object to be detected in the image to be detected is located according to the second multi-scale feature; A second feature extraction module is used to respectively extract features of the image content in the detection frame where each of the objects to be detected in the image to be detected is located as second content features of the objects to be detected; The category determination module is used to determine, for each object to be detected, a third visual cue feature that matches the second content feature of the object to be detected, and determine the category to which the determined third visual cue feature belongs as the category of the object to be detected.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images