Target detection method, target detection device, medium, equipment and product

Abstract

Embodiments of the present specification provide a target detection method, a target detection device, a computer readable storage medium, an electronic device and a computer program product, the method comprising: performing feature processing on a to-be-detected image to obtain a to-be-detected feature group containing a plurality of features; inputting the features in the to-be-detected feature group corresponding to the to-be-detected image into an encoder of a target detection model, and performing enhancement processing on the features in the to-be-detected feature group through a self-attention mechanism of the encoder in the model to obtain a plurality of to-be-detected enhanced features. Further, inputting target enhanced features about a target category object and the plurality of to-be-detected enhanced features into a decoder of the target detection model to determine the features about the target category object in the plurality of to-be-detected enhanced features; and finally determining position information of the target category object in the to-be-detected image according to the features about the target category object in the plurality of to-be-detected enhanced features.

Description

Technical Field

[0001] This specification relates to the field of computer vision technology, and in particular to a target detection method, a target detection device, a computer-readable storage medium, an electronic device, and a computer program product. Background Technology

[0002] Object detection is a research area within computer vision tasks. It has made significant progress from initial methods based on handcrafted features to current deep learning-based approaches. Object detection algorithms are also applied to various computer vision-related tasks, such as vehicle detection and product detection.

[0003] However, in situations where the scale and category of the detected targets are diverse (such as target detection from the perspective of a camera in a small shop setting), the accuracy and efficiency of the target detection results require the target detection algorithm to have higher robustness.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this specification, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this specification is to provide a target detection method, target detection device, computer-readable storage medium, electronic device, and computer program product that at least to some extent improves the robustness of the target detection algorithm.

[0006] Other features and advantages of this specification will become apparent from the following detailed description, or may be learned in part by practice of this specification.

[0007] According to one aspect of this specification, a target detection method is provided, the method comprising: performing feature processing on an image to be detected to obtain a test feature group containing N features, where N is a positive integer; inputting the N features in the test feature group to an encoder of a target detection model, and performing enhancement processing on the N features in the test feature group through a self-attention mechanism of the encoder to obtain N test enhanced features; inputting the target enhanced features related to a target category object and the N test enhanced features to a decoder of the target detection model to determine the features related to the target category object among the N test enhanced features; and determining the position information of the target category object in the image to be detected based on the features related to the target category object among the N test enhanced features.

[0008] In an exemplary embodiment, based on the foregoing scheme, the above-mentioned feature processing of the image to be detected to obtain a test feature group containing N features includes: performing feature extraction processing on the image to be detected to obtain N depth features; and performing position encoding on the N depth features of the image to be detected to obtain the test feature group.

[0009] The i-th feature in the above-mentioned feature group to be tested includes: the i-th depth feature and the position information of the i-th depth feature in the above-mentioned image to be detected, where i takes the value from 1 to N and is an integer including 1 and N.

[0010] In an exemplary embodiment, based on the foregoing scheme, the encoder includes a feature pyramid structure and / or a densely connected neural network structure;

[0011] The N features in the above-mentioned feature group to be tested are enhanced by the self-attention mechanism of the encoder to obtain N enhanced features to be tested, including: performing feature fusion processing on the N features in the above-mentioned feature pyramid structure and / or the above-mentioned densely connected neural network structure, and enhancing the N features in the above-mentioned feature group to be tested by the self-attention mechanism of the encoder to obtain N enhanced features to be tested.

[0012] In an exemplary embodiment, based on the foregoing scheme, before inputting the target enhancement features of the target category object and the N enhancement features to be tested into the decoder of the target detection model, the method further includes: performing feature processing on the target image containing the target category object to obtain a target feature group; and inputting the target feature group into the encoder of the target detection model, and performing enhancement processing on the features in the target feature group through the self-attention mechanism of the encoder to obtain the target enhancement features of the target category object.

[0013] In an exemplary embodiment, based on the foregoing scheme, the above-mentioned feature processing of the target image containing the target category object to obtain a target feature group includes: performing feature extraction processing on the target image to obtain M depth features and M positive integers related to the target category object; and performing position encoding on the M depth features of the target image to obtain the target feature group; wherein the j-th feature in the target feature group includes: the j-th depth feature and the position information of the j-th depth feature in the image to be detected, where j takes the value from 1 to M and is an integer including 1 and M.

[0014] In an exemplary embodiment, based on the foregoing scheme, the above-mentioned enhancement processing of the features in the target feature group through the self-attention mechanism of the encoder to obtain target enhanced features for the target category object includes: inputting the target feature group into the encoder of the target detection model, performing feature fusion processing on the features in the target feature group based on the feature pyramid structure and / or densely connected neural network structure in the encoder, and enhancing the features in the target feature group through the self-attention mechanism of the encoder to obtain target enhanced features for the target category object.

[0015] In an exemplary embodiment, based on the foregoing scheme, predicting the location information of the target category object in the image to be detected based on the features of the target category object among the N enhancement features to be tested includes: inputting the features of the target category object among the N enhancement features to be tested into the prediction layer; and obtaining the location information of the target category object in the image to be detected through regression processing of the prediction layer.

[0016] In an exemplary embodiment, based on the foregoing scheme, the target detection model adopts the transformer model.

[0017] According to another aspect of this specification, a target detection apparatus is provided, the apparatus comprising: a preprocessing module, a target detection model, and a prediction module.

[0018] The preprocessing module is used to perform feature processing on the image to be detected to obtain a test feature group containing N features, where N is a positive integer. The encoder of the target detection model is used to: after inputting the N features in the test feature group into the encoder, enhance the N features in the test feature group through the encoder's self-attention mechanism to obtain N enhanced features to be tested. The decoder of the target detection model is used to: after inputting the enhanced features about the target category object and the N enhanced features to be tested into the decoder of the target detection model, determine the features about the target category object in the N enhanced features to be tested. The prediction module is used to: determine the position information of the target category object in the image to be detected based on the features about the target category object in the N enhanced features to be tested.

[0019] According to another aspect of this specification, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target detection method as described in the above embodiments.

[0020] According to one aspect of this specification, a computer-readable storage medium is provided that stores instructions which, when executed on a computer or processor, cause the computer or processor to perform the target detection method as described in the above embodiments.

[0021] According to another aspect of this specification, a computer program product containing instructions is provided that, when the computer program product is run on a computer or processor, causes the computer or processor to perform the target detection method as described in the above embodiments.

[0022] The target detection method and apparatus, scene reconstruction model, computer-readable storage medium, electronic device, and computer program product provided in the embodiments of this specification have the following technical effects:

[0023] In the scheme provided in the embodiments of this specification, feature processing is performed on the image to be detected to obtain a set of features to be tested containing multiple features. The features in the set of features to be tested corresponding to the image to be tested are input into the encoder of the target detection model, and the features in the set of features to be tested are enhanced through the self-attention mechanism of the encoder in the model to obtain multiple enhanced features to be tested. The embodiments of this specification utilize the self-attention mechanism of the encoder in the target detection model to enhance the features corresponding to the image to be detected, which is beneficial to improving the feature generalization ability and the feature robustness.

[0024] Furthermore, the target enhancement features related to the target category object and the aforementioned multiple enhancement features to be tested are input into the decoder of the target detection model to determine the features related to the target category object among the multiple enhancement features to be tested; finally, based on the features related to the target category object among the multiple enhancement features to be tested, the location information of the target category object in the image to be detected is determined. The embodiments in this specification optimize the output features of the target detection model by combining the features of the target category object, which helps to increase the robustness of target detection.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification. It is obvious that the drawings described below are merely some embodiments of this specification, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0027] Figure 1This is a schematic flowchart of a target detection method provided in one embodiment of this specification.

[0028] Figure 2 This is a flowchart illustrating a target detection method provided in another embodiment of this specification.

[0029] Figure 3 This is a flowchart illustrating a target detection method provided in another embodiment of this specification.

[0030] Figure 4 This is a flowchart illustrating a target detection method provided in yet another embodiment of this specification.

[0031] Figure 5 This is a schematic diagram of the target detection device provided in one embodiment of this specification.

[0032] Figure 6 This is a schematic diagram of the target detection device provided in another embodiment of this specification.

[0033] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this specification. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this specification clearer, the embodiments of this specification will be described in further detail below with reference to the accompanying drawings.

[0035] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this specification more comprehensive and complete, and to fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of the embodiments described herein. However, those skilled in the art will recognize that the technical solutions described herein may be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., may be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this specification.

[0037] Furthermore, the accompanying drawings are merely illustrative diagrams of this specification and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0038] Generally, we consider object detection to be finding objects of interest in images or videos and simultaneously detecting their location and size. Unlike image classification, object detection involves not only classification but also localization, making it a multi-task problem.

[0039] Object detection techniques can be categorized into single-stage and two-stage methods. Single-stage methods include the YOLO (You Only Look Once) series, which first establishes a grid in the image and then performs class prediction and location regression within each grid cell to detect objects. However, when there are many objects in the image, especially those appearing at grid edges, this approach may result in object loss. Two-stage methods include the Faster-RCNN (Faster-Region-convolutional neural network) series. This series first detects regions of interest (ROIs) in the image using a Region Proposal Network (RPN), and then performs secondary optimization on each RPI to determine the final detected object. However, this two-stage framework leads to long training and inference cycles and a significant dependence on the proposal structure.

[0040] To address the problems existing in related technologies, this specification provides an embodiment of a target detection method, a target detection device, a computer-readable storage medium, an electronic device, and a computer program product. Specifically, the following will first describe... Figures 1 to 4 The embodiments of the target detection method provided in this specification are described in detail:

[0041] For example, Figure 1 This is a schematic flowchart illustrating a target detection method provided in one embodiment of this specification. (Reference) Figure 1 The embodiment shown in the figure includes: S110-S140.

[0042] In S110, feature processing is performed on the image to be detected to obtain a test feature group containing N features, where N is a positive integer.

[0043] In an exemplary embodiment, in a target detection scenario set up in a shop scene from the perspective of a camera, the video frames captured by the camera are the aforementioned "images to be detected".

[0044] In an exemplary embodiment, Figure 2 This is a schematic flowchart illustrating a target detection method according to another embodiment of this specification. (See reference) Figure 2 The image A to be detected is input into the preprocessing module 210. After processing by the preprocessing module 210, the feature group B corresponding to the image A to be detected is obtained. For example, the feature group B contains N features about the image A to be detected, where N is a positive integer.

[0045] In S120, the N features in the above-mentioned feature group to be tested are input into the encoder of the target detection model, and the N features in the above-mentioned feature group to be tested are enhanced through the self-attention mechanism of the encoder to obtain N enhanced features to be tested.

[0046] In the exemplary embodiment, Transformer is used as the object detection model described above. The Transformer framework is a basic model framework proposed by Google in 2017 based on the self-attention mechanism and used for Natural Language Processing (NLP) tasks. It has since been widely used in computer vision-related tasks, such as image classification.

[0047] For example, refer to Figure 2 The features in the target feature group B corresponding to the image A to be detected are input into the encoder (transformer encoder) 220 of the target detection model, and the N features in the target feature group B are enhanced through the self-attention mechanism of the encoder 220 to obtain N enhanced features C to be detected.

[0048] As can be seen, unlike existing object detection schemes, the examples in this manual are based on the transformer framework, which can effectively improve feature generalization ability and thus improve the robustness of object detection.

[0049] In S130, the target enhancement features related to the target category object and the aforementioned N enhancement features to be tested are input to the decoder of the target detection model to determine the features related to the target category object among the N enhancement features to be tested. And, in S140, based on the features related to the target category object among the N enhancement features to be tested, the position information of the target category object in the image to be detected is determined.

[0050] In an exemplary embodiment, the target category object is a previously known detection object. For example, in a target detection scenario set up from the camera's perspective in a small shop setting, if it is necessary to detect whether the product "soy sauce" exists in a certain video segment, then the video frames in that video segment are the "image to be detected," and the product "soy sauce" is the "target category object." The method for determining the "target enhancement features of the target category object" will be described in detail in the following embodiments.

[0051] In an exemplary embodiment, reference is made to Figure 2 The target augmentation feature D and the N augmentation features C to be tested are input into the decoder (transformer decoder) 230 of the target detection model to determine the feature E of the target category object among the N augmentation features to be tested. Further, the feature E of the target category object among the N augmentation features to be tested is input into the prediction layer 240 to determine the position information of the target category object in the image to be detected A.

[0052] In an exemplary embodiment, this specification embodiment sets "target enhancement features related to the target category object" and inputs them together with the enhancement features to be tested into the transformer decoder. During the decoding process of the enhancement features to be tested C in the decoder 230, features related to the target category object are found among the aforementioned N enhancement features to be tested. Furthermore, the location of the found features in the original image is predicted. This specification embodiment, by combining "target enhancement features related to the target category object," optimizes the output features of the transformer, thereby further improving the robustness of object detection.

[0053] As can be seen, the embodiments described herein differ from related technologies in that the target detection task requires identifying the target category and determining the target's location information. The embodiments in this specification only use a transformer decoder to identify whether a target object of a preset category exists in the image to be detected. Furthermore, if a target object exists in the image to be detected, the relevant location information is output.

[0054] In an exemplary embodiment, Figure 3This is a flowchart illustrating a target detection method provided in another embodiment of this specification.

[0055] refer to Figure 3 As a specific implementation of S110, S1102 and S1104 are executed.

[0056] In S1102, feature extraction processing is performed on the image to be detected to obtain N depth features. In S1104, the N depth features of the image to be detected are respectively position-encoded to obtain the feature group to be tested.

[0057] For example, refer to Figure 4 The preprocessing module 210 includes a feature extraction unit and a position encoding unit. The image to be processed, A, is input to the feature extraction unit to extract depth features. Further, the extracted depth features are encoded using position encoding, such as dividing the image to be detected, A, into multiple position units and encoding each position unit. Further, the depth features and position encodings are superimposed, thereby associating the depth features of the object to be detected in the image with the corresponding position encodings, thus determining the feature group to be detected. The i-th feature in the feature group to be detected includes: the i-th depth feature and the position information of the i-th depth feature in the image to be detected, where i takes values ​​from 1 to N, and is an integer containing 1 and N.

[0058] In other words, the depth features of the target object in the feature set contain its corresponding address information. By associating the retrieved target object with its corresponding location (e.g., associating the target object s with its location unit s' in the image), the location corresponding to the target object (compared to other locations in the image to be detected) can receive higher weight in the transformer encoder encoding process, which is beneficial to the prediction accuracy of the final location information.

[0059] The embodiments in this specification employ refined location coding to ensure that the location coding has a certain correlation with the object under test, thereby increasing the location coding weight of the region of interest and thus improving the accuracy of the final location information prediction.

[0060] Continue to refer to Figure 3 As a specific implementation of S120, S1202 is executed.

[0061] In S1202, based on the above-mentioned feature pyramid structure and / or the above-mentioned densely connected neural network structure, feature fusion processing is performed on the N features in the above-mentioned feature group to be tested, and the N features in the above-mentioned feature group to be tested are enhanced through the self-attention mechanism of the above-mentioned encoder to obtain N enhanced features to be tested.

[0062] For example, refer to Figure 4 The encoder 220 described above includes not only a self-attention mechanism but also a feature fusion mechanism. Specifically, to implement the feature fusion mechanism, the encoder 220 may include a feature pyramid network (FPN) structure and / or a densely connected convolutional network (DenseNet) structure.

[0063] Based on the above model structure, after inputting the N features in the above-mentioned feature group to be tested into the encoder of the target detection model, not only is the N features in the above-mentioned feature group to be tested enhanced through the self-attention mechanism of the encoder, but also the N features in the above-mentioned feature pyramid structure and / or the above-mentioned densely connected neural network structure are used to perform feature fusion processing on the N features in the above-mentioned feature group to be tested.

[0064] As can be seen, in order to solve the problem of multi-scale target detection, the example in this manual uses the transformer framework to build a network structure for multi-feature fusion, thereby realizing the fusion of multi-resolution features of the network. This is beneficial to improving the robustness of target detection for multi-scale targets in the target detection task, and to significantly improving the detection capability of multi-scale targets. This will better empower the target detection needs in the camera view of small shops, and thus realize the detection needs of multi-scale targets in the camera view of small shops.

[0065] Continue to refer to Figure 3 As a specific implementation of the method for determining the "target enhancement features of the target category object" in S130, S310 to S330 are executed.

[0066] As mentioned above, the target category objects are known detection objects in advance, and there can be one or more of them. For example, in a target detection scenario set up from the camera's perspective in a small shop setting, if it is necessary to detect whether the product "soy sauce" exists in a certain video segment, then the video frames in that video segment are the "images to be detected," and the product "soy sauce" is the "target category object." The determination of "target enhancement features about target category objects" includes: performing feature processing on the target image containing the target category objects to obtain a target feature set; and inputting the target feature set into the encoder of the target detection model, and enhancing the features in the target feature set through the self-attention mechanism of the encoder to obtain target enhancement features about the target category objects.

[0067] For example, S310 and S320 provide a specific implementation for determining the above-mentioned target feature set.

[0068] In S310, feature extraction processing is performed on the target image to obtain M depth features and M positive integers related to the target category object; and in S320, the M depth features of the target image are respectively position-encoded to obtain the target feature group.

[0069] For example, refer to Figure 4 The preprocessing module 210' includes a feature extraction unit and a position encoding unit. The target image H containing the target object is input to the feature extraction unit to extract depth features from the image. Further, the extracted depth features are encoded using position encoding, such as segmenting the target image H into multiple position units and encoding each position unit. Further, the depth features and position encodings are superimposed, thereby associating the extracted depth features related to the target object with the corresponding position encodings, thus determining the target feature group J. The j-th feature in the target feature group includes: the j-th depth feature and the position information of the j-th depth feature in the image to be detected, where j is an integer from 1 to M, inclusive.

[0070] In other words, the depth features of the target object in the target feature group J contain its corresponding address information. By associating the retrieved target object with its corresponding location (for example, associating the target object "soy sauce" with its location unit h in the image), the location corresponding to the target object (compared to other locations in the target image H) can receive higher weights during the transformer encoder encoding process, thus the resulting target enhancement features contain accurate location information of the target object.

[0071] For example, S330 provides a specific implementation of enhancing features in a target feature group.

[0072] In S330, the above target feature group is input into the encoder of the target detection model. Based on the feature pyramid structure and / or densely connected neural network structure in the encoder, feature fusion processing is performed on the features in the above target feature group. The features in the above target feature group are enhanced through the self-attention mechanism of the encoder to obtain target enhanced features for the above target category object.

[0073] For example, refer to Figure 4 The encoder 220' described above includes not only a self-attention mechanism but also a feature fusion mechanism. To implement the feature fusion mechanism, the encoder 220' may include an FPN structure and / or a DenseNet structure.

[0074] Based on the above model structure, after inputting the M features in the target feature group J into the encoder of the target detection model, not only are the M features in the target feature group J enhanced through the self-attention mechanism of the encoder, but also feature fusion processing is performed on the M features in the target feature group based on the FPN structure and / or DenseNet structure to obtain the target enhanced features D for the target category object.

[0075] Continue to refer to Figure 3 After obtaining N enhancement features C corresponding to the image to be detected through S1102, S1104 and S1102, and obtaining target enhancement features D corresponding to the target image through S310-S330, S130 is executed: the target enhancement features of the target category object and the above N enhancement features to be tested are input to the decoder of the above target detection model to determine the features of the above N enhancement features related to the target category object.

[0076] In an exemplary embodiment, reference is made to Figure 4 The target enhancement feature D for the target category object and the above N enhancement features to be tested are input into the decoder 230 of the above target detection model to determine the feature E for the target category object among the above N enhancement features to be tested.

[0077] In an exemplary embodiment, this specification embodiment sets "target enhancement features related to the target category object" and inputs them together with the enhancement features to be tested into the transformer decoder. This allows the decoder 230 to find features related to the target category object among the N enhancement features to be tested during the decoding process of the enhancement features to be tested C. This specification embodiment, by combining "target enhancement features related to the target category object," optimizes the output features of the transformer, thereby further improving the robustness of object detection.

[0078] As can be seen, the embodiments described herein differ from related technologies in that the target detection task requires identifying the target category and determining the target's location information. The embodiments in this specification only use the transformer decoder to identify whether there is a target object of a preset category in the image to be detected, which is highly targeted and the target detection results can more directly meet the user's needs.

[0079] Continue to refer to Figure 3 As a specific implementation of S140, S1402 and S1404 are executed.

[0080] In S1402, the features of the target category object from the N enhancement features to be tested are input into the prediction layer. And in S1404, the position information of the target category object in the image to be detected is obtained through regression processing by the prediction layer.

[0081] For example, the prediction layer described above can employ a multilayer perceptron (MLP). In this embodiment of the specification, the feature E concerning the target category object from the N enhancement features to be tested is input into the prediction layer 240. Since the feature E concerning the target category object carries location information, after regression processing by the prediction layer, the location information of the target category object in the image to be detected A can be determined.

[0082] Taking the target detection scenario from the perspective of a camera in a small shop as an example, and needing to detect whether the product "soy sauce" exists in a certain video segment, i.e., the product "soy sauce" is a "target category object". For each video frame of the target segment, each frame is processed using the target detection scheme provided in the embodiments of this specification. If the product "soy sauce" exists in the x-th image, the prediction layer will output the location information of the product "soy sauce" in the x-th image. If the product "soy sauce" does not exist in the y-th image, the prediction layer cannot predict the relevant location information. In this case, the system can remind the user that the product "soy sauce" does not exist in the current image (the y-th image).

[0083] The object detection scheme provided in the embodiments of this specification introduces the Transformer framework, utilizing the powerful self-attention mechanism of the Transformer to improve feature generalization ability and robustness of object detection, thereby improving object detection accuracy. Furthermore, to adapt to the multi-scale object detection problem under the camera's perspective in small shops, the examples in this specification construct a multi-feature fusion network structure based on the Transformer framework, thereby achieving multi-resolution feature fusion of the network. This improves the robustness of object detection for multi-scale objects and significantly enhances the detection capability for multi-scale objects, better enabling the object detection needs under the camera's perspective in small shop scenarios, and ultimately achieving the detection requirements for multi-scale objects under the camera's perspective in small shops.

[0084] It should be noted that the above figures are merely illustrative of the processes included in the methods according to exemplary embodiments of this specification, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may, for example, be executed synchronously or asynchronously in multiple modules.

[0085] The following are embodiments of the apparatus described in this specification, which can be used to execute the embodiments of the methods described in this specification. For details not disclosed in the apparatus embodiments of this specification, please refer to the embodiments of the methods described in this specification.

[0086] in, Figure 5 A schematic diagram of a target detection device applicable to one embodiment of this specification is shown. Please refer to... Figure 5 The target detection device shown in the figure can be implemented as all or part of an electronic device through software, hardware, or a combination of both. It can also be integrated as an independent module on a server or as an independent module in an electronic device.

[0087] The target detection device 500 described in the embodiments of this specification includes: a preprocessing module 510, a target detection model 520, and a prediction module 530.

[0088] The preprocessing module 510 is used to perform feature processing on the image to be detected to obtain a test feature group containing N features, where N is a positive integer. The encoder 5202 of the target detection model 520 is used to: after inputting the N features in the test feature group into the encoder, enhance the N features in the test feature group through the self-attention mechanism of the encoder to obtain N enhanced features to be tested. The decoder 5204 of the target detection model 520 is used to: after inputting the target enhanced features about the target category object and the N enhanced features to be tested into the decoder of the target detection model, determine the features about the target category object in the N enhanced features to be tested. The prediction module 530 is used to: determine the position information of the target category object in the image to be detected based on the features about the target category object in the N enhanced features to be tested.

[0089] In an exemplary embodiment, Figure 6 A schematic diagram illustrating the structure of a target detection device according to another exemplary embodiment of this specification is provided. See also... Figure 6 :

[0090] In an exemplary embodiment, based on the foregoing scheme, the preprocessing module 510 includes a feature extraction unit 5102 and an address encoding unit 5104.

[0091] The feature extraction unit 5102 is used to: perform feature extraction processing on the image to be detected to obtain N depth features; the address encoding unit 5104 is used to: perform position encoding on the N depth features of the image to be detected to obtain the feature group to be tested.

[0092] The i-th feature in the above-mentioned feature group to be tested includes: the i-th depth feature and the position information of the i-th depth feature in the above-mentioned image to be detected, where i takes the value from 1 to N and is an integer including 1 and N.

[0093] In an exemplary embodiment, based on the foregoing scheme, the encoder 5202 includes a feature pyramid structure and / or a densely connected neural network structure; the encoder 5202 is specifically used to: perform feature fusion processing on N features in the feature group to be tested based on the feature pyramid structure and / or the densely connected neural network structure, and enhance the N features in the feature group to be tested through the self-attention mechanism of the encoder to obtain N enhanced features to be tested.

[0094] In an exemplary embodiment, based on the foregoing scheme, the target detection device 500 includes: a target enhancement feature generation module 540.

[0095] The target enhancement feature generation module 540 includes a feature preprocessing unit 5402 and a feature enhancement unit 5404.

[0096] The feature preprocessing unit 5402 is configured to: perform feature processing on the target image containing the target category object before inputting the target enhancement features and the N enhancement features to be tested into the decoder of the target detection model to obtain a target feature group; the feature enhancement unit 5404 is configured to: input the target feature group into the encoder of the target detection model, and enhance the features in the target feature group through the self-attention mechanism of the encoder to obtain the target enhancement features related to the target category object.

[0097] In an exemplary embodiment, based on the aforementioned scheme, the feature preprocessing unit 5402 is specifically configured to: perform feature extraction processing on the target image to obtain M depth features and M positive integers related to the target category object; and perform position encoding on the M depth features of the target image to obtain the target feature group; wherein the j-th feature in the target feature group includes: the j-th depth feature and the position information of the j-th depth feature in the image to be detected, where j takes the value from 1 to M and is an integer including 1 and M.

[0098] In an exemplary embodiment, based on the foregoing scheme, the feature enhancement unit 5404 is specifically used to: input the target feature group to the encoder of the target detection model, perform feature fusion processing on the features in the target feature group based on the feature pyramid structure and / or densely connected neural network structure in the encoder, and enhance the features in the target feature group through the self-attention mechanism of the encoder to obtain target enhanced features for the target category object.

[0099] In an exemplary embodiment, based on the aforementioned scheme, the prediction module 530 is specifically used to: input the features of the target category object from the N enhancement features to be tested into the prediction layer; and obtain the position information of the target category object in the image to be detected through the regression processing of the prediction layer.

[0100] In an exemplary embodiment, based on the foregoing scheme, the target detection model adopts the transformer model.

[0101] It should be noted that the target detection device provided in the above embodiments is only illustrated by the division of the above functional modules when performing the target detection method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0102] Furthermore, the target detection device and target detection method embodiments provided in the above embodiments belong to the same concept. Therefore, for details not disclosed in the device embodiments of this specification, please refer to the target detection method embodiments of this specification above, which will not be repeated here.

[0103] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0104] This specification also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0105] Figure 7 This schematic diagram illustrates the structure of an electronic device according to an exemplary embodiment of this specification. Please refer to... Figure 7 As shown, the electronic device 700 includes a processor 701 and a memory 702.

[0106] In this embodiment, processor 701 is the control center of the computer system and can be a processor of a physical machine or a processor of a virtual machine. Processor 701 may include one or more processing cores, such as a 4-core processor or an 8-core processor. Processor 701 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 701 may also include a main processor and a coprocessor; the main processor is used to process data in the wake-up state, and the coprocessor is a low-power processor used to process data in the standby state.

[0107] In the embodiments described in this specification, the processor 701 is specifically used for:

[0108] Feature processing is performed on the image to be detected to obtain a test feature group containing N features, where N is a positive integer. The N features in the test feature group are input into the encoder of the target detection model, and the N features in the test feature group are enhanced through the self-attention mechanism of the encoder to obtain N enhanced features to be tested. The target enhanced features related to the target category object and the N enhanced features to be tested are input into the decoder of the target detection model to determine the features related to the target category object in the N enhanced features to be tested. And, based on the features related to the target category object in the N enhanced features to be tested, the position information of the target category object in the image to be detected is determined.

[0109] Furthermore, the above-mentioned feature processing of the image to be detected to obtain a test feature group containing N features includes: performing feature extraction processing on the image to be detected to obtain N depth features; and performing position encoding on the N depth features of the image to be detected to obtain the test feature group; wherein, the i-th feature in the test feature group includes: the i-th depth feature and the position information of the i-th depth feature in the image to be detected, where i takes the value from 1 to N and is an integer containing 1 and N.

[0110] Furthermore, the encoder described above includes a feature pyramid structure and / or a densely connected neural network structure;

[0111] The N features in the above-mentioned feature group to be tested are enhanced by the self-attention mechanism of the encoder to obtain N enhanced features to be tested, including: performing feature fusion processing on the N features in the above-mentioned feature pyramid structure and / or the above-mentioned densely connected neural network structure, and enhancing the N features in the above-mentioned feature group to be tested by the self-attention mechanism of the encoder to obtain N enhanced features to be tested.

[0112] Furthermore, the aforementioned processor 701 is also specifically used for:

[0113] Before inputting the target enhancement features of the target category object and the N enhancement features to be tested into the decoder of the target detection model, feature processing is performed on the target image containing the target category object to obtain a target feature group; and the target feature group is input into the encoder of the target detection model, and the features in the target feature group are enhanced by the self-attention mechanism of the encoder to obtain the target enhancement features of the target category object.

[0114] Furthermore, the above-mentioned feature processing of the target image containing the target category object to obtain the target feature group includes: performing feature extraction processing on the target image to obtain M depth features and M positive integers related to the target category object; and performing position encoding on the M depth features of the target image to obtain the target feature group; wherein the j-th feature in the target feature group includes: the j-th depth feature and the position information of the j-th depth feature in the image to be detected, where j takes the value from 1 to M and is an integer containing 1 and M.

[0115] Furthermore, the above-mentioned enhancement processing of the features in the target feature group through the self-attention mechanism of the encoder to obtain target enhanced features for the target category object includes: inputting the target feature group into the encoder of the target detection model, performing feature fusion processing on the features in the target feature group based on the feature pyramid structure and / or densely connected neural network structure in the encoder, and enhancing the features in the target feature group through the self-attention mechanism of the encoder to obtain target enhanced features for the target category object.

[0116] Furthermore, the above-mentioned prediction of the location information of the target category object in the image to be detected based on the features of the target category object among the N enhancement features to be tested includes: inputting the features of the target category object among the N enhancement features to be tested into the prediction layer; and obtaining the location information of the target category object in the image to be detected through the regression processing of the prediction layer.

[0117] Furthermore, the above target detection model adopts the transformer model.

[0118] Memory 702 may include one or more computer-readable storage media, which may be non-transitory. Memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments of this specification, the non-transitory computer-readable storage media in memory 702 is used to store at least one instruction for execution by processor 701 to implement the methods in the embodiments of this specification.

[0119] In some embodiments, the electronic device 700 further includes a peripheral device interface 703 and at least one peripheral device. The processor 701, memory 702, and peripheral device interface 703 are connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 703 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of a display screen 704, a camera 705, and an audio circuit 706.

[0120] Peripheral device interface 703 can be used to connect at least one input / output (I / O) related peripheral device to processor 701 and memory 702. In some embodiments of this specification, processor 701, memory 702, and peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments of this specification, any one or two of processor 701, memory 702, and peripheral device interface 703 can be implemented on separate chips or circuit boards. This specification does not specifically limit the embodiments in this regard.

[0121] Display screen 704 is used to display a user interface (UI). The UI may include graphics, text, icons, video, and any combination thereof. When display screen 704 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 701 for processing. In this case, display screen 704 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments of this specification, there may be one display screen 704, which is disposed on the front panel of electronic device 700; in other embodiments, there may be at least two display screens 704, respectively disposed on different surfaces of electronic device 700 or in a folded design; in still other embodiments, display screen 704 may be a flexible display screen, disposed on a curved or folded surface of electronic device 700. Furthermore, display screen 704 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 704 may be made of materials such as Liquid Crystal Display (LCD) or Organic Light-Emitting Diode (OLED).

[0122] Camera 705 is used to capture images or videos. Optionally, camera 705 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the electronic device, and the rear-facing camera is located on the back of the electronic device. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, virtual reality (VR) shooting, or other fusion shooting functions. In some embodiments of this specification, camera 705 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0123] The audio circuit 706 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 701 for processing. For stereo sound acquisition or noise reduction purposes, there may be multiple microphones, each located in a different part of the electronic device 700. The microphone may also be an array microphone or an omnidirectional microphone.

[0124] Power supply 707 is used to supply power to various components in electronic device 700. Power supply 707 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 707 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0125] The block diagrams of the electronic device shown in the embodiments of this specification do not constitute a limitation on the electronic device 700. The electronic device 700 may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0126] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of these terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0127] This specification also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. If the constituent modules of the above-described target detection device are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium.

[0128] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0129] It should be noted that the above description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims may be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0130] The above description is merely a specific embodiment of this specification, but the scope of protection of this specification is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this specification should be included within the scope of protection of this specification. Therefore, equivalent variations made in accordance with the claims of this specification are still within the scope of this specification.

Claims

1. A target detection method, wherein, The method includes: The image to be detected is subjected to feature processing to obtain a set of features to be detected containing N features, where N is a positive integer; The N features in the feature group to be tested are input into the encoder of the target detection model, and the N features in the feature group to be tested are enhanced through the self-attention mechanism of the encoder to obtain N enhanced features to be tested. The target augmentation features related to the target category object and the N augmentation features to be tested are input into the decoder of the target detection model to determine the features related to the target category object among the N augmentation features to be tested; Based on the features of the target category object among the N enhancement features to be tested, the location information of the target category object in the image to be detected is determined; Before inputting the target augmentation features and the N augmentation features to be tested into the decoder of the target detection model, the method further includes: Perform feature processing on the target image containing the target category object to obtain the target feature group; The target feature set is input into the encoder of the target detection model, and the features in the target feature set are enhanced through the self-attention mechanism of the encoder to obtain the target enhanced features for the target category object; The step of performing feature processing on the image to be detected to obtain a test feature group containing N features includes: performing feature extraction processing on the image to be detected to obtain N depth features; performing position encoding on the N depth features of the image to be detected to obtain the test feature group; wherein, the i-th feature in the test feature group includes: the i-th depth feature and the position information of the i-th depth feature in the image to be detected, where i takes the value from 1 to N and is an integer containing 1 and N; The encoder includes a feature pyramid structure and a densely connected neural network structure; the step of enhancing N features in the feature group to be tested through the self-attention mechanism of the encoder to obtain N enhanced features to be tested includes: performing feature fusion processing on the N features in the feature group to be tested based on the feature pyramid structure and the densely connected neural network structure, and enhancing the N features in the feature group to be tested through the self-attention mechanism of the encoder to obtain N enhanced features to be tested.

2. The target detection method according to claim 1, wherein, The step of performing feature processing on the target image containing the target category object to obtain the target feature group includes: The target image is subjected to feature extraction processing to obtain M depth features related to the target category object, where M is a positive integer; The M depth features of the target image are respectively encoded to obtain the target feature group; wherein, the j-th feature in the target feature group includes: the j-th depth feature and the position information of the j-th depth feature in the target image, and j takes the value from 1 to M and is an integer containing 1 and M.

3. The target detection method according to claim 1, wherein, The step of enhancing the features in the target feature group through the self-attention mechanism of the encoder to obtain target enhanced features for the target category object includes: The target feature group is input into the encoder of the target detection model. Based on the feature pyramid structure and densely connected neural network structure in the encoder, the features in the target feature group are fused. The features in the target feature group are then enhanced through the self-attention mechanism of the encoder to obtain the target enhanced features for the target category object.

4. The target detection method according to any one of claims 1 to 3, wherein, The step of determining the location information of the target category object in the image to be detected based on the features of the target category object among the N enhancement features to be tested includes: The features related to the target category object from the N enhancement features to be tested are input into the prediction layer; The location information of the target category object in the image to be detected is obtained through regression processing of the prediction layer.

5. The target detection method according to any one of claims 1 to 3, wherein, The target detection model uses the transformer model.

6. A target detection device, wherein, The device includes: a preprocessing module, a target detection model, a prediction module, and a target enhancement feature generation module; the target enhancement feature generation module includes: a feature preprocessing unit and a feature enhancement unit; The preprocessing module is used to perform feature processing on the image to be detected to obtain a test feature group containing N features, where N is a positive integer; The encoder of the target detection model is used to: after inputting N features from the feature group to be tested into the encoder, enhance the N features from the feature group to be tested through the self-attention mechanism of the encoder to obtain N enhanced features to be tested. The decoder of the target detection model is used to: after inputting the target enhancement features about the target category object and the N enhancement features to be tested into the decoder of the target detection model, determine the features about the target category object among the N enhancement features to be tested; The prediction module is used to: determine the location information of the target category object in the image to be detected based on the features of the target category object among the N enhancement features to be tested; The feature preprocessing unit is configured to: perform feature processing on the target image containing the target category object before inputting the target enhancement features and the N enhancement features to be tested into the decoder of the target detection model to obtain a target feature group; The feature enhancement unit is used to: input the target feature group into the encoder of the target detection model, and enhance the features in the target feature group through the self-attention mechanism of the encoder to obtain target enhanced features about the target category object; The preprocessing module is used to perform feature extraction processing on the image to be detected to obtain N depth features; and to perform position encoding on the N depth features of the image to be detected to obtain the feature group to be tested; wherein, the i-th feature in the feature group to be tested includes: the i-th depth feature and the position information of the i-th depth feature in the image to be detected, where i takes the value from 1 to N and is an integer containing 1 and N; The encoder includes a feature pyramid structure and a densely connected neural network structure; the step of enhancing N features in the feature group to be tested through the self-attention mechanism of the encoder to obtain N enhanced features to be tested includes: performing feature fusion processing on the N features in the feature group to be tested based on the feature pyramid structure and the densely connected neural network structure, and enhancing the N features in the feature group to be tested through the self-attention mechanism of the encoder to obtain N enhanced features to be tested.

7. A computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the target detection method as described in any one of claims 1 to 5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the target detection method as described in any one of claims 1 to 5.

9. A computer program product comprising instructions that, when run on a computer or processor, causes the computer or processor to perform the target detection method as described in any one of claims 1 to 5.

Images