Target detection method and apparatus

Abstract

A target detection method and apparatus. The method comprises: acquiring input data; extracting a feature from the input data to obtain a first feature, for example, using a feature extraction network to extract the feature from the input data; inputting the first feature into a decoder to obtain a plurality of groups of features, the decoder being used for decoding the input data at least once, and any single decoding process comprising: on the basis of an attention mechanism, decoding an inputted feature, and outputting a plurality of groups of features obtained by a current iteration, an input query of the attention mechanism comprising a plurality of parameters, and the plurality of parameters comprising parameters related to the morphology of a detectable instance of the decoder, wherein the parameters can be used for determining targets of which morphologies can be detected by the decoder; and, on the basis of the plurality of groups of features, determining at least one instance and corresponding attribute information.

Description

A target detection method and apparatus

[0001] This application claims priority to Chinese Patent Application No. 202411055255.8, filed on July 31, 2024, entitled "A Target Detection Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer vision, and more particularly to a method and apparatus for object detection. Background Technology

[0003] With the development of deep learning (DL) technology, target detection algorithms based on neural networks (NN) have been extensively and deeply studied, and their applications span various industries. Taking autonomous driving as an example, autonomous vehicles need to perceive various targets in the surrounding environment to calculate their future interactions with these targets, thereby planning subsequent driving routes. Among these, target detection, as the core function of the autonomous driving perception system, plays a crucial role in the subsequent behavior of the autonomous vehicle due to its high accuracy.

[0004] In existing target detection scenarios, the variety of target types, the significant differences in morphological features among different object types, the vast variations in different instances of the same object, the drastically different imaging effects of different objects, and the mutual occlusion between objects all contribute to the difficulty in guaranteeing target detection results. Therefore, how to accurately detect objects of various shapes has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a target detection method and apparatus, which are used to set different model parameters based on different target shapes, thereby adapting to target detection scenarios of different shapes and having a very strong target detection capability.

[0006] In a first aspect, this application provides a target detection method, comprising: first, acquiring input data, which may include data collected by one or more sensors or data read from storage space, such as image sensors or LiDAR sensors that can be used to collect environmental data; then, extracting features from the input data to obtain a first feature, for example, using a feature extraction network to extract features from the input data; then, inputting the first feature into a decoder to obtain multiple sets of features, the decoder being used to decode the input data at least once, any one of the decoding processes in the at least one decoding includes: decoding the input features based on an attention mechanism, outputting multiple sets of features obtained in the current iteration, the input query of the attention mechanism including multiple parameters, including parameters related to the morphology of instances that the decoder can detect, the values ​​of the multiple parameters being used to determine which forms of targets the decoder can detect; then, determining at least one instance and the attribute information corresponding to each instance based on the multiple sets of features.

[0007] In this embodiment, the decoder can be configured with various parameters related to the shape of the detectable visual field. Therefore, by setting the parameters of the decoder, the decoder can be made to have the ability to detect targets of the desired shape, thereby adapting to target detection scenarios of different shapes and having a very strong generalization ability.

[0008] In one possible implementation, the aforementioned parameters may include, but are not limited to, at least one of the following: query instance count K, geometric content group count N, attribute group count C, feature width D, etc. The query instance count is used to determine the number of instances in the input data, the geometric content group count is used to determine the number of partitions of instances in the input data, and the attribute group count is used to determine the number of attribute groups of instances in the input data. K, N, C, and D are positive integers.

[0009] In this embodiment, parameters can be set based on dimensions such as the number of query instances, geometric content grouping, attribute grouping, or feature length. For example, the query, key, and value in the attention mechanism can be set, i.e., QKV in the attention mechanism can be set from dimensions such as the number, geometric content grouping, or attribute grouping. Therefore, when extracting instance-related feature values, instances can be grouped according to the set number of geometric dimensions or attribute dimensions. By grouping the target by geometric dimensions, one or more groups can be formed to achieve segmented processing for targets with different shapes. That is, the target is divided into one or more parts, and feature extraction or detection of the target is performed at the segmentation granularity. Alternatively, the target can be grouped by attribute dimensions to form one or more groups of attributes, thereby dividing the target from the attribute dimension to perform targeted feature extraction for each group of attributes. This improves the detection accuracy for each group of attributes without increasing computational complexity.

[0010] In one possible implementation, the aforementioned determination of at least one instance and the attribute information corresponding to each instance based on multiple sets of features may include: extracting features from multiple sets of features output from the previous iteration according to multiple parameters to obtain multiple sets of features, wherein the multiple sets of features include K sets of N*C*D features. In this embodiment of the application, during the feature extraction process of the decoder, feature extraction can be performed according to the set K, N, C, and D, thereby extracting the K sets of N*C*D features required for target detection.

[0011] In one possible implementation, the aforementioned decoder includes at least one of a self-attention module, a cross-attention module, or a detection module. The self-attention module is used to extract features from each of the K sets of N*C*D features in the input based on an attention mechanism. The cross-attention module is used to extract features from the K sets of N*C*D features in the input based on a cross-attention mechanism. The detection module is used to extract features from the input features. In this embodiment, when deploying the decoder structure, a self-attention module and a cross-attention module can be deployed. This allows the self-attention mechanism to focus on features within each group, and the cross-attention mechanism to focus on the correlation between segments of the target, thereby making the finally extracted features more accurate in representing the target.

[0012] In one possible implementation, when there are multiple geometric content groups, the geometric content grouping is used to indicate that an instance is divided into multiple parts; when there is only one geometric content group, the geometric content grouping is used to indicate that an instance is treated as a group. Therefore, N can be set according to the specific shape of the instance, that is, the number of segments for the instance can be set according to the specific shape of the instance. For example, if the length-to-width ratio of the instance is large, such as greater than a threshold, N can be set to greater than 1; if the length-to-width ratio of the instance is not large, N can be set to 1, etc. For example, if the instance is a circular or square entity, it can be set to 1; if the instance is a relatively long and narrow object such as a lane or curb, N can be set to > 1. In this way, the instance is divided into multiple segments, and feature extraction is performed on the instance at the segment granularity. This achieves target detection for relatively long and narrow entities without increasing the computational complexity of the model.

[0013] In one possible implementation, any one of the aforementioned decoding processes in at least one decoding step may further include: determining a mask corresponding to each instance in at least one instance based on multiple sets of features output by the decoder, and using the mask corresponding to the at least one instance as input for the next decoding step to determine the position of at least one instance in the next iteration. In this embodiment, during the iterative decoding process, the masks of each instance can also be output, so that the next iteration focuses more on the region of the target represented by the mask, and outputs a more accurate target mask during continuous iteration, thereby enabling the finally extracted features to more accurately represent the target. Furthermore, in target segmentation scenarios, different segments belonging to the same target can be grouped into the same target, which is more conducive to extracting the correlation between target segments and improving the overall detection accuracy of the target.

[0014] In one possible implementation, the aforementioned determination of the mask corresponding to at least one instance based on multiple sets of features output by the decoder may include: obtaining the similarity between multiple sets of features to obtain a similarity map, where each element in the similarity map represents the similarity between feature values ​​corresponding to the same region between different feature groups; and generating the mask corresponding to at least one instance based on the values ​​of the elements in the similarity map.

[0015] In this embodiment, masks corresponding to each instance can be generated based on the similarity between multiple features. That is, based on features, it is possible to identify which regions belong to the same target, and in the next iteration, the mask corresponding to the target is used as the shape constraint of the target, so that the extracted target features can more accurately cover the region corresponding to the target. Therefore, with the iteration of the decoding process, more accurate target features can be obtained.

[0016] In one possible implementation, the aforementioned determination of at least one instance and the attribute information corresponding to each instance based on multiple sets of features includes: inputting multiple sets of features into at least one perceptual network to obtain at least one instance and the attribute information corresponding to each instance. In this application embodiment, a perceptual network can be used to identify instances and their corresponding attribute information based on the extracted multiple sets of features, such as the instance's shape, color, location, size, or meaning in the actual environment.

[0017] In one possible implementation, the aforementioned method may further include: filtering at least one instance and the attribute information corresponding to each instance, such as filtering instances with low confidence or instances irrelevant to downstream tasks, and outputting the filtered at least one instance and the attribute information corresponding to each instance. In this embodiment, the detected targets may be further filtered to output more relevant instances.

[0018] Secondly, this application provides a target detection device, comprising:

[0019] The acquisition module is used to acquire input data;

[0020] The feature extraction module is used to extract features from the input data to obtain the first feature;

[0021] The decoder module is used to input the first feature into the decoder to obtain multiple sets of features. The decoder is used to decode the input data at least once. Any decoding process in the at least one decoding includes: decoding the input features based on the attention mechanism and outputting multiple sets of features obtained in the current iteration. The input query of the attention mechanism includes multiple parameters, including parameters related to the morphology of the instances that the decoder can detect.

[0022] The detection module is used to determine at least one instance and the attribute information corresponding to each instance based on multiple sets of features.

[0023] The effects achieved by the second aspect or any optional implementation of the second aspect of this application can be referred to the description of the first aspect or any optional implementation of the first aspect above, and will not be repeated hereafter.

[0024] In one possible implementation, multiple parameters include at least one of the following: query instance count K, geometric content group count N, attribute group count C, and feature width D. The query instance count is used to determine the number of instances in the input data, the geometric content group count is used to determine the number of partitions of instances in the input data, and the attribute group count is used to determine the number of attribute groups of instances in the input data. K, N, C, and D are positive integers.

[0025] In one possible implementation, the detection module is specifically used to: extract features from multiple sets of features output from the previous iteration according to multiple parameters, to obtain multiple sets of features, which include K sets of N*C*D features.

[0026] In one possible implementation, the aforementioned decoder includes at least one of a self-attention module, a cross-attention module, or a detection module. The self-attention module is used to extract features from each of the K sets of N*C*D features in the input based on an attention mechanism. The cross-attention module is used to extract features from the K sets of N*C*D features in the input based on a cross-attention mechanism. The detection module is used to extract features from the input features.

[0027] In one possible implementation, when there are multiple geometric content groups, the geometric content grouping is used to indicate that an instance is divided into multiple parts; when there is only one geometric content group, the geometric content grouping is used to indicate that an instance is treated as a group.

[0028] In one possible implementation, any one of the decoding processes in at least one decoding operation further includes: determining a mask corresponding to at least one instance based on multiple sets of features output by the decoder, wherein the mask corresponding to at least one instance is used as input for the next decoding process to determine the position of at least one instance.

[0029] In one possible implementation, the decoder module is specifically configured to: obtain the similarity between multiple sets of features to obtain a similarity map; and generate a mask corresponding to at least one instance based on the values ​​of the elements in the similarity map.

[0030] In one possible implementation, the detection module is specifically used to: input multiple sets of features into at least one perceptual network to obtain at least one instance and attribute information corresponding to each instance.

[0031] In one possible implementation, the apparatus further includes a filtering module for filtering at least one instance and attribute information corresponding to each instance, and outputting the filtered at least one instance and attribute information corresponding to each instance.

[0032] Thirdly, embodiments of this application provide a target detection device, including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to execute processing-related functions in the target detection method shown in any of the first aspects above. Optionally, the target detection device may be a chip.

[0033] Fourthly, embodiments of this application provide an intelligent driving vehicle, which includes a processor and a memory, wherein the processor and the memory are interconnected by a circuit, and the processor calls program code in the memory to execute processing-related functions in the target detection method shown in any of the first aspects above.

[0034] Fifthly, embodiments of this application provide a digital processing chip or chip, the chip including a processing unit and a communication interface, the processing unit obtaining program instructions through the communication interface, the program instructions being executed by the processing unit, the processing unit being used to perform processing-related functions as described in the first aspect or any optional embodiment of the first aspect.

[0035] In a sixth aspect, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any optional implementation thereof.

[0036] In a seventh aspect, embodiments of this application provide a computer program product comprising a computer program / instructions, which, when executed by a processor, causes the processor to perform the method described in the first aspect or any optional implementation thereof. Attached Figure Description

[0037] Figure 1 is a schematic diagram of the architecture of an existing attention mechanism;

[0038] Figure 2 is a schematic diagram of the architecture of a multi-head attention mechanism in the prior art;

[0039] Figure 3 is a schematic diagram of an application scenario provided by an embodiment of this application;

[0040] Figure 4 is a schematic diagram of another application scenario provided by an embodiment of this application;

[0041] Figure 5 is a schematic diagram of another application scenario provided by an embodiment of this application;

[0042] Figure 6 is a schematic diagram of another application scenario provided by an embodiment of this application;

[0043] Figure 7 is a schematic diagram of another application scenario provided by an embodiment of this application;

[0044] Figure 8 is a schematic diagram of another application scenario provided by an embodiment of this application;

[0045] Figure 9 is a structural schematic diagram of a vehicle provided in an embodiment of this application;

[0046] Figure 10 is a flowchart illustrating a target detection method provided in an embodiment of this application;

[0047] Figure 11 is a flowchart illustrating another target detection method provided in an embodiment of this application;

[0048] Figure 12 is a schematic diagram of the structure of a decoder provided in an embodiment of this application;

[0049] Figure 13 is a schematic diagram of another decoder provided in an embodiment of this application;

[0050] Figure 14 is a flowchart illustrating another target detection method provided in an embodiment of this application;

[0051] Figure 15 is a flowchart illustrating another target detection method provided in an embodiment of this application;

[0052] Figure 16 is a flowchart illustrating another target detection method provided in an embodiment of this application;

[0053] Figure 17 is a schematic diagram of another decoder provided in an embodiment of this application;

[0054] Figure 18 is a flowchart illustrating another target detection method provided in an embodiment of this application;

[0055] Figure 19 is a flowchart illustrating another target detection method provided in an embodiment of this application;

[0056] Figure 20 is a schematic diagram of another decoder provided in an embodiment of this application;

[0057] Figure 21 is a schematic diagram of a detection effect provided in an embodiment of this application;

[0058] Figure 22 is a schematic diagram of another detection effect provided by an embodiment of this application;

[0059] Figure 23 is a schematic diagram of a target detection device provided in an embodiment of this application;

[0060] Figure 24 is a schematic diagram of another target detection device provided in an embodiment of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0062] To better understand the solutions of the embodiments of this application, the relevant terms and concepts that may be involved in the embodiments of this application will be introduced below.

[0063] (1) Deep Neural Networks

[0064] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple intermediate layers. Based on the position of these layers, the internal neural network of a DNN can be divided into three categories: input layer, intermediate layers, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are considered intermediate layers, or hidden layers.

[0065] Although DNNs appear complex, each layer can be represented as a linear relational expression: in, It is the input vector. It is the output vector. is the offset vector, also known as the bias parameter; w is the weight matrix (also called coefficients); and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also quite large. The definitions of these parameters in DNNs are as follows: Taking the coefficient w as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0066] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0067] It's important to note that the input layer does not have a W parameter. In deep neural networks, more intermediate layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0068] (2) Convolutional Neural Network

[0069] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is that it reduces the connections between layers in the CNN, while also reducing the risk of overfitting.

[0070] (3) Multilayer Perceptron (MLP)

[0071] MLP (Multi-Level Processing) neural networks are a type of feedforward neural network (FFN). It is a fully connected (meaning each neuron is connected to all neurons in the previous layer) feedforward neural network model. For example, in classification problems, it transforms the raw scores for each class through multiple fully connected layers, and then uses a softmax function to obtain the predicted probability for each class. An MLP can be viewed as a directed graph composed of multiple node layers, each fully connected to the next. Besides the input node, each node is a neuron (or processing unit) with a non-linear activation function.

[0072] (4) transformer

[0073] A transformer architecture is a feature extraction network that includes both an encoder and a decoder. Of course, in some cases, a transformer architecture may not include an encoder but may include a decoder.

[0074] Encoder: Learns features, such as pixel features, within the global receptive field using self-attention.

[0075] Decoder: Learns the features of the desired module, such as the features of the output box, through self-attention and cross-attention.

[0076] For example, the structure of a prior art Transformer layer can be as shown in Figure 1. The Transformer layer includes a multi-head attention network and a feedforward network module. Taking natural language processing as an example, the multi-head attention network obtains corresponding weight values ​​by calculating the correlation between words, thus obtaining context-related word representations, which is the core part of the Transformer structure. The feedforward network further transforms the obtained representations to obtain the final output of the Transformer layer. In addition to these two important components, a residual layer (ADD) and linear normalization (Norm) are also stacked on these two components to optimize the output of the Transformer layer.

[0077] (5) Attention mechanism

[0078] Attention mechanisms can quickly extract important features from sparse data. They provide an effective modeling approach for capturing global contextual information through QKV (Queries, Keys, Values). Assuming the input is Q(query), and the context is stored as key-value pairs (K, V), then the attention mechanism is essentially a mapping function from the query to a series of key-value pairs. The essence of the attention function can be described as a mapping from a query to a series of (key-value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in (K, V) form, then attention performs addressing by calculating the similarity between Q and K. The similarity calculated between Q and K reflects the importance of the extracted V values, i.e., the weights, and then a weighted sum is obtained to obtain the final feature value.

[0079] Attention calculation mainly consists of three steps. The first step is to calculate the similarity between the query and each key to obtain weights. Common similarity functions include dot product, concatenation, and perceptron. The second step typically uses a softmax function (which can normalize the weights, resulting in a probability distribution where the sum of all weight coefficients is 1, and also highlights the weights of important elements) to normalize these weights. Finally, the weights and their corresponding key values ​​are weighted and summed to obtain the final feature value. The specific calculation formula is as follows:

[0080] Where d is the dimension of matrix Q,K.

[0081] Furthermore, attention includes self-attention and cross-attention. Self-attention can be understood as a special type of attention where the inputs to the QKV features are consistent. Cross-attention, on the other hand, involves inconsistent inputs to the QKV features. Attention integrates the queried features as updated values ​​for the current features using the similarity between features (e.g., inner product) as weights. Self-attention is attention extracted based on the attention drawn from the feature map itself.

[0082] For convolutional networks, the kernel size limits the receptive field, often requiring multiple layers to focus on the entire feature map. Self-attention, on the other hand, has the advantage of global focus; it can obtain global spatial information about the feature map through simple lookups and assignments.

[0083] (6) Multi-head attention (MHA)

[0084] Given the same set of queries, keys, and values, the desired model is to learn different behaviors based on the same attention mechanism, and then combine these different behaviors as knowledge, such as capturing dependencies of various ranges within a sequence (e.g., short-range and long-range dependencies). Therefore, the attention mechanism combines different representation subspaces of queries, keys, and values. Compared to self-attention models, multi-head attention models increase the number of heads. For example, as shown in Figure 2, Q, K, and V first pass through a linear layer, then are input into scaled dot-product attention, repeated h times (i.e., multi-head), where each time one head is calculated, and parameters are not shared between heads. The parameters W for the linear transformation of Q, K, and V are different each time. Then, the results of the h-times scaled dot-product attention are concatenated, and a final linear transformation is performed to obtain the result of the multi-head attention.

[0085] (7) Mask

[0086] A mask can be understood as a label for data, also called a code. By fusing data with a corresponding mask, it allows for greater focus on specific parts of the data. Typically, masks are used to extract regions of interest (ROIs). For example, a pre-made ROI mask is fused with the image to be processed to obtain an ROI image, where image values ​​within the ROI remain unchanged, while values ​​outside the ROI are all 0. Masks can also serve a shielding function, such as using a mask to shield certain areas of an image, preventing them from participating in processing or parameter calculations, or processing or statistically analyzing only the unmasked areas.

[0087] (8) Attention map

[0088] A latent feature in MHA. A matrix that measures the similarity between a query and its key.

[0089] (9) Attention mask

[0090] One of the input parameters in MHA. The similarity of the regions marked by the attention mask will be set to 0.

[0091] The method provided in this application can be applied to various target detection scenarios. Target detection is widely used in scenarios such as intelligent security, industrial inspection, and autonomous driving. Taking autonomous driving as an example, autonomous vehicles need to perceive various targets in the surrounding environment to calculate their future interaction with these targets, thereby planning subsequent driving routes. Among these, the target detection function, as the core function of the autonomous driving perception system, plays a crucial role in the subsequent behavior of the autonomous vehicle due to its high accuracy.

[0092] Object detection solutions have evolved in response to the growing demand for object detection, progressing from image-based input to multi-sensor fusion input, from traditional image processing algorithms to deep learning algorithms, and from basic convolutional neural networks (CNNs) to the Transformer architecture. This gradual advancement has elevated object detection capabilities to increasingly intelligent levels, enabling applications in a wider range of scenarios. Object detection remains a highly challenging problem. Taking image input as an example, the sheer variety of target types, the vast differences in morphological features between different object types, the significant variations between different instances of the same object, the drastically different imaging effects of different objects, and the occlusion between objects all contribute to the difficulty in guaranteeing accurate object detection results.

[0093] To achieve good object detection results, different detection algorithms are typically deployed to suit different detection targets. For example, a single-query detection algorithm can be used for general object detection. This involves using multiple trained latent features to interact with sensor features (e.g., obtained through a CNN on an image) to extract features from multiple targets, which are then used to obtain information such as the target's location. This approach is simple and efficient, and performs well for general objects such as people, vehicles, and animals. However, for special objects such as lane lines, other methods are needed, such as combined query detection. In this combined query approach, the latent features of each target are decomposed into multiple keypoint features to interact with sensor features. Through multiple rounds of interaction and optimization, the keypoints gradually move to instances throughout the lane line. These keypoints are then combined in an orderly manner to form the lane line detection result. This algorithm is advantageous for detecting objects with long, narrow shapes and large spans, but its computational complexity is high, making its deployment on the edge costly.

[0094] For example, in a specific object detection scheme, in the first stage, a CNN is used to extract image features; a transformer encoder is used to encode the image features, and trained latent features are used to interact with these image features to extract updated latent features of the target. However, this scheme is also difficult to adapt to scenarios with diverse target shapes, especially for objects that are long and narrow with a large span. A single feature is usually insufficient to cover objects with long and narrow shapes and a large span, and the bounding box accuracy is usually low for this type of target detection. Furthermore, it lacks multi-attribute information modeling, making it difficult to extend to the prediction of multi-attribute information, such as detecting a car, which requires outputting wheel orientation, whether the turn signals are on, etc.

[0095] For example, in a specific object detection scheme, in the first stage, a CNN is used to extract image and laser point cloud features, which are then aligned to the bird's-eye view (BEV) space to obtain BEV features. Multiple road keypoint latent features are used as queries to interact with the BEV features and update the keypoint latent features and keypoint positions. Keypoint positions belonging to the same lane instance are combined into a point sequence for output. However, in this scheme, queries are performed on a keypoint-by-keypoint basis, and the output is a point sequence composed of keypoints. Therefore, a lane instance will correspond to multiple keypoints, requiring multiple Transformer Decoder calculations. Lanes with larger spans will require more keypoints, leading to high computational complexity and significant deployment costs on the edge. Considering only the geometric information of predicted lane instances, and lacking consideration of road attribute information (such as type and color), it is difficult to directly apply to real-world products.

[0096] Therefore, in this application, considering the differences in target shape and the problem of multiple target attributes, a target detection method is provided that can adapt to diverse target shapes or multiple attributes, with low computational complexity and low deployment cost.

[0097] The method provided in this application can be applied to scenarios such as intelligent driving, security, or industrial inspection. For example, the method provided in this application can be deployed in the intelligent driving system of a vehicle. By using the method provided in this application to perform environmental perception on the environmental images collected by the vehicle, that is, to detect targets in the environment and output the targets in the environment where the vehicle is located, such as lanes, obstacles, or road signs, the vehicle can further realize the intelligent driving function based on the environmental perception results.

[0098] The specific application scenarios of the methods provided in the embodiments of this application will be described below in conjunction with specific application scenarios.

[0099] Scenario 1: Intelligent Driving Function

[0100] Intelligent driving functions typically include lane departure warning (LDW) or lane keeping assist (LKA) features within advanced driving assistance systems (ADAS). For example, lane departure warning and lane keeping assist require real-time detection of lane markings and determination of whether the vehicle is deviating from its current lane. If the vehicle deviates from its lane without a turn signal being given, a warning signal will be issued, the steering wheel will vibrate, or even the steering wheel will be actively pulled back to remind the driver to return to the lane. Using the method provided in this application, when the distance between the lane markings and the vehicle's position is detected to be gradually decreasing, the vehicle determines that it is deviating from its current lane, and the lane departure warning system will alert the driver by vibrating the steering wheel. As shown in Figure 3, when lane deviation occurs, the lane keeping assist system can control the steering wheel to steer the vehicle back into its current lane.

[0101] Scenario 2: Road Structure Cognition

[0102] A key subsystem in intelligent driving systems is the road structure recognition system. This system receives lane and traffic light information, generates road relationship topology, and guides the vehicle through road intersections. The road structure recognition system is built upon lane information in the surrounding environment; accurate lane information helps generate an accurate intersection relationship network. As shown in Figure 4, the left side shows the original lane centerlines and traffic light information received by the road structure recognition system. Each entrance and exit of this intersection contains multiple lanes, thus providing numerous alternative lane-changing routes. Using the method provided in this embodiment, combined with the traffic light status, the road structure recognition system generates all feasible lane-changing routes, as shown on the right side of Figure 4.

[0103] Scenario 3: Driving Route Planning

[0104] A key subsystem in intelligent driving systems is the path planning system. This system receives environmental and navigation information and outputs the vehicle's future direction. The path planning system requires input of the locations of other road users, lane instances, and their lane markings. Its function is to guide the vehicle to plan the correct path. As shown in Figure 5, the vehicle detects another vehicle obstructing its path and needs to detour around it. The available routes are to the left (dashed line path) and to the right (solid line path). The vehicle detects that the right detour path would cross a solid lane line, and solid lines between lanes represent prohibited crossings; therefore, the right detour is not feasible. The left detour path, however, would cross a dashed lane line, which represents a permissible crossing; therefore, the left detour is feasible.

[0105] Scenario 4: Adaptive Cruise Control (ACC)

[0106] The adaptive cruise control function in ADAS (Adaptive Cruise Control) needs to adaptively adjust the vehicle's speed based on the position and speed of vehicles ahead in the lane, thereby achieving automatic cruise control without collision. As shown in the left image of Figure 6, when there are no other traffic participants (targets) ahead of the vehicle in the lane, the vehicle proceeds at a preset speed or the road speed limit, which is 80 km / h in this image. When the vehicle's perception system detects that other traffic participants have entered the lane ahead using the method provided in this application embodiment, as shown in the right image of Figure 6, the vehicle will automatically reduce its speed to 70 km / h based on the position and speed of the other vehicle to avoid collision with the vehicle in front.

[0107] Scenario 5: Target detection and target path prediction in intelligent driving or security scenarios

[0108] Trajectory prediction uses cameras to perceive road scenes and target detection algorithms to obtain information such as the position, orientation, and size of important traffic participants in the environment. By accumulating multiple frames of detection results, the speed and direction of each target can be obtained, thereby predicting the future trajectory of the target and serving as the basis for subsequent decision-making and control of the intelligent driving vehicle. As shown in the left image of Figure 7, the intelligent driving vehicle predicts the future movement direction of surrounding vehicles. The color of the area indicates that the corresponding vehicle may pose a threat to the driving safety of the vehicle in the future, requiring the intelligent driving system to avoid it. As shown in the right image of Figure 7, in this scenario, by detecting the orientation and position of pedestrians, their future movement is predicted, thereby identifying potential crowd formations in advance.

[0109] Scenario 6: Industrial Product Quality Inspection

[0110] In industrial quality inspection scenarios, manufacturing defects are frequently found in industrial products. As shown in Figure 8, the left image displays a screw thread defect, while the right image shows a crack in a metal product. Defects in industrial products can develop at different angles, penetrating from the surface into the interior, thus posing a risk of damage during use. Industrial products can be addressed according to the type and severity of defects, optimizing resource allocation in terms of cost control and resource utilization.

[0111] In light of the aforementioned application scenarios, the method provided in this application embodiment can be deployed on devices such as terminals or servers.

[0112] In one possible implementation, the method provided in this application can be deployed in a terminal, such as a mobile phone, tablet personal computer (TPC), media player, smart TV, laptop computer (LC), augmented reality (AR) / virtual reality (VR), in-vehicle terminal, intelligent driving vehicle, personal digital assistant (PDA), personal computer (PC), camera, camcorder, smartwatch, wearable device (WD), etc., and this application does not limit it in this regard.

[0113] In one possible implementation, the method provided in this application embodiment can also be deployed on a server, such as a cloud server or a server connected to a terminal. For example, the method provided in this application embodiment can be deployed on a cloud server connected to a terminal. The cloud server can receive images from the terminal, perform target detection using the method provided in this application embodiment, and output one or more targets in the detected image, as well as attribute information corresponding to each target, such as the target's position, size, angle, color, or environment-related attributes.

[0114] For example, taking the method provided in this application embodiment as an example of deployment in a vehicle, the structure of the vehicle can be as shown in Figure 9.

[0115] Referring to Figure 9, which is a schematic diagram of a vehicle structure provided in an embodiment of this application, the vehicle 100 can be configured in an intelligent driving mode. For example, the vehicle 100 can control itself while in intelligent driving mode, and can determine the current state of the vehicle and its surrounding environment through human operation, determine whether there are obstacles in the surrounding environment, and control the vehicle 100 based on the obstacle information. When the vehicle 100 is in intelligent driving mode, it can also be set to operate without human interaction.

[0116] Please refer to Figure 9 first. Figure 9 is a structural schematic diagram of a vehicle provided in an embodiment of this application, and a functional block diagram of a vehicle 100 provided in an embodiment of this application. The vehicle 100 can be configured to a full or partial intelligent driving mode. For example, the vehicle 100 can obtain environmental information about its surroundings through the perception system 120, and obtain an intelligent driving strategy based on the analysis of the surrounding environmental information to achieve full intelligent driving, or present the analysis results to the user to achieve partial intelligent driving.

[0117] Vehicle 100 may include various subsystems, such as an infotainment system 110, a perception system 120, a decision control system 130, a drive system 140, and a computing platform 150. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0118] In some embodiments, the infotainment system 110 may include a communication system 111, an entertainment system 112, and a navigation system 113.

[0119] Communication system 111 may include wireless communication system 111, which can communicate wirelessly with one or more devices directly or via a communication network. For example, wireless communication system 111 may use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. Wireless communication system 111 may communicate using WiFi and wireless local area network (WLAN). In some embodiments, wireless communication system 146 may communicate directly with devices using an infrared link, Bluetooth, or ZigBee. Wireless communication system 111 may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0120] The entertainment system 112 may include a central control screen, a microphone, and speakers. Users can listen to the radio and play music within the vehicle using the entertainment system 112; or connect their mobile phones to the vehicle and project their screens onto the central control screen, which may be touch-sensitive, allowing users to operate the system. In some cases, the microphone can capture the user's voice signal, and analysis of this signal can enable the user to control certain aspects of the vehicle 100, such as adjusting the interior temperature. In other cases, music can be played to the user through the speakers.

[0121] The navigation system 113 may include map services to provide navigation for the vehicle 100, and the navigation system 113 may be used in conjunction with the vehicle's global positioning system 121 and inertial measurement unit 122. The map may be a two-dimensional map, a high-precision map, or a map constructed based on data collected during the vehicle's operation.

[0122] The perception system 120 may include several sensors for sensing information about the environment surrounding the vehicle 100. For example, the perception system 120 may include a global positioning system 121 (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU) 122, a lidar (LiDAR) 123, a millimeter-wave radar 124, an ultrasonic radar 125, and a camera device 126. The perception system 120 may also include sensors from the internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of the vehicle 100. The perception data collected by sensors in the vehicle mentioned below in this application may include data collected by the various units in the perception system 120.

[0123] The Global Positioning System 121 can be used to determine the geographical location of vehicle 100.

[0124] The inertial measurement unit 122 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In some embodiments, the inertial measurement unit 122 may be a combination of an accelerometer and a gyroscope.

[0125] The lidar 123 can use lasers to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the lidar 123 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components.

[0126] The millimeter-wave radar 124 can use radio signals to sense objects in the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing objects, the millimeter-wave radar 124 can also be used to sense the speed and / or direction of travel of objects.

[0127] The ultrasonic radar 125 can use ultrasonic signals to sense objects around the vehicle 100.

[0128] The camera device 126 can be used to capture image information of the surrounding environment of the vehicle 100. The camera device 126 may include a monocular camera, a binocular camera, a structured light camera, and a panoramic camera, etc. The image information acquired by the camera device 126 may include still image information or video stream information.

[0129] The decision control system 130 includes a computing system 131 that analyzes and makes decisions based on information acquired by the sensing system 120. The decision control system 130 also includes a vehicle controller 132 that controls the power system of the vehicle 100, and a steering system 133, a throttle 134, and a braking system 135 for controlling the vehicle 100.

[0130] The computing system 131 can process and analyze various information acquired by the perception system 120 to identify targets, objects, and / or features in the environment surrounding the vehicle 100. The targets may include pedestrians or animals, and the objects and / or features may include traffic signals, road boundaries, and obstacles. The computing system 131 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other techniques. In some embodiments, the computing system 131 may be used to map the environment, track objects, estimate object speeds, etc. The computing system 131 can analyze the acquired information and derive a control strategy for the vehicle.

[0131] The vehicle controller 132 can be used to coordinate the control of the vehicle's power battery and engine 141 to improve the power performance of the vehicle 100.

[0132] The steering system 133 can be used to adjust the forward direction of the vehicle 100. For example, in one embodiment, it can be a steering wheel system.

[0133] The throttle 134 is used to control the operating speed of the engine 141 and thus the speed of the vehicle 100.

[0134] Braking system 135 is used to control the deceleration of vehicle 100. Braking system 135 can use friction to slow down the rotational speed of wheel 144. In some embodiments, braking system 135 can convert the kinetic energy of wheel 144 into electric current. Braking system 135 may also take other forms to slow down the rotational speed of wheel 144 to control the speed of vehicle 100.

[0135] The drive system 140 includes components that provide powered motion to the vehicle 100. In one embodiment, the drive system 140 may include an engine 141, an energy source 142, a transmission system 143, and wheels 144. The engine 141 may be an internal combustion engine, an electric motor, an air-compressed engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. The engine 141 converts the energy source 142 into mechanical energy.

[0136] Examples of energy sources 142 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 142 may also provide energy to other systems of vehicle 100.

[0137] The drivetrain 143 transmits mechanical power from the engine 141 to the wheels 144. The drivetrain 143 may include a gearbox, a differential, and a drive shaft. In one embodiment, the drivetrain 143 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 144.

[0138] Some or all of the functions of vehicle 100 are controlled by computing platform 150. Computing platform 150 may include at least one processor 151, which can execute instructions 153 stored in a non-transitory computer-readable medium such as memory 152. In some embodiments, computing platform 150 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner.

[0139] Processor 151 can be any conventional processor, such as a commercially available CPU. Alternatively, processor 151 may also include a graphics processing unit (GPU), a field-programmable gate array (FPGA), a system-on-chip (SoC), an application-specific integrated circuit (ASIC), or a combination thereof. Processor 151 can be located on a device remote from the vehicle and can communicate wirelessly with the vehicle.

[0140] In some embodiments, memory 152 may contain instructions 153 (e.g., program logic) that can be executed by processor 151 to perform various functions of vehicle 100. Memory 152 may also contain additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the infotainment system 110, perception system 120, decision control system 130, and drive system 140.

[0141] The method provided in this application embodiment can be executed by a computing platform 150. For example, the processor 151 can read the program stored in the memory 152, perform target detection based on the data collected by the perception system 120, and send instructions to the decision control system 130 to control the vehicle to achieve intelligent driving function.

[0142] Of course, the method provided in this application embodiment can also be directly deployed in the perception system 120 to output information of one or more detected targets, or it can be deployed in the decision control system 130. The decision control system 130 perceives targets in the vehicle's environment based on data from the perception system, and controls the vehicle to realize the vehicle's intelligent driving function based on the perceived targets.

[0143] In addition to instruction 153, memory 152 may also store data such as road maps, route information, vehicle position, direction, speed, and other similar vehicle data, as well as other information. This information can be used by vehicle 100 and computing platform 150 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0144] The computing platform 150 can control the functions of the vehicle 100 based on inputs received from various subsystems, such as the drive system 140, the perception system 120, and the decision control system 130. For example, the computing platform 150 can utilize inputs from the decision control system 130 to control the steering system 133 to avoid obstacles detected by the perception system 120. In some embodiments, the computing platform 150 is operable to provide control over many aspects of the vehicle 100 and its subsystems.

[0145] Alternatively, one or more of these components may be installed separately from or associated with vehicle 100. For example, memory 152 may exist partially or completely separately from vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0146] Optionally, the above components are just an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 9 should not be construed as a limitation on the embodiments of this application.

[0147] The aforementioned vehicle 100 can be any vehicle or vehicle-mounted terminal capable of intelligent driving, such as a car, truck, motorcycle, bus, ship, airplane, helicopter, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, or train. This application embodiment does not impose any special limitations on this type of vehicle.

[0148] The method flow provided in the embodiments of this application will be described below in conjunction with the aforementioned application scenarios and vehicle structure.

[0149] Referring to Figure 10, a flowchart of a target detection method provided in an embodiment of this application is shown below.

[0150] 1001. Obtain input data.

[0151] The input data can include data collected by an image sensor or data read from storage space, depending on the specific application scenario.

[0152] The input data may specifically include one or more combinations of data such as images or point clouds. The input data may include information about one or more entities. For example, the input data may include an input image, which may contain one or more entities.

[0153] For example, in the intelligent driving function of a vehicle, the input image may specifically include images collected by the image sensors in the vehicle to perceive the environment in which the vehicle is located.

[0154] For example, in an industrial product quality inspection scenario, the input image may include a captured image of an industrial product, which can be used to determine the product quality through target detection.

[0155] 1002. Extract features from the input data to obtain the first feature.

[0156] Feature extraction networks can be used to extract features from input data and output the first feature. Typically, corresponding feature extraction networks can be set up for different types of inputs. For example, an image feature extraction network can be set up for image input data; a point cloud feature extraction network can be set up for point cloud input data; and for cases where both images and point clouds are input, both image feature extraction networks and point cloud feature extraction networks can be set up simultaneously, and the features output by the two feature extraction networks are fused before outputting the final feature.

[0157] Specifically, this feature extraction network can employ a CNN or a neural network containing convolutional layers. For instance, the feature extraction network could be the aforementioned encoder, which performs convolutional processing on the input data (such as an RGB image) to extract high-level semantic features of the image, i.e., the first feature. For example, ResNet can be used as the encoder, where the first feature extracted by the encoder has a resolution of 1 / 32 of the original resolution, and the output features have a dimension of 256.

[0158] 1003. Input the first feature into the decoder to obtain multiple sets of features.

[0159] After extracting the first feature from the input data, the first feature can be input into the decoder to obtain multiple sets of features, which may include features of one or more instances in the input data. The decoder can be used for single-step decoding or multiple iterative decoding. Taking any decoding process as an example, the input features can be decoded based on an attention mechanism, outputting multiple sets of features obtained in the current iteration. The input query of the attention mechanism includes multiple parameters, such as (Q, K, V) mentioned above. These multiple parameters include parameters related to the morphology of the instance detected by the decoder, or in other words, these multiple parameters are used to determine the decoding capability range of the decoder. These multiple parameters can be determined according to the morphology of the target to be decoded by the decoder. Different combinations of parameters may represent different target morphologies, which may include the specific shape of the target or the target's attributes. Therefore, in this embodiment, when extracting features, the decoder can extract features that are compatible with these multiple parameters to improve the accuracy of subsequent identification of at least one instance.

[0160] For example, during the training of the model provided in the embodiments of this application, various parameters of the decoder can be set according to the expected range of the form of the instances that the decoder can detect, so that the decoder has the ability to detect the form of the instances expected by the user, thereby detecting instances with a certain form in the actual scene to be detected, thus adapting to the detection scenarios of various forms of targets.

[0161] In one possible implementation, the aforementioned parameters include at least one of the following: query instance count K, geometric content group count N, attribute group count C, and feature width D, where K, N, C, and D are positive integers. The query instance count determines the number of instances in the input data, i.e., the number of instances corresponding to the decoder when extracting features. The geometric content group count determines the number of instance divisions in the input data. N is typically related to the shape of the instance; for example, a larger N can be set for elongated instances (e.g., aspect ratio greater than a threshold), while a smaller N can be set for rounder or shorter instances (e.g., aspect ratio not greater than a threshold). The attribute group count determines the number of attribute groups for instances in the input data. C is typically related to the types of attributes of the instance; for example, a larger C value can be set for instances containing multiple attributes, while a smaller C value can be set for instances with fewer attributes. Therefore, in this embodiment, the values ​​of K, N, C, and D can be set according to the target shape in the actual scene, thereby enabling the decoder to detect targets of various shapes in the scene.

[0162] Specifically, when there are multiple geometric content groups, the geometric content grouping is used to indicate that an instance is divided into multiple parts; when there is only one geometric content group, the geometric content grouping is used to indicate that an instance is treated as a group. In the embodiments of this application, instances can be grouped along a geometric dimension. For example, an elongated instance can be divided into multiple parts to facilitate segmented feature extraction, thereby improving the detection efficiency of the instance and enhancing the precision and accuracy of detecting the elongated instance.

[0163] Accordingly, in one possible implementation, the specific process of extracting multiple sets of features in any decoding process may include: extracting features from the multiple sets of features output in the previous iteration according to various parameters to obtain multiple sets of features, which include K sets of N*C*D features. Therefore, in the embodiments of this application, features can be extracted based on an attention mechanism, and targeted instance feature extraction can be performed based on pre-set K, N, C, and D.

[0164] Furthermore, in one possible implementation, the decoder may specifically deploy at least one of a self-attention module, a cross-attention module, or a detection module. The self-attention module is used to extract features from each of the K sets of N*C*D features in the input based on an attention mechanism. The cross-attention module is used to extract features from the K sets of N*C*D features in the input based on a cross-attention mechanism. The detection module is used to extract features from the input features.

[0165] Furthermore, in one possible implementation, during the iterative decoding process of the decoder, masks for each instance can be obtained based on multiple sets of features output by the decoder. These masks serve as input for the next iteration of the decoder, determining the position of each instance in the input data. Therefore, in this embodiment, during the iterative decoding process, masks for each instance can be output, thereby enabling more precise extraction of features in various dimensions based on the corresponding positions of each instance, improving the accuracy of the extracted features.

[0166] In one possible implementation, the similarity between multiple sets of features can be calculated to obtain a similarity map. Then, a mask corresponding to at least one instance can be generated based on the values ​​of the elements in the similarity map. An element can represent an element in the feature sequence. For example, the value of an element point greater than a preset value can be set to 1, and the value of an element point less than or equal to a preset value can be set to 0. This allows the regions belonging to the same instance to be identified based on the similarity between features, so that the subsequent decoding process can further extract features based on the regions where each instance is located, thereby improving the accuracy of the final output instance features.

[0167] 1004. Determine at least one instance and the attribute information corresponding to each instance based on multiple sets of features.

[0168] After extracting multiple sets of features representing one or more instances, at least one instance and the attribute information corresponding to each instance can be identified based on these multiple sets of features, such as the location, shape, size, and meaning of each instance in the environment.

[0169] Therefore, in this implementation, during the target detection process from the input data, parameters related to instance shapes within the decoder's detection capability range are set in the decoder. This allows for setting the necessary parameters based on instance shapes in the actual application scenario, enabling the setting of adaptive parameters for various target shapes to be detected, thus achieving detection of targets with different shapes. Furthermore, for elongated instances, no additional complex models are required, meaning that target detection of different shapes can be achieved without increasing computational complexity.

[0170] The foregoing has described the method embodiments provided in this application. The following will further describe the method embodiments provided in this application in conjunction with specific application scenarios.

[0171] For example, we will continue with the description of target detection in the context of intelligent driving of vehicles. The specific method architecture can be found in Figure 11.

[0172] The architecture can be divided into several parts, such as data acquisition module 101, feature extraction module 102, target decoding module 103 and post-processing module 104.

[0173] It should be noted that in the embodiments of this application, entities in the scene can be referred to as targets, and targets detected by executing the embodiments of this application can be referred to as instances. Targets or instances mentioned in the embodiments of this application all refer to entities in the actual scene.

[0174] The data acquisition module 101 can be used to collect data, such as environmental data (e.g., images or videos taken by a camera, laser point clouds, radar point clouds) through sensors, and preprocess the current data, scaling its range or value to a preset size and normalizing it.

[0175] The feature extraction module 102 can be used to extract features from the input data. For example, the data preprocessed by module 101 can be input into a trained feature extraction model, which outputs the extracted features. The feature extraction model can include a CNN or a CNN-based model, or it can include a Transformer or a Transformer-based model.

[0176] The target decoding module 103 can be used to further extract latent features from the features output by the feature extraction module 102 using a decoder based on an attention mechanism, and output target attribute information based on the latent features. The target decoding module 103 can be deployed as a Transformer Decoder. Specifically, it can use pre-trained target latent features as the query for the Transformer Decoder, and the features output by module 102 as the key and value. The target latent features are optimized through multiple iterations of the Transformer Decoder, or through multiple iterations of the Transformer Decoder, to output the final latent features. Based on the final output latent features, the attribute information of the detected instance is generated, including the instance's position, angle, shape, or meaning in the environment.

[0177] The post-processing module 104 can be used to post-process the attribute information of the instances output by the target decoding module 103, including information filtering and information correction. For example, it can filter the confidence level of the target predicted by the target decoding module 103 according to a set threshold, and discard lines below the threshold; or perform NMS filtering according to the target position; or both. The filtered target is the final prediction result.

[0178] The target decoding module 103 will be described in detail below.

[0179] The steps executed by the target decoding module 103 can be divided into several parts, as shown in Figure 12. These can be divided into query initialization, hidden feature update, output instance mask, and output instance attribute information, which will be described in detail below.

[0180] (1) Query initialization

[0181] First, set the query shape to (K, N, C, D), where K represents the number of query targets, N is the number of geometric content groups (i.e., dividing an instance into N parts), C represents the number of attribute content groups (e.g., classifying target attributes into multiple categories), and D represents the feature width. Abstract features of different dimensions are beneficial for extracting commonalities between different instances, different geometric regions, and different attribute groups, and for extracting features that better reflect the diversity of targets. As shown in Figure 12, the matrix includes K (N, C, D) as initial queries. Taking K=1, N=4, and C=2 as an example, this forms a 4x2 matrix, where each row represents a sub-region and each column represents an attribute group.

[0182] In this embodiment, a geometric regional modeling approach is adopted for the entire target, where each sub-region corresponds to a part of the target. This reduces the modeling difficulty for targets with long, narrow shapes and large spans. Furthermore, to distinguish targets corresponding to different attributes, the latent features of each sub-region are further segmented into attribute groups. These sub-regions and attribute segments are both explicitly distinguishable and interconnected, facilitating the extraction of global and comprehensive information about the target from latent features.

[0183] (2) Latent feature update and target mask prediction

[0184] After query initialization, the vectors are flattened into N×C×D vectors for each of the K targets. Simultaneously, the sensor features are transformed into (M, N×C×D) features, where M is the number of features. In the Transformer Decoder's self-attention and cross-attention, the MHA head is set to N×C. The number of heads is typically a multiple of N*C; for example, when C is set to 1, the number of heads is a multiple of N, and when N is set to 1, the number of heads is a multiple of C. That is, query feature segments, distinguished by region and attribute group dimensions, are used as different heads to interact with image features. Therefore, the interacted features are divided into two dimensions: region location and attribute type, predicting corresponding content. Features within these two dimensions are also correlated. Finally, the latent features are further extracted using FFN to obtain updated latent features.

[0185] When the number of Transformer Decoders is greater than 1 (i.e., L>1), multiple rounds of Transformer Decoders are cascaded to perform latent feature updates, with the input latent features of each subsequent round becoming the output latent features of the previous round. Furthermore, except for the first round of Transformer Decoders, subsequent Transformer Decoders all have C target masks as inputs to the cross-attention attention mask. These C target masks correspond to the N×C heads of the cross-attention, with each attribute group's features corresponding to its target mask. The remaining operations are consistent or similar across all Transformer Decoders.

[0186] In the L rounds of the Transformer Decoder, except for the last round, each round predicts the target mask, which is then output as the attention mask to the next round. After obtaining the latent features updated by the Transformer Decoder, these latent features are separated into attribute groups, i.e., divided into C N×D attribute features for each target. These multiple attribute features are transformed by the MLP and then compared pixel-by-pixel with the sensor's transformed features to obtain corresponding similarity maps. The similarity maps are then judged by a fixed threshold t (the specific value can be set according to the actual scene) to obtain the target mask. For example, portions higher than t are considered positive masks (e.g., set to positive or 1), while portions lower than t are considered negative masks (e.g., set to -1 or 0). This mask needs to be inverted before entering the next Transformer Decoder. In subsequent Transformer Decoders, each attribute feature will interact only with features in relevant regions using the C sets of masks, thus focusing more on features in relevant regions and shielding the deviations caused by features in irrelevant regions from the latent feature update.

[0187] As shown in Figure 13, the flowchart includes L iterations of Transformer Decoder. The output features of the previous Transformer Decoder and the corresponding mask are input into the next Transformer Decoder. At the same time, the features output by the feature extraction module 102 are used as the input of each Transformer Decoder, and the latent features output by the last Transformer Decoder are output.

[0188] (3) Output instance attribute information

[0189] The latent features obtained after L rounds of Transformer Decoder can be used to predict the attribute information of each target. Multiple different MLPs can be set up to predict target attribute information, or a larger model can be set up including multiple MLP network layers to predict target attribute information. For example, the latent features can be separated into C N×D attribute features for each of the K query targets. After being input into multiple MLPs, different types of attributes can be output, such as the target's color, position, size, or the meaning it represents in the environment.

[0190] In this embodiment, for targets of different shapes, the latent features (queries) of each target instance can be divided into N groups, with each group corresponding to a sub-region of the instance. This is equivalent to dividing an instance into multiple segments for querying, thus performing segmented queries on the target. Based on the self-attention and cross-attention of the Transformer Decoder, explicit features are extracted from the interior of each segment, and the correlation between features is combined to obtain more accurate latent features that are explicit and correlated with other features. When outputting the target location and attributes, each group outputs a portion of the sub-region corresponding to the instance, and the N groups jointly output the location and attributes of the entire instance, thereby outputting more accurate target attribute information based on more accurate latent features.

[0191] Furthermore, the target attributes can be grouped. For example, setting C to greater than 2 means extracting attribute features from C groups, with each group corresponding to different attributes of an instance. When outputting the target location and attributes, each attribute group outputs a portion of the attribute corresponding to that instance, and the C groups output all attributes of that instance. This allows for a target query method that decouples the target attributes, resulting in more accurate attribute information.

[0192] Furthermore, during the decoding process, the target's mask region is added as input. The similarity between the previously output latent features and the transformed extracted features is calculated, and the similarity matrix is ​​binarized using a fixed threshold. The resulting binarized mask is then used as the next attention mask. This allows for more focus on the features in the attention mask region when extracting features based on the attention mechanism, thus enabling more accurate extraction of the features corresponding to the target.

[0193] Therefore, in this embodiment, a single-query target detection method can be implemented, which has low computational requirements and does not significantly increase the computational load of the model. Furthermore, a segmented query modeling method based on multi-region decoupling is used to reduce the impact of limited target receptive field on inaccurate detection of objects with narrow shapes and large spans, thereby improving target detection accuracy. The segmented query modeling method based on multi-attribute decoupling solves the problem that existing methods are difficult to extend to multi-attribute modeling, and can output more attributes to meet product-level requirements. The prediction mask-based ROI region modeling method shields the influence of irrelevant region features during latent feature updates, and only interacts with the features of relevant regions, making it easier for latent features to obtain the most relevant features of the target, improving accuracy and accelerating convergence.

[0194] In the method provided in this application, different shapes of targets can be detected by adjusting the decoder parameters. Entities in real-world scenarios can typically be categorized as elongated entities with high aspect ratios, or entities with low aspect ratios or circular shapes. Different parameters can be adaptively set for entities of different shapes. The following exemplary embodiments for different scenarios are described.

[0195] Scenario 1: Vehicle Visual Perception

[0196] For example, in a vehicle's visual perception system, intelligent driving vehicles need to identify the vehicle's position and wheel orientation in environmental images captured by cameras during driving, in order to make future driving decisions. This system mainly consists of a convolutional neural network and a Transformer Decoder, and can be functionally divided into an image feature extraction module, a target decoding module, and a post-processing module. The steps in the scenario are described in detail below.

[0197] Taking an image as the input data as an example, the method flow provided in this application embodiment can be shown in Figure 14.

[0198] 1401. Image feature extraction.

[0199] The input image can include images captured by sensors installed in the vehicle. The feature extraction network can be the encoder in a Transformer. The encoder performs convolution processing on the input image to extract high-level semantic features. Specifically, an image classification model can be used, and the feature resolution can be gradually reduced through pooling or stride convolution to finally obtain the high-dimensional semantic features of the image.

[0200] For example, as shown in Figure 15, ResNet is used as the encoder here. The encoder extracts image features with a resolution of 1 / 32 of the original resolution and outputs image features with a dimension of 256.

[0201] The ResNet example used here is for illustrative purposes only. There are no limitations on the number of convolutional layers, intermediate feature dimensions, resolution, etc. It can be replaced with other model structures used for feature extraction or transformation.

[0202] 1402. Target Decoding.

[0203] Combined with the aforementioned target decoding module 103, which can deploy L layers of Transformer Decoders (or a single Transformer Decoder can perform multiple decoding operations), each Transformer Decoder layer includes a self-attention mechanism, a cross-attention mechanism, and an FFN. Taking the image features output from step 1401 as input, after query initialization, latent feature update, prediction of the target mask, and prediction of the target attributes, the final predicted attributes are output.

[0204] For example, the target decoding process can be shown in Figure 16, and the model structure corresponding to the specific decoding process can be shown in Figure 17.

[0205] First, for query initialization. In this application scenario, since the detection target is not a long, narrow object with a large span, such as a vehicle or task in the environment, and is different from a lane or other relatively long and narrow entity, the number of geometric content groups N is set to 1. For ease of understanding, the value of N is omitted here. In this embodiment, the goal of the detection task is to obtain the position of the car and the orientation of the wheels, so the attribute content group C is set to 2. The initial query shape for K targets is (K, 2, 128), which is obtained by adding two orthogonal pre-trained parameters, namely the instance-level abstract feature and the attribute-level abstract feature shown in Figure 17, with shapes of (K, 1, 128) and (1, 2, 128), respectively, where K represents the number of query instances. For simplicity, a single query is used as an example, i.e., K = 1, but this is not a limitation, and the specific value of K can be determined according to the actual application scenario.

[0206] Subsequently, the latent features are updated. After query initialization, the vector is combined into 256, and the image features are transformed into (M, 256) features, where M is the number of features. In the Transformer Decoder's self-attention and cross-attention, the MHA head setting can be 2. Except for the first round of Transformer Decoder, subsequent Transformer Decoders have two target masks as the attention mask inputs for cross-attention. The two target masks are the mask of the entire vehicle instance and the wheel mask, respectively. That is, the identified instances are the vehicle and the vehicle's wheels, corresponding to the tasks of predicting the vehicle position and wheel orientation, respectively. In cross-attention, two heads are deployed to implement these two tasks respectively. After sequentially passing through self-attention, cross-attention, and FFN, the updated latent features are obtained.

[0207] In one iteration of the latent feature update process, the target mask is also predicted as input for the next iteration. In the current iteration, after obtaining the latent features updated by the Transformer Decoder, these latent features are separated into two 128-attribute features based on their attribute groups. After MLP transformation, the attribute features are compared pixel-by-pixel with the image feature transformation features to obtain corresponding similarity maps. These similarity maps are then judged by a fixed threshold t, for example, setting values ​​greater than t to 1 (or positive) and values ​​less than t to 0 (or negative), thus obtaining the target mask. In this embodiment, the detection task aims to obtain the vehicle's position and wheel orientation. The former is related to the overall target vehicle, while the latter is only related to the target vehicle's wheels; therefore, the two attribute groups predict the mask for the entire vehicle body and the mask for the wheels, respectively. This mask needs to be inverted before entering the next Transformer Decoder.

[0208] Subsequently, target attribute information is predicted based on the updated latent features. The latent features obtained after L rounds of Transformer Decoder are used to predict the target car's position and wheel orientation. This feature is separated into two 128 attribute features. After passing through different MLPs, the first attribute feature predicts the target car's confidence score and bounding box, while the second attribute feature predicts the target car's wheel orientation angle.

[0209] The output of the target decoding module includes the following parts:

[0210] Target BBox: Represents the bounding box of the target location, with values ​​within the image. Its shape is 4.

[0211] Target confidence: Represents the probability of the target's existence or the reliability of the detection result for the target, with a value ranging from 0 to 1. The shape is 1.

[0212] Target wheel orientation angle: indicates the direction of the target wheel, with a value range of 0 to 360°, and a shape of 1.

[0213] 1403. Post-processing.

[0214] For one or more targets output in step 1402, filtering can be performed based on the confidence level of each target. For example, lines below a threshold will be discarded. Additionally, NMS filtering can be performed on the target bounding boxes. The filtered targets are the final prediction results.

[0215] Therefore, in this embodiment, the query latent features that explicitly distinguish the predicted attributes of different segments are used. Simultaneously, during attention interactions, different attribute segments are explicitly distinguished, increasing the diversity between different attention maps in the MHA. This makes different attribute segments independent yet interconnected, which is beneficial for the latent features to extract global and comprehensive information about the target. Furthermore, a target mask predicted by the previous layer is added to the (not the first) Transformer Decoder, allowing the latent features to interact only with relevant region features, focusing more on the features of the target's region, thereby extracting more important information, improving target detection accuracy, and accelerating model convergence.

[0216] Scenario 2: Lane Detection

[0217] Similarly, in the perception system of intelligent driving vehicles, the vehicle needs to identify the location and type of road structures in the environment during driving in order to make future driving decisions. This system mainly consists of convolutional neural networks and Transformer Decoders, and can be functionally divided into multi-type feature extraction modules, road structure decoding modules, and post-processing modules.

[0218] Taking images and point clouds as input data as an example, the method flow provided in this application embodiment can be shown in Figure 18.

[0219] 1801. Multi-type feature extraction.

[0220] In scenarios with multiple types of input data, feature extractors can be set separately for different types of data.

[0221] For example, if the input data includes images collected by an image sensor and point cloud data collected by a LiDAR, an image encoder and a point cloud encoder can be set up separately. In order to fuse features extracted from different types of data, a BEV fusion processor can also be set up.

[0222] For example, the specific feature extraction process can be illustrated in Figure 19. The image encoder performs convolution processing on the input image (RGB image) to extract high-level semantic features. The LiDAR encoder performs voxelization processing on the input LiDAR point cloud and extracts high-level semantic features from the point cloud. The BEV fusion unit uses camera intrinsic and extrinsic parameters to transform or map image features to the BEV space and fuses them with point cloud features to obtain BEV features. The image features extracted by the image encoder have a resolution of 1 / 32 of the original resolution, and the output dimension is 512. The feature shape obtained by the LiDAR encoder is 288×160, corresponding to a physical range of 99.2 meters in front, 16 meters behind, 32 meters to the left, and 32 meters to the right, with a single pixel size of 0.4 meters × 0.4 meters. The shape of the BEV features output by the BEV fusion unit remains unchanged. The transformation from image features to BEV space can use common transformation networks. Feature fusion under the two modalities of BEV can use a common feature dimension concatenation method, and then be reduced to 512 dimensions.

[0223] It should be noted that the embodiments of this application do not limit the number of convolutional layers, intermediate feature dimensions, resolution, etc. of the image encoder, LiDAR encoder and BEV fusion unit. They can be replaced with other structures for feature extraction and transformation. The network structure in this embodiment is only an example of implementing the embodiments of this application.

[0224] 1802. Road structure decoding.

[0225] Similar to step 1402 above, L layers of Transformer Decoders can also be set here, each containing a self-attention, a cross-attention, and an FFN. The features output from step 1801 are received, and steps such as query initialization, latent feature update, prediction of road structure mask, and prediction of lane lines are performed to output road structure information.

[0226] For example, the process of decoding road structures can be illustrated in Figure 20.

[0227] First, query initialization. In this embodiment, since the detection target is a long and narrow object with a large span, the number of geometric content groups N is set to 4, that is, each target is divided into 4 sub-regions. The initial query shape for K targets is (K,4,2,64), which is obtained by adding three orthogonal trained parameters, namely the instance-level abstract feature, subpart-level abstract feature, and attribute-level abstract feature shown in Figure 20, in a staggered manner. The shapes are represented as (K,1,1,64), (1,4,1,64), and (1,1,2,64), respectively, where K is the number of query instances. The goal of the detection task in this embodiment is to obtain the position and attributes of the lane centerline and the lane boundary line, so the attribute content group C is set to 2. For simplicity, only a single query is used as an example, i.e., K=1, as shown in Figure 20.

[0228] Secondly, for latent feature updates, after query initialization, the centerline group is extracted, and a keypoint is predicted for each sub-region. This keypoint, after positional encoding, serves as the positional encoding for the query, and is used to match the query with the key and the positional encoding of the key. Here, since the receptive field of the keypoint in the centerline basically covers the entire lane instance, this keypoint can be used as the positional encoding source for the entire query containing multiple attribute groups. This is then used to form a 512 vector, and the BEV feature is transformed into (M, 512) features, where M is the number of features. In the Transformer Decoder's self-attention and cross-attention, the MHA head is set to 8. Except for the first round of the Transformer Decoder, subsequent Transformer Decoders have two target masks as the attention mask input for cross-attention. The two target masks are the centerline mask and the lanebound mask, corresponding to the centerline and lanebound prediction tasks, respectively, and are applied to the attribute feature parts corresponding to the 8 heads in cross-attention. After sequential processing through self-attention, cross-attention, and FFN, the updated latent features will be obtained.

[0229] Between multiple rounds of Transformer Decoder, the centerline group of latent features is extracted and undergoes an additional transformation to predict the keypoint offset, thereby further correcting the keypoint position based on the keypoint offset, so that the keypoint gradually approaches the target position under the action of multiple rounds of decoder. For example, the keypoint offset is added to the keypoint and used as the new keypoint input to the next Transformer Decoder.

[0230] In any iteration of latent feature update, a road structure mask can be predicted and used as input for the next iteration. Specifically, after obtaining the latent features updated by the Transformer Decoder, these latent features are separated according to attribute groups, i.e., divided into two (4, 64) attribute features. These attribute features are flattened, transformed by MLP, and then compared pixel-by-pixel with the transformed features of the image features to obtain corresponding similarity maps. The similarity maps are then judged by a fixed threshold t to obtain the target mask. In this embodiment, the detection task aims to obtain the position and attributes of the lane centerline and the lane boundary line. The former is related to the centerline of the lane instance, and the latter is related to the boundary line of the lane instance; therefore, the two attribute groups predict the centerline mask and the lanebound mask, respectively. This mask needs to be inverted when it enters the next Transformer Decoder.

[0231] Subsequently, the road structure is predicted based on the latent features obtained from the aforementioned process. The latent features obtained after L rounds of Transformer Decoder are used to predict the position and attributes of the lane centerline and the lane boundary lines. This feature is separated into two (4, 64) attribute features. After passing through different MLPs, the centerline group features predict the position offset and attributes of the centerline, and the lanebound group features predict the position offset and attributes of the lanebound. The position of each group is obtained by adding the position offset of each group to the keypoints. Different sub-regions predict different segments of the lane line, and the sets of predictions from all sub-regions are sequentially combined to form the complete lane line. For example, the centerline group predicts the centerline position offset, and its output shape after passing through the MLP is (4, 16, 2), where 16 is the number of predicted points in each sub-region. Finally, the output is sequentially combined into a point sequence of shape (96, 2).

[0232] The output of the road structure decoding module includes the following parts:

[0233] centerline coordinates: Represents the sequence of points at the centerline position. The shape is (96, 2).

[0234] centerline confidence score: Indicates the probability that the centerline exists. Values ​​range from 0 to 1. Shape is 1.

[0235] centerline type: Indicates lane type, such as regular lane, emergency lane, etc. Shape is 96.

[0236] lanebound coordinates: a sequence of points on both sides of the lane boundary line. The shape is (2, 96, 2).

[0237] lanebound type: The type of lane boundary lines, such as solid lines, dashed lines, etc. Its shape is (2, 96).

[0238] 1803. Post-processing.

[0239] After obtaining the road structure prediction results, the attribute information of each road structure output in step 1802 can be filtered. For example, filtering can be performed based on the confidence level of the lane centerline according to a set threshold, and lines below the threshold will be discarded. The filtered result is the final prediction result.

[0240] In this implementation, query latent features are used to explicitly distinguish different target regions and predicted attributes of different segments. Simultaneously, during attention interactions in the decoding process, different region segments and attribute segments are explicitly distinguished, increasing the diversity between different attention maps in the MHA. This makes different region segments and attribute segments independent yet interconnected, which is beneficial for the latent features to extract global and comprehensive information about the target. Finally, a target mask predicted by the previous layer is added to the (not the first) Transformer Decoder, allowing the latent features to interact only with relevant region features, extracting more important information, improving target detection accuracy, and accelerating model convergence.

[0241] Furthermore, to facilitate understanding of the effects achieved by the methods provided in the embodiments of this application, the effects are compared in some datasets as follows.

[0242] Tables 1 and 2 show the quantitative results of the embodiments provided in Scenario 2 of this application evaluated on the validation set, with the additions and subtractions all relative to the baseline model.

[0243] Table 1

[0244] Table 2

[0245] As shown in Table 1, in Scenario 2, the near-end position accuracy of the center line and boundary line of the left lane is improved by more than 2.2%; the near-end and far-end position accuracy of the boundary lines on both sides of the middle lane is improved by 7-15%. As shown in Table 2, Scenario 2 improves the recall and accuracy by an average of >3% for various lane types.

[0246] Table 3 shows the evaluation results on the simulation difficulty verification set of the embodiments provided in Scenario 2 above in this application.

[0247] Table 3

[0248] Compared to the baseline model, lane loss is reduced by 16%, and missed detections of important lanes (such as difficult splits and target lanes) are reduced by 14%. False detections of lanes near the vehicle that affect driving decisions are reduced by 22%. Lane jump jitter is reduced by 56%, and the accuracy and stability of lane far-end detection positions are significantly improved.

[0249] For example, in the case of road change scenarios, the baseline model is compared with the embodiment provided in Scenario 2 of this application, as shown in Figure 21. Typically, in road change scenarios, lane line loss is a common problem, i.e., lane line tracking is lost. In Figure 21, the right side of the real-world image is an image taken with a telephoto lens and the detection results superimposed, an image taken with a wide-angle lens and the detection results superimposed, the middle image is the centerline detection result, and the right side is the centerline + lanebound detection result. Clearly, in this scenario, the baseline model fails to detect the centerline and lanebound of the vehicle's lane, while the embodiment in Scenario 2 of this application can detect them correctly.

[0250] For example, lane lines are easily lost in ramp scenarios, as shown in Figure 22. In this scenario, the lane where the vehicle is located splits into two lanes at the far end, so two lane line instances need to be detected. The baseline model missed detecting a centerline and its corresponding lanebound in this scenario, while scenario two of this application correctly detected the centerline and lanebound.

[0251] The flow of the method in the embodiments of this application has been described above. The structure of the apparatus for performing the method provided in the embodiments of this application will be described below.

[0252] Referring to Figure 23, a schematic diagram of a target detection device provided in an embodiment of this application is shown. The target detection device includes:

[0253] Module 2301 is used to acquire input data;

[0254] Feature extraction module 2302 is used to extract features from input data to obtain the first feature;

[0255] The decoder module 2303 is used to input the first feature into the decoder to obtain multiple sets of features. The decoder is used to decode the input data at least once. Any decoding process in the at least once includes: decoding the input features based on the attention mechanism and outputting multiple sets of features obtained in the current iteration. The input query of the attention mechanism includes multiple parameters, including parameters related to the morphology of the instance that the decoder can detect.

[0256] The detection module 2304 is used to determine at least one instance and the attribute information corresponding to each instance based on multiple sets of features.

[0257] In one possible implementation, multiple parameters include at least one of the following: query instance count K, geometric content group count N, attribute group count C, and feature width D. The query instance count is used to determine the number of instances in the input data, the geometric content group count is used to determine the number of partitions of instances in the input data, and the attribute group count is used to determine the number of attribute groups of instances in the input data. K, N, C, and D are positive integers.

[0258] In one possible implementation, the decoder module 2303 is specifically used to: extract features from multiple sets of features output from the previous iteration according to multiple parameters, to obtain multiple sets of features, including K sets of N*C*D features.

[0259] In one possible implementation, the aforementioned decoder includes at least one of a self-attention module, a cross-attention module, or a detection module. The self-attention module is used to extract features from each of the K sets of N*C*D features in the input based on an attention mechanism. The cross-attention module is used to extract features from the K sets of N*C*D features in the input based on a cross-attention mechanism. The detection module is used to extract features from the input features.

[0260] In one possible implementation, when there are multiple geometric content groups, the geometric content grouping is used to indicate that an instance is divided into multiple parts; when there is only one geometric content group, the geometric content grouping is used to indicate that an instance is treated as a group.

[0261] In one possible implementation, any one of the decoding processes in at least one decoding operation further includes: determining a mask corresponding to at least one instance based on multiple sets of features output by the decoder, wherein the mask corresponding to at least one instance is used as input for the next decoding process to determine the position of at least one instance.

[0262] In one possible implementation, the decoder module 2303 is specifically used to: obtain the similarity between multiple sets of features to obtain a similarity map; and generate a mask corresponding to at least one instance based on the values ​​of the elements in the similarity map.

[0263] In one possible implementation, the detection module 2304 is specifically used to: input multiple sets of features into at least one perceptual network to obtain at least one instance and attribute information corresponding to each instance.

[0264] In one possible implementation, the apparatus further includes a filtering module 2305, configured to filter at least one instance and attribute information corresponding to each instance, and output the filtered at least one instance and attribute information corresponding to each instance.

[0265] Figure 24 shows a schematic diagram of the hardware structure of a target detection device 240 provided in an embodiment of this application. This target detection device 240 can be used to implement the steps of the methods shown in Figures 10 to 22.

[0266] The target detection device 240 shown in Figure 24 may include a processor 2401, a memory 2402, a communication interface 2403, and a bus 2404. The processor 2401, the memory 2402, and the communication interface 2403 can be connected via the bus 2404.

[0267] The processor 2401 is the control center of the target detection device 240. It can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adapted to the actual application scenario.

[0268] As an example, processor 2401 may include one or more CPUs, and may also include other processors, such as the CPU, NPU or GPU shown in Figure 24.

[0269] The memory 2402 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0270] In one possible implementation, the memory 2402 may exist independently of the processor 2401. The memory 2402 can be connected to the processor 2401 via a bus 2404 and is used to store data, instructions, or program code. When the processor 2401 calls and executes the instructions or program code stored in the memory 2402, it can implement the methods provided in the embodiments of this application, such as the methods shown in Figures 10 to 22.

[0271] In another possible implementation, the memory 2402 can also be integrated with the processor 2401.

[0272] The communication interface 2403 is used for the target detection device 240 to connect with other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 2403 may include a receiving unit for receiving data and a transmitting unit for transmitting data.

[0273] Bus 2404 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 24, but this does not indicate that there is only one bus or one type of bus.

[0274] It should be noted that the structure shown in FIG24 does not constitute a limitation on the target detection device 240. In addition to the components shown in FIG24, the target detection device 240 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0275] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0276] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0277] This application also provides a computer-readable storage medium storing a program for training a model or performing inference tasks, which, when run on a computer, causes the computer to perform all or part of the steps in the methods described in the embodiments shown in Figures 10 to 22 above.

[0278] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the aforementioned processor or processor functions, and one or more interfaces. When the digital processing chip integrates a memory, it can perform the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the method steps of any one or more of the foregoing embodiments based on the program code stored in the external memory.

[0279] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0280] The target detection device provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the target detection device to execute the methods described in the embodiments shown in Figures 10 to 22. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0281] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0282] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0283] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0284] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0285] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0286] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps in this application does not imply that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved. The division of modules in this application is a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed may be through some ports, and the indirect coupling or communication connection between modules may be electrical or other similar forms, which are not limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules can be selected to achieve the purpose of the solution in this application according to actual needs.

Claims

A target detection method characterized by comprising: The method comprises: acquiring input data; extracting features from the input data to obtain first features; inputting the first features into a decoder to obtain a plurality of groups of features, the decoder being configured to decode the input data at least once, any decoding process in the at least once decoding comprising: decoding the input features based on an attention mechanism, and outputting a plurality of groups of features obtained in the current iteration, the input query of the attention mechanism comprising a plurality of parameters, the plurality of parameters comprising parameters related to the morphology of instances detectable by the decoder; determining at least one instance and attribute information corresponding to each instance according to the plurality of groups of features. The method of claim 1, wherein The plurality of parameters comprises at least one of a query instance number K, a geometric content grouping number N, an attribute grouping number C, and a feature width D, the query instance number being used to determine the number of instances in the input data, the geometric content grouping number being used to determine the division number of instances in the input data, the attribute grouping number being used to determine the grouping number of attributes of instances in the input data, K, N, C, and D being positive integers. The method according to claim 2, characterized in that The determination of the at least one instance and the attribute information corresponding to each instance according to the plurality of groups of features comprises: extracting features from the plurality of groups of features output in the last iteration according to the plurality of parameters to obtain the plurality of groups of features, the plurality of groups of features comprising K groups of N*C*D features. The method according to claim 3, characterized in that The decoder comprises at least one of a self-attention module, a cross-attention module, or a detection module, the self-attention module being configured to extract features from each group of features in the input K groups of N*C*D features based on an attention mechanism, the cross-attention module being configured to extract features from the input K groups of N*C*D features based on a cross-attention mechanism, and the detection module being configured to extract features from the input features. According to any one of claims 2-4, in the case where the number of geometric content groupings is a plurality of groupings, the geometric content groupings are used to indicate that one instance is divided into a plurality of parts; and in the case where the number of geometric content groupings is one grouping, the geometric content grouping is used to indicate that one instance is taken as one group. Any decoding process in the at least once decoding further comprises: determining a mask corresponding to the at least one instance according to the plurality of groups of features output by the decoder, the mask corresponding to the at least one instance being taken as the input of the next decoding process, and being used to determine the position of the at least one instance. The method according to any one of claims 1-5, characterized in that The determination of the mask corresponding to the at least one instance according to the plurality of groups of features output by the decoder comprises: obtaining a similarity between the plurality of groups of features to obtain a similarity map; The method according to claim 6, characterized in that generating the mask corresponding to the at least one instance according to the value of an element in the similarity map. The determination of the at least one instance and the attribute information corresponding to each instance according to the plurality of groups of features comprises: inputting the plurality of groups of features into at least one perception network to obtain the at least one instance and the attribute information corresponding to each instance. The method according to any one of claims 1-7, characterized in that The method further comprises: ​ The method according to any one of claims 1-8, characterized in that ​ Filter the at least one instance and the attribute information corresponding to each instance, and output the filtered at least one instance and the attribute information corresponding to each instance. A target detection device characterized by comprising: Comprise: An acquisition module is used to acquire input data; A feature extraction module is used to extract features from the input data to obtain first features; A decoder module is used to input the first features into a decoder to obtain multiple groups of features, the decoder is used to decode the input data at least once, and any decoding process in the at least once decoding includes decoding the input features based on an attention mechanism to output multiple groups of features obtained in the current iteration, and the input query of the attention mechanism includes multiple parameters, and the multiple parameters include parameters related to the morphology of the instances detectable by the decoder; A detection module is used to determine at least one instance and attribute information corresponding to each instance according to the multiple groups of features. The apparatus of claim 10, wherein The multiple parameters include at least one of a query instance number K, a geometric content grouping number N, an attribute grouping number C, and a feature width D, the query instance number is used to determine the number of instances in the input data, the geometric content grouping number is used to determine the division number of the instances in the input data, and the attribute grouping number is used to determine the grouping number of the attributes of the instances in the input data, K, N, C and D are positive integers. The apparatus of claim 11, wherein The detection module is specifically used to: Extract features from the multiple groups of features output from the last iteration according to the multiple parameters to obtain the multiple groups of features, and the multiple groups of features include K groups of N*C*D features. The apparatus of claim 12, wherein The decoder includes at least one of a self-attention module, a cross-attention module or a detection module, the self-attention module is used to extract features from each group of features in the input K groups of N*C*D features based on an attention mechanism, the cross-attention module is used to extract features from the input K groups of N*C*D features based on a cross-attention mechanism, and the detection module is used to extract features from the input features. The device according to any one of claims 11-13, wherein In the case that the number of geometric content groups is multiple groups, the geometric content group is used to indicate that one instance is divided into multiple parts; In the case that the number of geometric content groups is one group, the geometric content group is used to indicate that one instance is taken as a group. The apparatus of any one of claims 10-14, wherein Any decoding process in the at least once decoding further includes: Determine a mask corresponding to the at least one instance according to the multiple groups of features output by the decoder, and the mask corresponding to the at least one instance is used as the input of the next decoding process to determine the position of the at least one instance. The apparatus of claim 15, wherein The determination of the mask corresponding to the at least one instance according to the multiple groups of features output by the decoder includes: Obtain a similarity between the multiple groups of features to obtain a similarity graph; Generate the mask corresponding to the at least one instance according to the value of the element in the similarity graph. The apparatus of any one of claims 10-16, wherein The detection module is specifically used to: Input the multiple groups of features into at least one perception network to obtain the at least one instance and the attribute information corresponding to each instance. The apparatus of any one of claims 10-17, wherein The device further comprises: A filtering module is configured to filter the at least one instance and the attribute information corresponding to each instance, and output the filtered at least one instance and the attribute information corresponding to each instance. A target detection device characterized by comprising: The application relates to a computer program product comprising a program for executing the steps of the method according to any one of claims 1 to 9 when the program is executed by a processing unit. A computer-readable storage medium, characterized by The application relates to a computer program product comprising a program for executing the steps of the method according to any one of claims 1 to 9 when the program is executed by a processing unit. A computer program product, characterized in that The application relates to a computer program product comprising a program for executing the steps of the method according to any one of claims 1 to 9 when the program is executed by a processing unit.

Images