Target detection method, method and device for establishing spatial correlation perception model

Abstract

The embodiment of the application discloses a target detection method, a method and a device for establishing a space correlation perception model. The main method comprises the following steps: acquiring N frames of sensor data to be detected, wherein N is a positive integer; performing target detection on each frame of sensor data by using a deep convolutional neural network to obtain a second target detection result of each frame of sensor data; and performing first correction prediction on the second target detection result of each frame of sensor data based on the space correlation between targets in the second target detection result of the same frame of sensor data to obtain a first target detection result of each frame of sensor data. The accuracy of target detection can be improved by the method.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a target detection method, a method and apparatus for establishing a spatial correlation perception model. Background Technology

[0002] Object detection is currently primarily used in fields such as autonomous driving and robotics. It is a crucial component of autonomous driving systems, as autonomous vehicles not only need to identify the type of obstacles but also the precise location and orientation of objects to provide the planning and control module with information for developing a suitable route. Object detection in autonomous vehicles mainly relies on sensors, including LiDAR, millimeter-wave radar, and onboard cameras. These sensors acquire multiple frames of data, which are then processed by object detection algorithms to detect objects within each frame.

[0003] Although some target detection algorithms based on sensor data already exist, their detection accuracy still needs improvement. Summary of the Invention

[0004] In view of this, this application provides a target detection method, a method and apparatus for establishing a spatial correlation perception model, so as to improve the accuracy of target detection.

[0005] This application provides the following solution:

[0006] Firstly, this application provides a target detection method, which includes:

[0007] Obtain N frames of sensor data to be detected, where N is a positive integer;

[0008] Target detection is performed on each frame of sensor data using a deep convolutional neural network to obtain the second target detection result for each frame of sensor data.

[0009] Based on the spatial correlation between targets in the second target detection results of the same frame of sensor data, the first correction prediction is performed on the second target detection results of each frame of sensor data to obtain the first target detection results of each frame of sensor data.

[0010] According to one achievable method in an embodiment of this application, the step of performing target detection on each frame of sensor data using a deep convolutional neural network to obtain a second target detection result for each frame of sensor data includes:

[0011] Multiple candidate regions are generated for each frame of sensor data;

[0012] The feature extraction of candidate regions is performed on each frame of sensor data using a deep convolutional neural network, and the extracted features are used to obtain the second target detection result of each frame of sensor data.

[0013] According to one achievable method in this application embodiment, the first correction prediction of the second target detection results of each frame of sensor data based on the spatial correlation between targets in the second target detection results of the same frame of sensor data includes:

[0014] Each frame of sensor data is used as the current input frame sensor data, and the following steps are executed:

[0015] Using the second target detection result of the current input frame sensor data, an information graph of the current input frame sensor data is established and input into the spatial association perception model. The information graph includes nodes and edges between nodes. The nodes include targets detected in the current input frame sensor data, and the edges represent the associations between nodes.

[0016] The spatial correlation perception model uses a graph convolutional neural network to extract features from the information graph of the current input frame sensor data, and the mapping layer uses the extracted features to perform a first correction prediction to obtain the first target detection result of the current input frame sensor data.

[0017] According to one achievable method in this application embodiment, the step of establishing an information map of the current input frame sensor data using the second target detection result of the current input frame sensor data includes:

[0018] The region of interest (RoI) of each target in the current input frame sensor data is determined using the second target detection result of the current input frame sensor data. The RoI information of each target is used as the information of each node in the information graph. The edges in the information graph are determined using the positional relationship between the RoIs of each target.

[0019] According to one achievable method in an embodiment of this application, the feature extraction of the information graph of the current input frame sensor data using a graph convolutional neural network includes:

[0020] In each iteration, the features of each edge in the information graph are calculated by performing nonlinear transformation on the features of the nodes at both ends of the edge. The edges connected to the nodes are then merged using pooling to obtain the features of the node in the next iteration.

[0021] After the iteration is completed, the features of the same node obtained in all iterations are merged to obtain the features of that node extracted from the information graph and provided to the mapping layer.

[0022] According to one achievable method in an embodiment of this application, N is greater than 1; the method further includes:

[0023] Target tracking is performed on the first target detection results of each frame of sensor data to associate information of the same target, thereby obtaining a sequence of detection results for each target;

[0024] Based on the temporal correlation of information about the same target in the detection result sequence of each target, a second correction prediction is performed on the first target detection result of each frame of sensor data to obtain the third target detection result of each frame of sensor data.

[0025] According to one achievable method in the embodiments of this application, based on the temporal correlation of information of the same target in the detection result sequence of each target, the second correction prediction of the first target detection result of each frame of sensor data includes:

[0026] Input the sensor data of each frame and the detection result of the first target into the time-related perception model respectively;

[0027] The temporal correlation perception model utilizes the contextual target features of the current input frame sensor data to process the features of each target in the first target detection result of the current input frame sensor data using a self-attention mechanism, thereby obtaining the feature representation of each target in the current input frame sensor data. The contextual target features of the current input frame sensor data include the features of each target in the sensor data before and / or after the current input frame sensor data, where N is a preset positive integer. The third target detection result of the current input frame sensor data is obtained by mapping the feature representation of each target in the current input frame sensor data.

[0028] According to one possible implementation method in an embodiment of this application, the sensor data includes point cloud data;

[0029] The target detection results include the target's category information, location information, size information, and orientation information.

[0030] Secondly, a method for establishing a spatial correlation sensing model is provided, the method comprising:

[0031] Obtain a first training sample, which includes multi-frame sensor data, a second target detection result obtained by performing target detection on the multi-frame sensor data, and labels annotating each target information in the multi-frame sensor data;

[0032] The spatial association perception model is obtained by training the first training sample.

[0033] The spatial association perception model is based on the spatial association between targets in the second target detection results of each frame of sensor data, and performs a first correction prediction on the second target detection results of each frame of sensor data to obtain the first target detection results of each frame of sensor data.

[0034] The training objective includes minimizing the difference between the first target detection result and the corresponding label in each frame of sensor data.

[0035] According to one feasible method in an embodiment of this application, training the spatial association perception model using the first training data includes:

[0036] An information graph of each frame of sensor data is established using the second target detection results of each frame of sensor data. The information graph includes nodes and edges between nodes. The nodes include the targets detected in the second target detection results, and the edges represent the associations between nodes.

[0037] The information map of each frame of sensor data is used as the input of the spatial correlation perception model. The spatial correlation perception model uses a graph convolutional neural network to extract features from the information map, and the mapping layer uses the extracted features to perform a first correction prediction to obtain the first target detection result of each frame of sensor data.

[0038] Thirdly, a target detection device is provided, the device comprising:

[0039] The data acquisition module is configured to acquire N frames of sensor data to be detected, where N is a positive integer;

[0040] The target detection module is configured to perform target detection on each frame of sensor data using a deep convolutional neural network to obtain a second target detection result for each frame of sensor data.

[0041] The spatial correlation sensing module is configured to perform a first correction prediction on the second target detection results of each frame of sensor data based on the spatial correlation between each target in the second target detection results of the same frame of sensor data, so as to obtain the first target detection results of each frame of sensor data.

[0042] Fourthly, an apparatus for establishing a spatial correlation sensing network is provided, the apparatus comprising:

[0043] The first sample acquisition module is configured to acquire a first training sample, which includes multi-frame sensor data, a second target detection result obtained by target detection of the multi-frame sensor data, and labels annotating each target information in the multi-frame sensor data.

[0044] The first model training module is configured to train the spatial association perception model using the first training samples; wherein the spatial association perception model performs a first correction prediction on the second target detection results of each frame of sensor data based on the spatial association between each target in the second target detection results of each frame of sensor data, to obtain the first target detection results of each frame of sensor data; the training objective includes minimizing the difference between the first target detection results of each frame of sensor data and the corresponding label.

[0045] According to a fifth aspect, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in any one of the first or second aspects above.

[0046] According to a sixth aspect, an electronic device is provided, characterized in that it comprises:

[0047] One or more processors; and

[0048] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first or second aspects above.

[0049] According to the specific embodiments provided in this application, the following technical effects can be achieved:

[0050] 1) After performing target detection on each frame of sensor data, this application uses the spatial correlation of each target within the frame to make a first correction prediction on the target detection results of each frame of sensor data, thereby improving the accuracy of target detection.

[0051] 2) After performing target detection on each frame of sensor data, this application can further utilize the temporal correlation of the same target information between frames to perform a second correction prediction on the target detection results of each frame of sensor data, thereby further improving the accuracy of target detection.

[0052] 3) In the spatial association perception model, GCN uses a multi-layer nonlinear transformation to continuously update the features of the edges in multi-layer iterations, so that the features of the target learned in the end integrate the association relationship between nodes and have high computational efficiency.

[0053] 4) In the temporal association perception model, through the processing of the self-attention mechanism, the feature representation of each target takes into account the temporal association effect of the target, making the second correction prediction based on the feature representation of the target more accurate, and the learning process based on the self-attention mechanism more efficient.

[0054] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 An exemplary system architecture diagram is shown that can be applied to embodiments of this application;

[0057] Figure 2 A flowchart of the target detection method provided in the embodiments of this application;

[0058] Figure 3 A schematic diagram illustrating the first corrected prediction using a spatial relationship perception model, provided in an embodiment of this application;

[0059] Figure 4 A structural diagram of a time-related sensing model provided in an embodiment of this application;

[0060] Figure 5 A schematic diagram of a preferred process provided in an embodiment of this application is shown;

[0061] Figure 6 A flowchart illustrating the method for establishing a spatial association perception model provided in this application embodiment;

[0062] Figure 7 A flowchart illustrating the method for establishing a time-related perception model provided in this application embodiment;

[0063] Figure 8 A schematic block diagram of the target detection apparatus according to one embodiment is shown;

[0064] Figure 9 A schematic block diagram of the apparatus for establishing a spatial correlation perception model according to one embodiment is shown;

[0065] Figure 10 A schematic block diagram of an apparatus for establishing a time-related perception model according to one embodiment is shown;

[0066] Figure 11 This is an architectural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0068] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0069] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0070] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0071] Figure 1 An exemplary system architecture to which embodiments of this application can be applied is shown. For example... Figure 1 As shown, the system may include: sensors, a vehicle-side control system, and a server-side control system.

[0072] The sensors and vehicle-mounted control system are located on the vehicle itself. This vehicle can be an autonomous vehicle or a driver-assisted vehicle, etc.

[0073] Sensors may include, but are not limited to, radar, vision sensors, and far-infrared sensors. Radar may include, but is not limited to, lidar, millimeter-wave radar, and ultrasonic radar. Vision sensors may include, but are not limited to, cameras and webcams. Radar can acquire point cloud data for each frame, and vision sensors can acquire image data for each frame; both are suitable for the method provided in this application.

[0074] The vehicle-side control system can send sensor data (sensor data) collected by sensors to the server-side control system. The server-side control system then performs target detection using the method provided in this application and returns the target detection results to the vehicle-side control system, which generates specific decision information based on the target detection results. Alternatively, the server-side control system can perform target detection using the method provided in this application, generate specific decision information, and then return the decision information to the vehicle-side control system, which directly executes the decision information to control the vehicle. Alternatively, if the vehicle-side control system possesses powerful computing capabilities, it can also perform target detection and generate specific decision information using the method provided in this application to control the vehicle.

[0075] The server-side control system can be a single server, a server cluster consisting of multiple servers, or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPs) services, such as high management difficulty and weak service scalability.

[0076] It should be understood that Figure 1 The number of sensors, vehicle-side control systems, and server-side control systems shown is merely illustrative. Depending on implementation requirements, any number of sensors, vehicle-side control systems, and server-side control systems can be included.

[0077] Figure 2 This is a flowchart of a target detection method provided in an embodiment of this application. The method can be... Figure 1 The server-side control system in the system shown can execute the commands, but it can also be executed by a vehicle-side control system with strong computing capabilities. For example... Figure 2 As shown, the method may include the following steps:

[0078] Step 202: Obtain N frames of sensor data to be detected, where N is a positive integer.

[0079] Step 204: Perform target detection on each frame of sensor data using a deep convolutional neural network to obtain the second target detection result for each frame of sensor data.

[0080] Step 206: Based on the spatial correlation between targets in the second target detection results of the same frame of sensor data, perform a first correction prediction on the second target detection results of each frame of sensor data to obtain the first target detection results of each frame of sensor data.

[0081] As can be seen from the above process, after performing target detection on each frame of sensor data, this application uses the spatial correlation of each target within the frame to make a first correction prediction on the target detection results of each frame of sensor data, thereby improving the accuracy of target detection.

[0082] It should be noted that the terms "multiple frames" and "multiple" in this application refer to a quantity of more than one.

[0083] The steps in the above process will be described in detail below. First, step 202, namely "acquiring multiple frames of sensor data", will be described in detail with reference to the embodiment.

[0084] The sensor data acquired in this embodiment can be N frames of point cloud data collected by radar equipment such as lidar or millimeter-wave radar. N can be 1 or a number greater than 1, i.e., multiple frames. These multiple frames of point cloud data are typically collected continuously. Given that radar possesses high accuracy and precision, can provide autonomous vehicles with information about targets at a relatively long distance, and is not easily affected by shadows, sunlight, etc., multiple frames of point cloud data collected by radar equipment are preferred. The radar equipment can preferably be a multi-line rotating lidar or a solid-state lidar.

[0085] In addition, this application is also applicable to multi-frame image data acquired through a visual sensor, such as multi-frame image data continuously acquired by a camera.

[0086] It should be noted that the multi-frame sensor data acquired in this step is usually multi-frame sensor data collected by the sensor in a short period of time, with similar collection ranges and similar target information contained therein.

[0087] The following describes in detail step 204, namely, "to perform target detection on each frame of sensor data using a deep convolutional neural network and obtain the second target detection result of each frame of sensor data", with reference to the embodiments.

[0088] This step involves basic target detection of sensor data. Any target detection method in the existing technology can be used to perform target detection on each frame of sensor data. The resulting target detection result is called the second target detection result.

[0089] One possible approach is to first generate multiple candidate regions from each frame of sensor data, then use a deep convolutional neural network to extract features from each frame of sensor data, and finally use the extracted features to obtain the second target detection result for each frame of sensor data.

[0090] Specifically, multiple candidate regions can be generated for each frame of sensor data. For example, if the sensor data is image data, candidate regions for the image can be generated using techniques such as Selective Search. As another example, if the sensor data is point cloud data, candidate regions for the point cloud data can be generated by projecting the point cloud onto a raster coordinate system.

[0091] Then, networks such as Faster R-CNN (Fast Region Convolutional Neural Network), SSD (Single Shot MultiBox Detector, an end-to-end object detector based on convolutional neural networks), YOLO (a single-stage object detection model based on convolutional neural networks), and sparse convolutional neural networks are used to extract features from each candidate region. These features are then used for classification to obtain regions containing the target as bounding boxes. Further information about the target type, location (usually the center point of the bounding box), and size (usually the length, width, and height of the bounding box) is obtained for each bounding box. For point cloud data, the target's orientation information can also be further obtained.

[0092] Since this part of the basic target detection uses existing technology, it will not be described in detail here.

[0093] The following describes in detail step 206, namely, "based on the spatial correlation between targets in the second target detection result of the same frame of sensor data, perform a first correction prediction on the second target detection result of each frame of sensor data to obtain the first target detection result of each frame of sensor data", with reference to the embodiments.

[0094] In this step, the sensor data of each frame and the second target detection result are input into the spatial correlation perception model. The spatial correlation perception model uses the spatial correlation between different targets in the same frame to perform a first correction prediction on the second target detection result of the sensor data of each frame, so as to obtain the first target detection result of the sensor data of each frame.

[0095] like Figure 3 As shown, the implementation of the first correction prediction for the second target detection result of each frame of sensor data mainly involves the following two parts of processing:

[0096] The first part of the processing: using the sensor data of the current input frame (represented as F) i The second target detection result establishes an information graph of the current input frame sensor data. The information graph includes nodes and edges between nodes. Nodes include targets detected in the current input frame sensor data, and edges represent the relationships between nodes.

[0097] One possible approach is to use the second target detection result of the current input frame sensor data to determine the RoI (Region of Interest) of each target in the current input frame sensor data, use the RoI information of each target as the information of each node in the information graph, and use the positional relationship between the RoIs of each target to determine the edges in the information graph.

[0098] In determining each target RoI, the bounding boxes obtained from the second target detection results can be directly used as each RoI. However, as a preferred implementation, each bounding box can be expanded outward according to a preset ratio to obtain each RoI. For example, each bounding box can be expanded outward along the length, width, and height directions respectively. Obtain each RoI.

[0099] When constructing the infographic, each RoI (Royal Interest Area) is treated as a node. The features of each node can be obtained from the features of the area covered by the RoI. Since the RoI contains the bounding boxes from the second object detection results, the extended portion of the RoI can be obtained using bilinear interpolation based on the features of the bounding boxes. Edges between nodes in the infographic can be determined based on the positional relationships between RoIs. For example, if the distance between the center points of two RoIs is within a preset first distance range, then the two RoIs are considered related, and an edge exists between their corresponding nodes. Similarly, if the boundary distance between two RoIs is within a preset second distance range, then the two RoIs are considered related, and an edge exists between their corresponding nodes. Furthermore, for a given RoI, the three closest RoIs (the number 3 is a preset value, but other preset values ​​can also be used) can be identified as RoIs related to that RoI, and an edge exists between their corresponding nodes.

[0100] The second part of the processing: After the information graph is built, it is input into the spatial association perception model. The spatial association perception model uses GCN (Graph Convolutional Network) to extract features from the information graph, and the classification layer uses the extracted features to perform a first correction prediction to obtain the first target detection result of the sensor data in the current input frame.

[0101] GCN obtains edge feature vectors by performing a nonlinear transformation on the feature vectors of nodes in the information graph, and continuously updates the information of each edge in multiple iterations to ensure the extraction of spatial correlation information between targets. GCN uses the feature vectors of nodes and edges to obtain the feature vectors of each target, and then maps the target feature vectors through a mapping layer to predict the corrected target detection result, i.e., the first target detection result. The GCN and mapping layer described above constitute a spatial correlation perception model, where the mapping layer can include classification networks and / or regression networks, etc.

[0102] In the aforementioned GCN, a lightweight graph convolution design can be employed. This involves: firstly, in each iteration, performing a non-linear transformation on the features of the nodes at both ends of an edge to calculate the features of each edge in the information graph. For example, the following formula can be used:

[0103]

[0104] in, This represents the feature obtained by node m in the l-th iteration of the information graph. The superscript represents the feature obtained by node n in the l-th iteration of the information graph, where the superscript indicates the iteration number and the subscript indicates the node. Let F() represent the feature obtained in the l-th iteration of the edge between node m and node n. Let F() represent the nonlinear transformation function.

[0105] Then, by using max pooling, the edges connecting the nodes are merged to obtain the features of that node in the next iteration. For example, the following formula can be used:

[0106]

[0107] in, This represents the set of all nodes that have an edge with node m, and max represents the maximum value.

[0108] Finally, the features of the same node obtained from all iterations are merged to obtain the final features of that node. For example, the following formula can be used:

[0109]

[0110] in, This represents the characteristics of node m obtained by GCN, L is the maximum number of iterations, and Concat() represents the merging function.

[0111] The aforementioned GCN employs an L-layer nonlinear transformation to continuously update edge features across multiple iterations. This allows the final learned target features to incorporate the relationships between nodes while maintaining high computational efficiency. Furthermore, the overall computational efficiency can be improved by further controlling the number of convolutional kernels in a single nonlinear transformation layer. For example, the number of convolutional kernels in a single nonlinear transformation layer can be 64 or 32.

[0112] The aforementioned first target detection results can also include: the type information, location information (usually the center point of the detection box), and size information (usually the length, width, and height of the detection box) of the targets contained in each detection box. For point cloud data, the orientation information of the targets can also be further obtained.

[0113] It can be seen that GCN corrects the target detection results obtained from basic target detection based on the spatial correlation between each target in the target detection results, thereby achieving high-precision spatial feature correlation optimization and improving the accuracy of target detection.

[0114] To further improve the accuracy of target detection, in Figure 2 After step 206 shown, the following steps can be performed further:

[0115] Step 208: Perform target tracking on the first target detection results of each frame of sensor data to associate information of the same target and obtain a sequence of detection results for each target.

[0116] The target tracking in this step essentially involves identifying information belonging to the same target in each frame of sensor data and associating this information. For example, assuming there are T frames of sensor data, after associating the information of the same target in the first target detection results of each frame of sensor data, a detection result sequence is formed for each target. This detection result sequence contains information about that target in the first target detection results of the T frames of sensor data. Assuming there are P targets in each frame of sensor data, then for the j-th target, its corresponding detection result sequence S... j Represented as [s j，1 ,s j，2 ,…,s j,T Where T is a positive integer greater than 1, and P is a positive integer greater than or equal to 1.

[0117] Target tracking can employ any target tracking method, such as region matching, feature matching, or contour-based tracking. The most common method is feature matching, which uses the characteristics of a target in one frame of sensor data to match targets with the most similar features in other frames of sensor data, classifying them as the same target. Targets identified as the same target can be labeled with the same ID. Given the large number of existing target tracking technologies, they will not be detailed here.

[0118] Step 210: Based on the temporal correlation of the information of the same target in the detection result sequence of each target, perform a second correction prediction on the first target detection result of each frame of sensor data to obtain the third target detection result of each frame of sensor data.

[0119] This step is performed by a temporally correlated perception model. One possible approach is to input sensor data from each frame, along with the first target detection result and the target tracking result, into the temporally correlated perception model. The model utilizes the contextual target features of the current input frame sensor data to process the features of each target in the first target detection result of the current input frame sensor data using a self-attention mechanism, obtaining the feature representation of each target in the current input frame sensor data. It then uses these feature representations to map the third target detection result of the current input frame sensor data. This third target detection result can be used as the final target detection result.

[0120] Figure 4 This is a structural diagram of a time-related sensing model provided in an embodiment of this application. Figure 4 As shown, the time-related perception model can include an encoding layer, a self-attention layer, a feed-forward network layer, and a regression layer.

[0121] The coding layer is used to extract target features from each frame of input sensor data, thereby obtaining the features of each target in each frame of sensor data.

[0122] The self-attention layer is used to utilize the contextual target features of the current input frame sensor data to process the features of each target in the first target detection result of the current input frame sensor data using a self-attention mechanism, thereby obtaining the feature representation of each target in the current input frame sensor data.

[0123] The contextual target features of the current input frame sensor data can include the features of each target in the sensor data of the N frames preceding and / or following the current input frame sensor data, where N is a preset positive integer. Taking N=3 as an example, for the currently input i-th frame sensor data F... i For target j in the i-th frame, the features of target j in the first three frames and last three frames of sensor data can be used to process the features of target j in the i-th frame of sensor data using a self-attention mechanism, so as to obtain the feature representation of target j in the i-th frame of sensor data.

[0124] For the same target j, the feature sequence X in T frames of sensor data, X = [x j，1 ,x j，2 ,…,x j,T After processing by the self-attention mechanism, the output feature representation sequence of each target j is H, where H = [h j，1 ,h j，2 ,…,h j,T Then the following formula can be used:

[0125]

[0126] Where Q, K, and V are matrices composed of the query vector, key vector, and value vector, respectively, and d k K represents the spatial dimension.

[0127] Q = W q X

[0128] K = W k X

[0129] V = W v X

[0130] The above W q W k and W v These are the parameter matrices used to map X to the three vector spaces, and the model parameters learned by the temporal correlation perceptual network.

[0131] By employing a self-attention mechanism, the feature representation of each target considers its temporal correlation, leading to more accurate subsequent predictions based on the target's feature representation. Besides self-attention, a conventional attention mechanism can also be used, but it will not be detailed here. However, due to the efficient learning advantages of self-attention, it is the preferred choice.

[0132] Feedforward network layers are used to map the feature representations of each target to a unified vector space.

[0133] The regression layer is used to predict the third target detection result from the sensor data of the current input frame by utilizing the feature representations of each target mapped to a unified vector space. This third target detection result can also include: the type information, location information (usually the center point of the detection box), and size information (usually the length, width, and height of the detection box) of the targets contained in each detection box. For point cloud data, the orientation information of the targets can also be further obtained.

[0134] As can be seen, the temporal correlation sensing model essentially uses the temporal correlation of the same target between frames to correct and predict the first target detection result of each frame of sensor data, and obtains the final target detection result of each frame of sensor data.

[0135] Since the detection result sequences of each target obtained in step 208 have corresponding confidence levels, sequences with high confidence levels are considered to belong to the same target with a high probability, while sequences with low confidence levels are considered to belong to the same target with a low probability. Therefore, as one possible approach, the aforementioned time-related perception module can only perform corrective predictions on the first detection result of the target corresponding to the detection result sequence with a confidence level greater than or equal to a preset confidence threshold.

[0136] Specifically, targets corresponding to detection result sequences with confidence levels greater than or equal to a preset confidence threshold can be identified as targets to be corrected. Based on the temporal correlation reflected in the detection result sequences of the targets to be corrected, a second correction prediction is performed on the information of the targets to be corrected in the first target detection results of each frame of sensor data. This targeted correction prediction can reduce computational complexity and improve computational efficiency.

[0137] Figure 5 The diagram illustrates a preferred process provided in an embodiment of this application, as shown below. Figure 5As shown in the figure, for the laser point cloud data collected by the LiDAR of an autonomous vehicle, each frame of point cloud data F i First, the basic target detection is used to obtain the second target detection result, which includes information about each target, such as the target's type, size, location, and orientation.

[0138] Point cloud data F of each frame i After the detection results of the second target are constructed into an information graph, the spatial correlation perception model performs a first correction prediction using the spatial correlation between targets within the frame, thus obtaining the point cloud data F for each frame. i The first target detection result includes information about each target, such as its type, size, location, and orientation. This target information is a revised and more accurate version.

[0139] Point cloud data F of each frame i After target tracking, the first target detection result is used to correlate the information of the same target in each frame of point cloud data to obtain the detection result sequence of each target.

[0140] Point cloud data F of each frame i The first target detection result and the detection result sequence of each target are input into the temporal correlation sensing model. After the temporal correlation sensing model performs a second correction prediction using the temporal correlation of the same target between frames, the point cloud data F of each frame is obtained. i The third target detection result also includes information about each target, such as its type, size, location, and orientation. This third target detection result serves as the final target detection result.

[0141] The basic target detection and target tracking used in the above target detection process can be implemented using existing technologies, while the spatial correlation sensing model and the temporal correlation sensing model need to be pre-trained. The training methods for the spatial correlation sensing model and the temporal correlation sensing model are described in detail below.

[0142] Figure 6 A flowchart illustrating the method for establishing a spatial association perception model provided in this application embodiment is shown below. Figure 6 As shown, the method may include the following steps:

[0143] Step 602: Obtain the first training sample, which includes multi-frame sensor data, the second target detection result obtained by target detection of multi-frame sensor data, and the labels of each target information in multi-frame sensor data.

[0144] When training a spatial association perception model, multiple frames of sensor data can be acquired by sensors capturing images of known targets. Since the target information is known, such as the target type, location, size, and orientation, this known target information can be used to annotate the multiple frames of sensor data.

[0145] Then, target detection is performed on multiple frames of sensor data using basic target detection to obtain the second target detection result. For details on basic target detection, please refer to [link to relevant documentation]. Figure 2 The details regarding step 204 in the illustrated method embodiment will not be repeated here.

[0146] Step 604: Train a spatial correlation perception model using the first training samples; wherein the spatial correlation perception model performs a first correction prediction on the second target detection results of each frame of sensor data based on the spatial correlation between each target in the second target detection results of each frame of sensor data, and obtains the first target detection results of each frame of sensor data; the training objective includes: minimizing the difference between the first target detection results of each frame of sensor data and the corresponding label.

[0147] Specifically, an information graph of each frame of sensor data can be first constructed using the second target detection results of each frame. This information graph includes nodes and edges between nodes, where nodes include targets detected in the second target detection results, and edges represent the relationships between nodes. Then, the information graph of each frame of sensor data is used as input to a spatial association sensing model. The spatial association sensing model uses a Geometric Node (GCN) to extract features from the information graph, and the mapping layer uses the extracted features to perform a first correction prediction to obtain the first target detection result of each frame of sensor data.

[0148] In constructing the information graph, the RoIs of each target in each frame of sensor data can be determined using the second target detection results. For example, the detection boxes obtained from the second target detection results can be directly used as RoIs. However, in a preferred embodiment, each detection box can be expanded outward according to a preset ratio to obtain each RoI. Then, the RoI information of each target is used as the information of each node in the information graph, and the edges in the information graph are determined using the positional relationships between the RoIs of each target. For example, if the distance between the center points of two RoIs is within a preset first distance range, then the two RoIs are considered to be related, and there is an edge between the corresponding nodes. Similarly, if the boundary distance between two RoIs is within a preset second distance range, then the two RoIs are considered to be related, and there is an edge between the corresponding nodes. Furthermore, for a single RoI, the three nearest RoIs can be identified as RoIs related to that RoI, and there is an edge between the corresponding nodes.

[0149] Spatial association perception models can include Geographic Networks (GCNs) and mapping layers. GCNs obtain edge feature vectors by performing a nonlinear transformation on the feature vectors of nodes in the information graph, and continuously update the information of each edge through multiple iterations, ensuring the extraction of spatial association information between targets. GCNs use the feature vectors of nodes and edges to obtain the feature vectors of each target, and then use a mapping layer to map the target feature vectors, thereby predicting the corrected target detection result, i.e., the first target detection result. The mapping layer can include classification networks and / or regression networks, etc. See [link to relevant documentation] for details. Figure 3 The relevant details in the illustrated embodiments will not be repeated here.

[0150] During training, the spatial association sensing model uses the labels in the first training sample as the output target. In other words, the training objective is to minimize the difference between the first target detection result and the corresponding label in each frame of sensor data output by the spatial association sensing model. In the embodiments of this specification, a loss function can be constructed based on the above training objective. In each iteration, the model parameters are updated using methods such as gradient descent, based on the value of the loss function, until a preset training termination condition is met. The training termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0151] Figure 7 A flowchart illustrating the method for establishing a time-related perception model provided in this application embodiment is shown below. Figure 7 As shown, the method may include the following steps:

[0152] Step 702: Obtain the second training sample, which includes multi-frame sensor data, the first target detection result obtained by target detection of multi-frame sensor data, the target detection result sequence obtained by target tracking of the first target detection result of each frame sensor data, and the labels of each target information in the multi-frame sensor data.

[0153] When training a temporal correlation perception model, multiple frames of sensor data can be acquired by the sensor collecting data on a known target. Since the target information is known, such as the target type, location, size, and orientation, this known target information can be used to annotate the multiple frames of sensor data.

[0154] Then, target detection is performed on multiple frames of sensor data using basic target detection to obtain the first target detection result. For more information on basic target detection, please refer to [link to relevant documentation]. Figure 2 The details regarding step 204 in the illustrated method embodiment will not be repeated here.

[0155] Alternatively, a second target detection result can be obtained by performing target detection on multiple frames of sensor data using basic target detection. Then, a first correction prediction can be made on the second target detection results of each frame of sensor data using a pre-trained spatial correlation sensing model to obtain the first target detection result of each frame of sensor data. For more information on spatial correlation sensing models, please refer to [link to relevant documentation]. Figure 2 The method embodiment shown includes relevant descriptions of step 206 and... Figure 3 The embodiments shown are not described in detail here.

[0156] The first target detection results from each frame of sensor data obtained above can be used to correlate information about the same target using target tracking methods, resulting in a sequence of detection results for each target. This section can be found in [link to relevant documentation]. Figure 2 The details regarding step 208 in the illustrated method embodiment will not be repeated here.

[0157] This yields multiple frames of sensor data, the first target detection results from the multiple frames of sensor data, the detection result sequence for each target, and the labels for each target information in the multiple frames of sensor data, thus forming the second training sample.

[0158] Step 704: Train a time-related perception model using the second training samples; wherein the time-related perception model performs a second correction prediction on the first target detection result of each frame of sensor data based on the temporal correlation of the information of the same target in the detection result sequence of each target, and obtains the third target detection result of each frame of sensor data; the training objective includes: minimizing the difference between the third target detection result of each frame of sensor data and the corresponding label.

[0159] During training, sensor data from each frame and its first target detection result are input into the temporal association perception model. The temporal association perception model uses the contextual target features of the current input frame sensor data to process the features of each target in the first target detection result of the current input frame sensor data using a self-attention mechanism, thereby obtaining the feature representation of each target in the current input frame sensor data. The contextual target features of the current input frame sensor data include the features of each target in the N frames of sensor data before and / or after the current input frame sensor data, where N is a preset positive integer. The third target detection result of the current input frame sensor data is obtained by mapping the feature representation of each target in the current input frame sensor data.

[0160] For the specific structure of the time-related perception model, please refer to [link / reference]. Figure 4 The relevant details in the illustrated embodiments will not be repeated here.

[0161] The temporal correlation sensing model uses the labels in the second training samples as the output target. In other words, the training objective is to minimize the difference between the third target detection result and the corresponding label in each frame of sensor data output by the temporal correlation sensing model. In the embodiments of this specification, a loss function can be constructed based on the above training objective. In each iteration, the model parameters are updated using methods such as gradient descent, based on the value of the loss function, until a preset training termination condition is met. The training termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0162] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0163] According to another embodiment, a target detection device is provided. Figure 8 A schematic block diagram of the target detection apparatus according to one embodiment is shown, the apparatus being disposed in... Figure 1 The server-side control system in the illustrated architecture can be an application located within the server-side control system, or it can be a plugin or software development kit (SDK) functional unit within an application located within the server-side control system, or it can be located within the vehicle-side control system. This embodiment of the invention does not impose any particular limitation on this. Figure 8 As shown, the device 800 may include: a data acquisition module 801, a target detection module 802, and a spatial correlation sensing module 803, and may further include a target tracking module 804 and a temporal correlation sensing module 805. The main functions of each component module are as follows:

[0164] The data acquisition module 801 is configured to acquire multiple frames of sensor data.

[0165] The target detection module 802 is configured to perform target detection on each frame of sensor data using a deep convolutional neural network to obtain a second target detection result for each frame of sensor data.

[0166] The spatial association perception module 803 is configured to perform a first correction prediction on the second target detection results of each frame of sensor data based on the spatial association between each target in the second target detection results of the same frame of sensor data, so as to obtain the first target detection results of each frame of sensor data.

[0167] As one possible implementation method, the target detection module 802 can be specifically configured to: generate multiple candidate regions for each frame of sensor data in the multi-frame sensor data; extract features of the candidate regions for each frame of sensor data through a deep convolutional neural network; and use the extracted features to obtain the second target detection result of each frame of sensor data.

[0168] As one possible implementation method, the spatial association perception module 803 can be specifically configured to: take each frame of sensor data as the current input frame sensor data, and perform the following: establish an information graph of the current input frame sensor data using the second target detection result of the current input frame sensor data and input it into the spatial association perception model, wherein the information graph includes nodes and edges between nodes, the nodes include targets detected in the current input frame sensor data, and the edges represent the association between nodes; the spatial association perception model uses a graph convolutional neural network to extract features from the information graph of the current input frame sensor data, and the mapping layer uses the extracted features to perform a first correction prediction to obtain the first target detection result of the current input frame sensor data.

[0169] As one possible implementation method, the spatial association perception module 803 can be specifically configured to: determine the region of interest (RoI) of each target in the current input frame sensor data using the second target detection result of the current input frame sensor data, use the RoI information of each target as the information of each node in the information graph, and determine the edges in the information graph using the positional relationship between the RoIs of each target.

[0170] As one possible approach, GCN calculates the features of each edge in the information graph by performing nonlinear transformations on the features of the nodes at both ends of the edge in each iteration, and merges the edges connected to the nodes through pooling to obtain the features of the node in the next iteration. After the iteration ends, the features of the same node obtained in all iterations are merged to obtain the features of the node extracted from the information graph and provided to the mapping layer.

[0171] The target tracking module 804 is configured to perform target tracking on the first target detection results of each frame of sensor data to associate information of the same target and obtain a sequence of detection results for each target.

[0172] The temporal correlation perception module 805 is configured to perform a second correction prediction on the first target detection result of each frame of sensor data based on the temporal correlation of the information of the same target in the detection result sequence of each target, so as to obtain the third target detection result of each frame of sensor data.

[0173] As one possible implementation method, the temporal correlation perception module 805 can be specifically configured to: input sensor data of each frame and its first target detection result to the temporal correlation perception model respectively; the temporal correlation perception model uses the contextual target features of the current input frame sensor data to process the features of each target in the first target detection result of the current input frame sensor data using a self-attention mechanism to obtain the feature representation of each target in the current input frame sensor data, wherein the contextual target features of the current input frame sensor data include the features of each target in the N frames of sensor data before and / or after the current input frame sensor data, where N is a preset positive integer; and use the feature representation of each target in the current input frame sensor data to map and obtain the third target detection result of the current input frame sensor data.

[0174] Furthermore, the target tracking module 804 also obtains the confidence level of the detection result sequence of each target during target tracking.

[0175] The temporal correlation perception module 805 can determine that the target corresponding to the detection result sequence with a confidence level greater than or equal to a preset confidence threshold is the target to be corrected; based on the temporal correlation reflected in the detection result sequence of the target to be corrected, a second correction prediction is performed on the information of the target to be corrected in the first target detection result of each frame of sensor data.

[0176] As one typical application scenario, the aforementioned sensor data can include point cloud data, that is, point cloud data collected by radar devices such as lidar and millimeter-wave radar. The target detection results include the target's category information, location information, size information, and orientation information.

[0177] According to another embodiment, an apparatus for establishing a spatial association perception model is provided. Figure 9 A schematic block diagram of an apparatus for establishing a spatial correlation perception model is shown according to one embodiment. Figure 9 As shown, the device 900 may include: a first sample acquisition module 901 and a first model training module 902. Each component module may have the following functions:

[0178] The first sample acquisition module 901 is configured to acquire a first training sample, which includes multi-frame sensor data, a second target detection result obtained by target detection of the multi-frame sensor data, and labels annotating each target information in the multi-frame sensor data.

[0179] The first sample acquisition module 901 can acquire multiple frames of sensor data obtained by the sensor from collecting data on a known target. Since the target information is known, such as the target type, location, size, and orientation, the multi-frame sensor data can be labeled using the known target information.

[0180] Then, target detection is performed on multiple frames of sensor data using basic target detection to obtain the second target detection result. For details on basic target detection, please refer to [link to relevant documentation]. Figure 2 The details regarding step 204 in the illustrated method embodiment will not be repeated here.

[0181] The first model training module 902 is configured to train the spatial association perception model using the first training samples; wherein the spatial association perception model performs a first correction prediction on the second target detection results of each frame of sensor data based on the spatial association between each target in the second target detection results of each frame of sensor data, to obtain the first target detection results of each frame of sensor data; the training objective includes minimizing the difference between the first target detection results of each frame of sensor data and the corresponding label.

[0182] As one possible approach, the first model training module 902 can first construct an information graph of each frame of sensor data using the second target detection results of each frame. This information graph includes nodes and edges between nodes, where nodes include targets detected in the second target detection results, and edges represent the relationships between nodes. Then, the information graph of each frame of sensor data is used as input to a spatial association perception model. The spatial association perception model uses GCN to extract features from the information graph, and the mapping layer uses the extracted features to perform a first correction prediction to obtain the first target detection results of each frame of sensor data.

[0183] In constructing the information graph, the RoI (Royalty Area) of each target in each frame of sensor data can be determined using the second target detection results. For example, the detection boxes obtained from the second target detection results can be directly used as RoIs. However, in a preferred embodiment, each detection box can be expanded outward according to a preset ratio to obtain each RoI. Then, the RoI information of each target is used as the information of each node in the information graph, and the edges in the information graph are determined using the positional relationships between the RoIs of each target.

[0184] Spatial association perception models can include Geographic Networks (GCNs) and mapping layers. GCNs obtain edge feature vectors by performing a nonlinear transformation on the feature vectors of nodes in the information graph, and continuously update the information of each edge through multiple iterations, ensuring the extraction of spatial association information between targets. GCNs use the feature vectors of nodes and edges to obtain the feature vectors of each target, and then use a mapping layer to map the target feature vectors, thereby predicting the corrected target detection result, i.e., the first target detection result. The mapping layer can include classification networks and / or regression networks, etc. See [link to relevant documentation] for details. Figure 3 The relevant details in the illustrated embodiments will not be repeated here.

[0185] During training, the spatial association sensing model uses the labels in the first training sample as the output target. In other words, the training objective is to minimize the difference between the first target detection result and the corresponding label in each frame of sensor data output by the spatial association sensing model. In the embodiments of this specification, a loss function can be constructed based on the above training objective. In each iteration, the model parameters are updated using methods such as gradient descent, based on the value of the loss function, until a preset training termination condition is met. The training termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0186] According to another embodiment, an apparatus for establishing a time-related perception model is provided. Figure 10 A schematic block diagram of an apparatus for establishing a time-related perception model according to one embodiment is shown. Figure 10 As shown, the device 1000 may include: a second sample acquisition module 1001 and a second model training module 1002. Each component module may have the following functions:

[0187] The second sample acquisition module 1001 is configured to acquire a second training sample. The second training sample includes multi-frame sensor data, a first target detection result obtained by performing target detection on the multi-frame sensor data, a sequence of target detection results obtained by performing target tracking on the first target detection results of each frame of sensor data, and labels annotating the target information in the multi-frame sensor data.

[0188] The second sample acquisition module 1001 can acquire multiple frames of sensor data obtained by the sensor from collecting data on a known target. Since the target information is known, such as the target type, location, size, and orientation, the known target information can be used to annotate the multiple frames of sensor data.

[0189] Then, obtain the first target detection result obtained by performing target detection on multi-frame sensor data using basic target detection. (See the section on basic target detection for more details.) Figure 2 The details regarding step 204 in the illustrated method embodiment will not be repeated here.

[0190] Alternatively, a second target detection result can be obtained by performing target detection on multiple frames of sensor data using basic target detection. Then, a first correction prediction can be made on the second target detection results of each frame of sensor data using a pre-trained spatial correlation sensing model to obtain the first target detection result of each frame of sensor data. For more information on spatial correlation sensing models, please refer to [link to relevant documentation]. Figure 2 The method embodiment shown includes relevant descriptions of step 206 and... Figure 3 The embodiments shown are not described in detail here.

[0191] The first target detection results from each frame of sensor data obtained above can be used to correlate information about the same target using target tracking methods, resulting in a sequence of detection results for each target. This section can be found in [link to relevant documentation]. Figure 2 The details regarding step 208 in the illustrated method embodiment will not be repeated here.

[0192] The second model training module 1002 is configured to train a temporal correlation sensing network using the second training samples; wherein the temporal correlation sensing network performs a second correction prediction on the first target detection result of each frame of sensor data based on the temporal correlation of information of the same target in the detection result sequence of each target, to obtain the third target detection result of each frame of sensor data; the training objective includes minimizing the difference between the third target detection result of each frame of sensor data and the corresponding label.

[0193] During training, the second model training module 1002 inputs sensor data from each frame and the first target detection result to the temporal correlation perception model. The temporal correlation perception model uses the contextual target features of the current input frame sensor data to process the features of each target in the first target detection result of the current input frame sensor data using a self-attention mechanism, thereby obtaining the feature representation of each target in the current input frame sensor data. The contextual target features of the current input frame sensor data include the features of each target in the N frames of sensor data before and / or after the current input frame sensor data, where N is a preset positive integer. The third target detection result of the current input frame sensor data is obtained by mapping the feature representation of each target in the current input frame sensor data.

[0194] For the specific structure of the time-related perception model, please refer to [link / reference]. Figure 4 The relevant details in the illustrated embodiments will not be repeated here.

[0195] The temporal correlation sensing model uses the labels in the second training samples as the output target. In other words, the training objective is to minimize the difference between the third target detection result and the corresponding label in each frame of sensor data output by the temporal correlation sensing model. In the embodiments of this specification, a loss function can be constructed based on the above training objective. In each iteration, the model parameters are updated using methods such as gradient descent, based on the value of the loss function, until a preset training termination condition is met. The training termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0196] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0197] And an electronic device, comprising:

[0198] One or more processors; and

[0199] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0200] in, Figure 11 The architecture of an electronic device is illustrated, which may include a processor 1110, a video display adapter 1111, a disk drive 1112, an input / output interface 1113, a network interface 1114, and a memory 1120. The processor 1110, video display adapter 1111, disk drive 1112, input / output interface 1113, network interface 1114, and memory 1120 can communicate with each other via a communication bus 1130.

[0201] The processor 1110 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0202] The memory 1120 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1120 can store the operating system 1121 for controlling the operation of the electronic device 1100, and the basic input / output system (BIOS) 1122 for controlling the low-level operations of the electronic device 1100. Additionally, it can store a web browser 1123, a data storage management system 1124, and a target detection device 1125, etc. The aforementioned target detection device 1125 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 1120 and is called and executed by the processor 1110.

[0203] Input / output interface 1113 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0204] Network interface 1114 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0205] Bus 1130 includes a pathway for transmitting information between various components of the device, such as processor 1110, video display adapter 1111, disk drive 1112, input / output interface 1113, network interface 1114, and memory 1120.

[0206] It should be noted that although the above-described device only shows the processor 1110, video display adapter 1111, disk drive 1112, input / output interface 1113, network interface 1114, memory 1120, bus 1130, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0207] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0208] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0209] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A target detection method, characterized in that, The method includes: Obtain N frames of sensor data to be detected, where N is a positive integer; For each frame of sensor data in the N frames of sensor data, a corresponding candidate region is generated. The feature of the candidate region is extracted from each frame of sensor data using a deep convolutional neural network. The second target detection result of each frame of sensor data is obtained by using the extracted features of each candidate region. The sensor data of each frame and the corresponding second target detection result are input into the spatial association perception model, so that the spatial association perception model performs a first correction prediction on the second target detection result of each frame of sensor data based on the spatial association between different targets in the second target detection result of the same frame of sensor data, so as to obtain the first target detection result of each frame of sensor data.

2. The method according to claim 1, characterized in that, The spatial association sensing model, based on the spatial association between different targets in the second target detection results of the same frame of sensor data, performs a first correction prediction on the second target detection results of each frame of sensor data, including: Each frame of sensor data is used as the current input frame sensor data, and the following steps are executed: Using the second target detection result of the current input frame sensor data, an information graph of the current input frame sensor data is established and input into the spatial association perception model. The information graph includes nodes and edges between nodes. The nodes include targets detected in the current input frame sensor data, and the edges represent the associations between nodes. The spatial correlation perception model uses a graph convolutional neural network to extract features from the information graph of the current input frame sensor data, and the mapping layer uses the extracted features to perform a first correction prediction to obtain the first target detection result of the current input frame sensor data.

3. The method according to claim 2, characterized in that, The step of establishing an information map of the current input frame sensor data using the second target detection result of the current input frame sensor data includes: The region of interest (RoI) of each target in the current input frame sensor data is determined using the second target detection result of the current input frame sensor data. The RoI information of each target is used as the information of each node in the information graph. The edges in the information graph are determined using the positional relationship between the RoIs of each target.

4. The method according to claim 2, characterized in that, The feature extraction of the information graph of the current input frame sensor data using a graph convolutional neural network includes: In each iteration, the features of each edge in the information graph are calculated by performing nonlinear transformation on the features of the nodes at both ends of the edge. The edges connected to the nodes are then merged using pooling to obtain the features of the node in the next iteration. After the iteration is completed, the features of the same node obtained in all iterations are merged to obtain the features of that node extracted from the information graph and provided to the mapping layer.

5. The method according to claim 1, characterized in that, The N is greater than 1; the method further includes: Target tracking is performed on the first target detection results of each frame of sensor data to associate information of the same target, thereby obtaining a sequence of detection results for each target; Based on the temporal correlation of information about the same target in the detection result sequence of each target, a second correction prediction is performed on the first target detection result of each frame of sensor data to obtain the third target detection result of each frame of sensor data.

6. The method according to claim 5, characterized in that, Based on the temporal correlation of information about the same target in the detection result sequence of each target, the second correction prediction of the first target detection result of each frame of sensor data includes: Input the sensor data of each frame and the detection result of the first target into the time-related perception model respectively; The temporal correlation perception model utilizes the contextual target features of the current input frame sensor data to process the features of each target in the first target detection result of the current input frame sensor data using a self-attention mechanism, thereby obtaining the feature representation of each target in the current input frame sensor data. The contextual target features of the current input frame sensor data include the features of each target in the sensor data before and / or after the current input frame sensor data, where N is a preset positive integer. The third target detection result of the current input frame sensor data is obtained by mapping the feature representation of each target in the current input frame sensor data.

7. The method according to any one of claims 1 to 6, characterized in that, The sensor data includes point cloud data; The target detection results include the target's category information, location information, size information, and orientation information.

8. A method for establishing a spatial correlation perception model, characterized in that, The method includes: A first training sample is obtained, which includes multiple frames of sensor data, a second target detection result obtained by target detection of the multiple frames of sensor data, and labels annotating each target information in the multiple frames of sensor data; the second target detection result includes: generating corresponding candidate regions for each frame of sensor data in the multiple frames of sensor data, extracting features of the candidate regions of each frame of sensor data through a deep convolutional neural network, and obtaining the target detection result of each frame of sensor data using the extracted features of each candidate region; The spatial association perception model is obtained by training the first training sample. The spatial association perception model is based on the spatial association between different targets in the second target detection results of each frame of sensor data, and performs a first correction prediction on the second target detection results of each frame of sensor data to obtain the first target detection results of each frame of sensor data. The training objective includes minimizing the difference between the first target detection result and the corresponding label in each frame of sensor data.

9. The method according to claim 8, characterized in that, The spatial association perception model is trained using the first training data, including: An information graph of each frame of sensor data is established using the second target detection results of each frame of sensor data. The information graph includes nodes and edges between nodes. The nodes include the targets detected in the second target detection results, and the edges represent the associations between nodes. The information map of each frame of sensor data is used as the input of the spatial correlation perception model. The spatial correlation perception model uses a graph convolutional neural network to extract features from the information map, and the mapping layer uses the extracted features to perform a first correction prediction to obtain the first target detection result of each frame of sensor data.

10. A target detection device, characterized in that, The device includes: The data acquisition module is configured to acquire N frames of sensor data to be detected, where N is a positive integer; The target detection module is configured to generate corresponding candidate regions for each frame of sensor data in the N frames of sensor data, extract features of the candidate regions for each frame of sensor data through a deep convolutional neural network, and obtain the second target detection result of each frame of sensor data using the extracted features of each candidate region. The spatial correlation sensing module is configured to input the sensor data of each frame and the corresponding second target detection result into the spatial correlation sensing model, so that the spatial correlation sensing model performs a first correction prediction on the second target detection result of each frame of sensor data based on the spatial correlation between different targets in the second target detection result of the same frame of sensor data, and obtains the first target detection result of each frame of sensor data.

11. A device for establishing a spatial correlation sensing network, characterized in that, The device includes: The first sample acquisition module is configured to acquire a first training sample, which includes multiple frames of sensor data, a second target detection result obtained by target detection of the multiple frames of sensor data, and labels annotating each target information in the multiple frames of sensor data; the second target detection result includes: generating corresponding candidate regions for each frame of sensor data in the multiple frames of sensor data, extracting features of the candidate regions of each frame of sensor data through a deep convolutional neural network, and obtaining the target detection result of each frame of sensor data using the extracted features of each candidate region; The first model training module is configured to train the spatial association perception model using the first training samples; wherein the spatial association perception model performs a first correction prediction on the second target detection results of each frame of sensor data based on the spatial association between different targets in the second target detection results of each frame of sensor data, to obtain the first target detection results of each frame of sensor data; the training objective includes minimizing the difference between the first target detection results of each frame of sensor data and the corresponding label.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 9.

13. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 9.

Images