Multi-target obstacle tracking method and autonomous vehicle

Abstract

The invention relates to a multi-target obstacle tracking method and an autonomous vehicle. The method comprises the steps that a self-attention mechanism encoder is used for encoding information of candidate obstacles and historical association result information, and first interaction information between the candidate obstacles and second interaction information between association relations are obtained; decoding the first interaction information and the second interaction information by using an attention mechanism decoder to obtain a similarity calculation result between each association relationship and each candidate obstacle; processing the similarity calculation result based on a preset matching algorithm to obtain a matching result between each candidate obstacle and each association relationship; and associating the candidate obstacles with the historical obstacles corresponding to the matched association relationship to obtain an updated obstacle association relationship, the updated obstacle association relationship being used for determining a tracking result of the multi-target obstacle. By adopting the method, the multi-target obstacle tracking effect can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a multi-target obstacle tracking method and an autonomous driving vehicle. Background Art

[0002] With the development of artificial intelligence, multi-object tracking has been applied in many fields, such as tracking obstacles around vehicles in the field of autonomous driving, and tracking and analyzing pedestrians in surveillance cameras in the field of security. One of the goals of multi-object tracking is to associate objects in multiple frames of a video.

[0003] In the related art, there are multi-target obstacle tracking methods based on machine learning models of rules and manually designed features. However, these methods have problems of difficult convergence and poor association effect. Summary of the invention

[0004] The purpose of the present disclosure is to provide a multi-target obstacle tracking method, device, storage medium and autonomous driving vehicle to solve the problems existing in the related art.

[0005] In order to achieve the above-mentioned object, the first part of the embodiment of the present disclosure provides a multi-target obstacle tracking method, the method comprising:

[0006] Encode each candidate obstacle information in the current frame sensor result by using a self-attention mechanism encoder to obtain first interaction information between the candidate obstacles;

[0007] Encoding the historical association result information corresponding to the current frame sensor result using a self-attention mechanism encoder to obtain second interactive information between the association relationships, wherein the historical association result information represents the association relationship between historical obstacles in the historical frame sensor results before the current frame sensor result on the historical frame sensor results;

[0008] Decoding the first interaction information and the second interaction information using an attention mechanism decoder to obtain a decoding result, wherein the decoding result includes a similarity calculation result between each of the association relationships and each of the candidate obstacles;

[0009] Processing the similarity calculation result based on a preset matching algorithm to obtain a matching result between each candidate obstacle and each association relationship;

[0010] The candidate obstacles are associated with the historical obstacles corresponding to the matching association relationships to obtain updated obstacle association relationships, and the updated obstacle association relationships are used to determine the tracking results of the multi-target obstacles.

[0011] Optionally, the decoding result further includes third interaction information between each of the association relationships and each of the candidate obstacles, wherein the third interaction information serves as historical association result information corresponding to a next frame sensor result of the current frame sensor result.

[0012] Optionally, the method further comprises:

[0013] The second interaction information is processed by using a pre-trained state prediction model to obtain state information of each candidate obstacle in the current frame sensor result, wherein the state prediction model is trained by multiple groups of sample data including the second interaction information and state information labels of the candidate obstacles, and the state information of each candidate obstacle in the current frame sensor result is used to determine each candidate obstacle information in the current frame sensor result.

[0014] Optionally, the decoding result includes the similarity calculation result and third interaction information between each of the association relationships and each of the candidate obstacles, and the use of the attention mechanism decoder to decode the first interaction information and the second interaction information to obtain a decoding result includes:

[0015] Processing the first interaction information based on the attention mechanism decoder to obtain a key matrix and a value matrix;

[0016] Processing the second interaction information based on the attention mechanism decoder to obtain a query matrix;

[0017] Based on the key matrix and the query matrix, obtaining the similarity calculation result;

[0018] The third interaction information is obtained based on the similarity calculation result and the value matrix.

[0019] Optionally, the method further comprises:

[0020] An association relationship corresponding to a target candidate obstacle is created, wherein the target candidate obstacle is a candidate obstacle for which there is no matching association relationship.

[0021] Optionally, the creating an association relationship corresponding to the target candidate obstacle includes:

[0022] In the absence of historical association result information corresponding to the current frame sensor result, initial association result information is created based on each candidate obstacle information in the current frame sensor result. The initial association result information serves as the historical association result information corresponding to the next frame sensor result of the current frame sensor result. The initial association result information includes initial association relationships corresponding to each candidate obstacle.

[0023] Optionally, the creating an association relationship corresponding to the target candidate obstacle includes:

[0024] In the case where it is determined according to the matching result that there is no target candidate obstacle with a matching association relationship, an initial association relationship corresponding to the target candidate obstacle is created according to the target candidate obstacle information.

[0025] Optionally, the method further comprises:

[0026] For any obstacle association relationship, if it is determined according to the obstacle association relationship that there is no candidate obstacle associated with the target obstacle in the obstacle association relationship in the most recent preset number of frame sensor results, the obstacle association relationship is deleted.

[0027] Optionally, the processing of the similarity calculation result based on a preset matching algorithm to obtain a matching result between each candidate obstacle and each association relationship includes:

[0028] The similarity calculation result is used as an input of a differentiable Hungarian matching algorithm to obtain a matching result between each candidate obstacle and each association relationship output by the differentiable Hungarian matching algorithm.

[0029] Optionally, each candidate obstacle information is candidate obstacle information in the current frame sensor result collected by a preset number of sensors, the attention mechanism decoder includes the preset number of sub-attention mechanism decoders stacked in sequence, and the self-attention mechanism encoder is used to encode each candidate obstacle information to obtain the first interaction information between each candidate obstacle, including:

[0030] Encode the candidate obstacle information corresponding to each sensor by using the self-attention mechanism encoder corresponding to each sensor, and obtain the first interaction information between the candidate obstacles corresponding to each sensor;

[0031] The using the attention mechanism decoder to decode the first interaction information and the second interaction information to obtain a decoding result includes:

[0032] Using the M-th sub-attention mechanism decoder to process the corresponding first input parameter and the corresponding second input parameter to obtain a decoding result output by the M-th sub-attention mechanism, wherein the decoding result includes a similarity calculation result and the third interaction information;

[0033] Determine the similarity calculation result included in the decoding result output by the last sub-attention mechanism decoder stacked in sequence as the similarity calculation result between each of the association relationships and each of the candidate obstacles;

[0034] Among them, when M is equal to 1, the corresponding first input parameter is the second interaction information, and the corresponding second input parameter is the first interaction information between the candidate obstacles corresponding to the first sensor. When M is a positive integer greater than 1, the corresponding first input parameter is the third interaction information included in the decoding result output by the M-1th sub-attention mechanism decoder, and the corresponding second input parameter is the first interaction information between the candidate obstacles corresponding to the Mth sensor.

[0035] Optionally, the method further comprises:

[0036] Obtaining detection result data of each candidate obstacle in the current frame sensor result;

[0037] Using a feature extraction model corresponding to the detection result data, extract features from the detection result data of each candidate obstacle to obtain feature vectors corresponding to each candidate obstacle under the feature extraction model;

[0038] Based on the feature vectors of the candidate obstacles respectively corresponding to the feature extraction model, information of the candidate obstacles in the current frame sensor result is obtained.

[0039] The second part of the embodiment of the present disclosure provides a multi-target obstacle tracking device, the device comprising:

[0040] A first encoding module is configured to encode each candidate obstacle information in the current frame sensor result by using a self-attention mechanism encoder to obtain first interaction information between the candidate obstacles;

[0041] A second encoding module is configured to encode the historical association result information corresponding to the current frame sensor result using a self-attention mechanism encoder to obtain second interaction information between the various association relationships, wherein the historical association result information represents the association relationship between the historical obstacles in the historical frame sensor results before the current frame sensor result on the historical frame sensor results;

[0042] A decoding module is configured to decode the first interaction information and the second interaction information using an attention mechanism decoder to obtain a decoding result, wherein the decoding result includes a similarity calculation result between each of the association relationships and each of the candidate obstacles;

[0043] A matching module, configured to process the similarity calculation result based on a preset matching algorithm to obtain a matching result between each candidate obstacle and each association relationship;

[0044] The association module is configured to associate the candidate obstacle with the historical obstacle corresponding to the matching association relationship to obtain an updated obstacle association relationship, wherein the updated obstacle association relationship is used to determine the tracking result of the multi-target obstacle.

[0045] The third part of the embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of any one of the methods described in the first part are implemented.

[0046] The fourth part of the embodiment of the present disclosure provides an electronic device, including:

[0047] a memory having a computer program stored thereon;

[0048] A processor, configured to execute the computer program in the memory to implement the steps of any one of the methods described in the first part.

[0049] The fifth aspect of the embodiment of the present disclosure provides an autonomous driving vehicle, including:

[0050] a memory having a computer program stored thereon;

[0051] A processor, configured to execute the computer program in the memory to implement the steps of any one of the methods described in the first part.

[0052] By adopting the above technical solution, at least the following beneficial technical effects can be achieved:

[0053] A self-attention mechanism encoder is used to encode information of each candidate obstacle to obtain first interaction information between each candidate obstacle, and a self-attention mechanism encoder is used to encode historical association result information to obtain second interaction information between each association relationship, and an attention mechanism decoder is used to decode the first interaction information and the second interaction information to obtain similarity calculation results between each association relationship and each candidate obstacle, and the similarity calculation results are processed based on a preset matching algorithm to obtain matching results between each candidate obstacle and each association relationship, and the candidate obstacle is associated with the historical obstacle corresponding to the matching association relationship to obtain an updated obstacle association relationship, and the updated obstacle association relationship is used to determine the tracking results of the multi-target obstacles. Since the updated obstacle association relationship can be obtained by inputting the information of each candidate obstacle in the current frame sensor result, the tracking of multiple target obstacles is realized, so that the multi-target obstacle tracking method of the embodiment of the present disclosure has an end-to-end characteristic, and the overall process maintains consistency in the calculation of the association results in the training and reasoning stages, which is conducive to the overall optimization process of each link converging to the global optimum, improving the multi-target obstacle tracking effect, and at the same time making it easier to improve and maintain the subsequent system capabilities, reducing the labor cost of manual parameter adjustment.

[0054] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0056] Figure 1 The figure is a flowchart of a multi-target obstacle tracking method according to an exemplary embodiment of the present disclosure.

[0057] Figure 2 It is a structural schematic diagram of a multi-target obstacle tracking model according to an exemplary embodiment of the present disclosure.

[0058] Figure 3 It is a structural schematic diagram of another multi-target obstacle tracking model according to an exemplary embodiment of the present disclosure.

[0059] Figure 4 The figure is a block diagram of a multi-target obstacle tracking device according to an exemplary embodiment of the present disclosure.

[0060] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

[0061] Figure 6It is a functional block diagram of an autonomous driving vehicle according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0063] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.

[0064] It should be noted that, in the present disclosure, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises", or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or device.

[0065] See also Figure 1 , Figure 1 is a flowchart of a multi-target obstacle tracking method according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the multi-target obstacle tracking method includes the following steps:

[0066] S110, using a self-attention mechanism encoder to encode each candidate obstacle information in the current frame sensor result to obtain first interaction information between each candidate obstacle.

[0067] S120, using a self-attention mechanism encoder to encode historical association result information corresponding to the current frame sensor result to obtain second interactive information between each association relationship, the historical association result information represents the association relationship between historical obstacles in the historical frame sensor results before the current frame sensor result on the historical frame sensor results.

[0068] In some embodiments, the sensor result can be understood as data collected by the sensor, such as data collected by an image sensor, data collected by a lidar, data collected by an ultrasonic sensor, data collected by a millimeter wave radar, etc.

[0069] In some implementations, the candidate obstacle in the current frame sensor result may be an obstacle actually obtained by performing obstacle detection on the current frame sensor result.

[0070] In this case, the method of the embodiment of the present disclosure may further include the following step of obtaining information of each candidate obstacle in the current frame sensor result:

[0071] The detection result data of each candidate obstacle in the current frame sensor result is obtained; the detection result data of each candidate obstacle is subjected to feature extraction using a feature extraction model corresponding to the detection result data to obtain feature vectors corresponding to each candidate obstacle under the feature extraction model; and information of each candidate obstacle in the current frame sensor result is obtained based on the feature vectors corresponding to each candidate obstacle under the feature extraction model.

[0072] In the embodiment of the present disclosure, obstacle detection can be performed on the current frame sensor result to obtain detection result data corresponding to each candidate obstacle in the current frame sensor result. According to different data expression forms or sensors, the detection result data corresponding to the candidate obstacle can have one or more forms, for example, point cloud data, sensor result data, parameter data, etc. can be included.

[0073] It should be noted that when performing obstacle detection on the current frame sensor result, obstacles can be different for different fields or application scenarios and can be defined according to actual needs. For example, in the field of autonomous driving, obstacles such as other vehicles, roadblocks, and pedestrians other than autonomous driving vehicles can be detected. For another example, in the field of security, pedestrians can be detected as obstacles.

[0074] In the disclosed embodiment, after obtaining the detection result data of each candidate obstacle in the current frame sensor result, the detection result data of each candidate obstacle can be feature extracted using a feature extraction model corresponding to the detection result data to obtain feature vectors corresponding to each candidate obstacle under the feature extraction model.

[0075] For example, when the detection result data includes point cloud data, point cloud feature extraction models such as Pointnet can be used to extract features from the point cloud data to obtain corresponding feature vectors. For another example, when the detection result data includes image data, image feature extraction models such as convolutional neural networks (CNN) can be used to extract features from the image data to obtain corresponding feature vectors. For another example, when the detection result data includes parameter data, parameter feature extraction models such as feedforward neural networks (FFN) can be used to extract features from the parameter data to obtain corresponding feature vectors.

[0076] In some embodiments, when the detection result data includes multiple data types, after obtaining the feature vectors corresponding to each candidate obstacle under the feature extraction model, the feature vectors corresponding to each candidate obstacle under the feature extraction model can be spliced ​​to obtain the information of each candidate obstacle in the current frame sensor result.

[0077] It should be noted that, in the disclosed embodiment, the current frame sensor result may be the first frame sensor result, or any frame sensor result after the first frame. When the current frame sensor result is the first frame sensor result, there is no earlier historical frame sensor result, and it is impossible to obtain the historical association result information corresponding to the current frame sensor result. Therefore, in some embodiments, obtaining the information of each candidate obstacle in the current frame sensor result and the historical association result information corresponding to the current frame sensor result may be performed when the current frame sensor result is a sensor result after the first frame sensor result, that is, when the historical association result information corresponding to the current frame sensor result is obtained.

[0078] In some embodiments, the self-attention mechanism encoder for encoding candidate obstacle information may be an encoder in a Transformer model, which is a neural network model based on a self-attention mechanism and is used to process sequence data.

[0079] Optionally, the self-attention mechanism encoder may include one or more sub-encoders, each sub-encoder may include a self-attention module and a feedforward neural network module, optionally, the self-attention module may include a self-attention layer and a summation and normalization layer, the feedforward neural network module may include a feedforward neural network layer and a summation and normalization layer, optionally, the self-attention layer may be a multi-head self-attention layer.

[0080] It should be noted that the structure of the self-attention mechanism encoder in the embodiment of the present disclosure can refer to the relevant technology. The above description is only an example and is not intended to limit the structure of the self-attention mechanism encoder.

[0081] In the embodiment of the present disclosure, a self-attention mechanism encoder can be used to encode the obtained information of each candidate obstacle, extract the first interaction information between each candidate obstacle, and then another self-attention mechanism encoder can be used to encode the obtained historical association result information to extract the second interaction information between each association relationship.

[0082] S130, using the attention mechanism decoder to decode the first interaction information and the second interaction information to obtain a decoding result.

[0083] The decoding result includes the similarity calculation result between each association relationship and each candidate obstacle.

[0084] In addition, the decoding result may also include third interaction information between each association relationship and each candidate obstacle.

[0085] In some embodiments, the attention mechanism decoder for decoding the first interaction information and the second interaction information may be a decoder in a Transformer model.

[0086] Optionally, the attention mechanism decoder may include one or more sub-decoders, each sub-decoder may include an attention module and a feedforward neural network module, optionally, the attention module may include an attention layer and a summation and normalization layer, the feedforward neural network module may include a feedforward neural network layer and a summation and normalization layer, optionally, the attention layer may be a multi-head attention layer.

[0087] It should be noted that the structure of the attention mechanism decoder in the embodiment of the present disclosure can refer to the relevant technology. The above description is only an example and is not intended to limit the structure of the attention mechanism decoder.

[0088] In the disclosed embodiment, after obtaining the first interaction information and the second interaction information, the attention mechanism decoder can be used to decode the first interaction information and the second interaction information to obtain a decoding result.

[0089] In some implementations, using an attention mechanism decoder to decode the first interaction information and the second interaction information to obtain a decoding result may include the following steps:

[0090] The first interaction information is processed based on the attention mechanism decoder to obtain a key matrix and a value matrix; the second interaction information is processed based on the attention mechanism decoder to obtain a query matrix; based on the key matrix and the query matrix, a similarity calculation result is obtained; based on the similarity calculation result and the value matrix, the third interaction information is obtained.

[0091] In the disclosed embodiment, when the attention mechanism decoder is used to decode the first interaction information and the second interaction information, a key matrix and a value matrix are obtained by processing the first interaction information, and a query matrix is ​​obtained by processing the second interaction information, and a similarity calculation result is obtained based on the key matrix and the query matrix, and the third interaction information is obtained further based on the similarity calculation result and the value matrix, so that the similarity calculation result and the third interaction information can both refer to the candidate obstacle information in the current frame sensor result and the association relationship information of the historical obstacles on the historical frame sensor result, so that the similarity calculation result and the third interaction information can both be fused and expressed into the interaction information between the candidate obstacles in the current frame sensor result and the association relationship information of the historical obstacles on the historical frame sensor result.

[0092] S140: Process the similarity calculation result based on a preset matching algorithm to obtain a matching result between each candidate obstacle and each association relationship.

[0093] In some implementations, the preset matching algorithm may be a configurable Hungarian matching algorithm. By using the configurable Hungarian matching algorithm as a matching algorithm, the overall process may be end-to-end configurable.

[0094] In the disclosed embodiment, after the similarity calculation result is obtained, the similarity calculation result can be used as an input of a preset matching algorithm, and processed by the preset matching algorithm to obtain a matching result between each candidate obstacle and each association relationship.

[0095] For example, after obtaining the similarity calculation result, the similarity calculation result can be used as the input of the differentiable Hungarian matching algorithm, and processed by the differentiable Hungarian matching algorithm to obtain the matching result between each candidate obstacle and each association relationship output by the differentiable Hungarian matching algorithm.

[0096] S150: Associating the candidate obstacle with the historical obstacle corresponding to the matching association relationship to obtain an updated obstacle association relationship.

[0097] In some implementations, the updated obstacle association relationship is used to determine the tracking results of multiple target obstacles.

[0098] In some implementations, the updated association relationships corresponding to the various obstacles may be determined as the tracking results of the various target obstacles.

[0099] In the disclosed embodiment, after being processed by a preset matching algorithm, a matching result between each candidate obstacle and each association relationship can be obtained. For example, a certain candidate obstacle can be matched with a certain association relationship. Thus, the candidate obstacle can be associated with the historical obstacle corresponding to the matching association relationship to obtain an updated obstacle association relationship.

[0100] By adopting the above method, the information of each candidate obstacle is encoded by a self-attention mechanism encoder to obtain the first interaction information between each candidate obstacle, and the historical association result information is encoded by the self-attention mechanism encoder to obtain the second interaction information between each association relationship, and the first interaction information and the second interaction information are decoded by an attention mechanism decoder to obtain the similarity calculation result between each association relationship and each candidate obstacle, and the similarity calculation result is processed based on a preset matching algorithm to obtain the matching result between each candidate obstacle and each association relationship, and the candidate obstacle is associated with the historical obstacle corresponding to the matching association relationship to obtain the updated obstacle association relationship, and the updated obstacle association relationship is used to determine the tracking result of the multi-target obstacle. Since the updated obstacle association relationship can be obtained by inputting the information of each candidate obstacle in the current frame sensor result, the tracking of multiple target obstacles is realized, so that the multi-target obstacle tracking method of the embodiment of the present disclosure has an end-to-end characteristic, and the overall process maintains consistency in the calculation of the association results in the training and reasoning stages, which is conducive to the overall optimization process of each link converging to the global optimum, improving the multi-target obstacle tracking effect, and at the same time making it easier to improve and maintain the subsequent system capabilities, reducing the labor cost of manual parameter adjustment.

[0101] In combination with the foregoing content, it can be seen that the decoding result also includes the third interaction information between each association relationship and each candidate obstacle. In some embodiments, when the decoding result also includes the third interaction information between each association relationship and each candidate obstacle, the third interaction information can be used as the historical association result information corresponding to the next frame sensor result of the current frame sensor result.

[0102] That is to say, in the embodiment of the present disclosure, when the current frame is any frame sensor result after the first frame, for example, the fifth frame sensor result, when performing multiple obstacle association on the fifth frame sensor result, each candidate obstacle information in the fifth frame sensor result can be obtained, and the third interaction information obtained when performing multiple obstacle association on the fourth frame sensor result can be obtained.

[0103] In combination with the foregoing content, it can be seen that in the embodiments of the present disclosure, in some implementations, the method of the embodiments of the present disclosure may further include the following steps:

[0104] The second interaction information is processed by using a pre-trained state prediction model to obtain state information of each candidate obstacle in the current frame sensor result, wherein the state prediction model is trained by multiple groups of sample data including the second interaction information and state information labels of the candidate obstacles, and the state information of each candidate obstacle in the current frame sensor result is used to determine each candidate obstacle information in the current frame sensor result.

[0105] In the embodiment of the present disclosure, after the second interactive information is acquired, the second interactive information may be processed using a pre-trained state prediction model to predict the state information of each candidate obstacle in the current frame sensor result of the current frame. In combination with the foregoing, it can be seen that in some embodiments, the candidate obstacles in the current frame sensor result may also include obstacles that are not detected in the current frame sensor result and are predicted based on the sensor result before the current frame sensor result. Therefore, in the embodiment of the present disclosure, the information of each candidate obstacle in the current frame sensor result may be further determined based on the state information of each candidate obstacle in the current frame sensor result. Optionally, the information of each candidate obstacle in the current frame sensor result may be determined based on the information of each candidate obstacle in the historical frame sensor result before the current frame sensor result and the state information of each candidate obstacle in the current frame sensor result.

[0106] In some implementations, after predicting and determining each candidate obstacle information in the current frame sensor result, the second interaction information may be updated based on the predicted and determined each candidate obstacle information in the current frame sensor result to improve the accuracy of the second interaction information.

[0107] In some implementations, the state information of the candidate obstacle may include speed, direction, acceleration, etc.

[0108] In some implementations, the candidate obstacle information may be a feature vector representing the position of the candidate obstacle.

[0109] Thus, according to the state information of the candidate obstacle and the candidate obstacle information in the current frame sensor result, the candidate obstacle information in the next frame sensor result can be obtained.

[0110] In the disclosed embodiment, the pre-trained state prediction model may be a supervised learning model, which may be obtained by training with a plurality of groups of sample data including the second interaction information and state information labels of the candidate obstacles.

[0111] In some implementations, the method of the embodiment of the present disclosure may further include the following steps:

[0112] An association relationship corresponding to a target candidate obstacle is created, where the target candidate obstacle is a candidate obstacle for which there is no matching association relationship.

[0113] Considering that the candidate obstacle is an obstacle in the current frame sensor result, when a candidate obstacle does not have a matching association relationship, the candidate obstacle may appear for the first time. In order to facilitate the association of the candidate obstacle in subsequent frame sensor results, in the embodiment of the present disclosure, a corresponding initial association relationship may be created for the candidate obstacle.

[0114] In some embodiments, a self-attention mechanism encoder can be used to encode the candidate obstacle information in the current frame sensor result to obtain the first interaction information between the candidate obstacles. Then, the first interaction information corresponding to the target candidate obstacle can be extracted from the first interaction information, and an initial association relationship corresponding to the target candidate obstacle can be created based on the first interaction information.

[0115] It is understandable that there are two situations in which there is no matching association relationship for a target candidate obstacle. In the first situation, when the current frame sensor result is the first frame sensor result, the multi-obstacle association process has not been performed. At this time, the candidate obstacles in the current frame sensor result are all target candidate obstacles. In the second situation, when the current frame sensor result is the sensor result after the first frame sensor result, it is possible that a candidate obstacle in the current frame sensor result appears for the first time in the current frame sensor result, or a candidate obstacle does not appear in the preset number of frame sensor results before the current frame, resulting in the obstacle association relationship corresponding to the candidate obstacle being deleted. At this time, the candidate obstacle that appears for the first time and the candidate obstacle that appears again after the obstacle association relationship is deleted do not have a matching association relationship, and thus can become a target candidate obstacle. Therefore, in the embodiments of the present disclosure, according to different situations of the target candidate obstacles, there can be different methods for creating an association relationship corresponding to the target candidate obstacles.

[0116] Therefore, in some implementations, creating an association relationship corresponding to a target candidate obstacle may include the following steps:

[0117] In the absence of historical association result information corresponding to the current frame sensor result, initial association result information is created based on the information of each candidate obstacle in the current frame sensor result. The initial association result information serves as the historical association result information corresponding to the next frame sensor result of the current frame sensor result. The initial association result information includes initial association relationships corresponding to each candidate obstacle.

[0118] In the embodiment of the present disclosure, when there is no historical association result information corresponding to the current frame sensor result, that is, when the current frame is the first frame sensor result, an initial association relationship corresponding to each candidate obstacle can be created based on each candidate obstacle information in the current frame sensor result. Then, the created initial association relationship corresponding to each candidate obstacle can be composed of initial association result information to serve as the historical association result information corresponding to the next frame sensor result of the current frame sensor result.

[0119] In addition, in some implementations, creating an association relationship corresponding to a target candidate obstacle may further include the following steps:

[0120] When it is determined according to the matching result that there is no target candidate obstacle with a matching association relationship, an initial association relationship corresponding to the target candidate obstacle is created according to the target candidate obstacle information.

[0121] In the embodiment of the present disclosure, when the current frame sensor result is a sensor result subsequent to the first frame sensor result, it may be determined based on the matching result that there is no association relationship matching the target candidate obstacle. Therefore, when it is determined based on the matching result that there is no target candidate obstacle with a matching association relationship, an initial association relationship corresponding to the target candidate obstacle may be created based on the target candidate obstacle information.

[0122] By adopting the above method, by creating corresponding association relationships for candidate obstacles that do not have matching association relationships, subsequent association of multiple obstacles can be facilitated.

[0123] In addition, considering that there may be a scenario where a candidate obstacle is missing in one or more frames of sensor results due to occlusion, and the candidate obstacle reappears after one or more frames, or there is a scenario where a candidate obstacle no longer appears in the sensor results starting from a certain frame, for example, an obstacle no longer appears in the image over time. Therefore, in order to accurately deal with different scenarios and improve the effect of associating multiple obstacles, in some embodiments, the method of the embodiment of the present disclosure may also include the following steps:

[0124] For any obstacle association relationship, if it is determined according to the obstacle association relationship that there is no candidate obstacle associated with the target obstacle in the obstacle association relationship in the most recent preset number of frame sensor results, the obstacle association relationship is deleted.

[0125] In the disclosed embodiment, a number of sensor result frames, such as 10 frames, can be preset, so that for any obstacle association relationship, if it is determined that there are no candidate obstacles associated with the target obstacle in the obstacle association relationship in the most recent preset number of frames of sensor results, it means that the target obstacle may no longer appear in the sensor results, and the obstacle association relationship corresponding to it can be deleted to save resources. Otherwise, it can be temporarily saved so that when it appears again in the future, the corresponding obstacle association relationship can be quickly obtained and associated with the obstacle association relationship to avoid the loss of the obstacle association relationship due to occlusion and other reasons, thereby improving the accuracy of obstacle association.

[0126] For example, assuming that the number of sensor result frames is preset to 10 frames, the current frame is the 20th frame, and there are three obstacle association relationships in the historical association result information, namely, the association relationship corresponding to obstacle A, the association relationship corresponding to obstacle B, and the association relationship corresponding to obstacle C. Assuming that obstacle A no longer appears from the 11th frame to the current 20th frame, the association relationship corresponding to obstacle A can be deleted after the current 20th frame. When processing the 20th frame, the most recent 10 frames are the 11th frame to the 20th frame.

[0127] It should be noted that the number of sensor result frames is pre-set, that is, the number of sensor result frames can be set according to the actual scene, for example, according to the movement speed of obstacles in the scene, the sensor result acquisition frame rate, etc.

[0128] In some implementations, each candidate obstacle information is candidate obstacle information in a current frame sensor result collected by a preset number of sensors, and the attention mechanism decoder includes a preset number of sub-attention mechanism decoders stacked in sequence. In this case, encoding each candidate obstacle information using a self-attention mechanism encoder to obtain first interaction information between each candidate obstacle may include the following steps:

[0129] The candidate obstacle information corresponding to the multiple sensors is encoded using the self-attention mechanism encoder corresponding to each sensor, so as to obtain the first interaction information between the candidate obstacles respectively corresponding to the multiple sensors.

[0130] Further, using the attention mechanism decoder to decode the first interaction information and the second interaction information to obtain a decoding result may include the following steps:

[0131] The Mth sub-attention mechanism decoder is used to process the corresponding first input parameter and the corresponding second input parameter to obtain a decoding result output by the Mth sub-attention mechanism, where the decoding result includes a similarity calculation result and the third interaction information; the similarity calculation result included in the decoding result output by the last sub-attention mechanism decoder stacked in sequence is determined as the similarity calculation result between each association relationship and each candidate obstacle.

[0132] Among them, when M is equal to 1, the corresponding first input parameter is the second interaction information, and the corresponding second input parameter is the first interaction information between the candidate obstacles corresponding to the first sensor. When M is a positive integer greater than 1, the corresponding first input parameter is the third interaction information included in the decoding result output by the M-1th sub-attention mechanism decoder, and the corresponding second input parameter is the first interaction information between the candidate obstacles corresponding to the Mth sensor.

[0133] In some embodiments, when M is equal to 1, the corresponding first input parameter is the second interaction information obtained by encoding the third interaction information after processing the previous frame output by the last sub-attention mechanism decoder stacked in sequence through the self-attention mechanism encoder.

[0134] In the embodiment of the present disclosure, a preset number of sensors may be set to collect sensor results, so that the aforementioned candidate obstacle information is composed of the candidate obstacle information in the current frame sensor results collected by the preset number of sensors. In this case, self-attention mechanism encoders corresponding to the preset number of sensors may be set, so that after obtaining the candidate obstacle information in the current frame sensor results collected by the preset number of sensors, the self-attention mechanism encoders corresponding to each sensor are used to encode the candidate obstacle information corresponding to each sensor, so as to obtain the first interaction information between the candidate obstacles corresponding to each sensor.

[0135] Optionally, the preset number of sensors may include sensors of the same type, such as multiple image sensors, multiple laser radars, etc. Optionally, the preset number of sensors may also include multiple types of sensors, with the number of each type of sensor being one or more.

[0136] In addition, a preset number of sub-attention mechanism decoders may be set, and the preset number of sub-attention mechanism decoders may be stacked in sequence for decoding.

[0137] During decoding, the first sub-attention mechanism decoder has no adjacent previous sub-attention mechanism decoder, and can obtain the third interaction information after the previous frame processed by the last sub-attention mechanism decoder stacked in sequence, and then set the second interaction information obtained by encoding the third interaction information obtained by the self-attention mechanism encoder as the first input parameter of the first sub-attention mechanism decoder. At the same time, the first interaction information between the candidate obstacles corresponding to the first sensor can be set as the second input parameter of the first sub-attention mechanism decoder. After the first sub-attention mechanism decoder processes the first input parameter and the second input parameter, the corresponding third interaction parameter can be output.

[0138] Starting from the second sub-attention mechanism decoder, the first input parameter of each sub-attention mechanism decoder is the third interaction information output by the adjacent previous sub-attention mechanism decoder, and the second input parameter of each sub-attention mechanism is the first parameter output by the corresponding self-attention mechanism encoder. Thus, after being processed by each sub-attention mechanism decoder in sequence, the similarity calculation result output by the last sub-attention mechanism decoder stacked in sequence can be determined as the similarity calculation result between each association relationship and each candidate obstacle.

[0139] Subsequently, the similarity calculation results output by the last stacked sub-attention mechanism decoder can be processed based on the preset matching algorithm to obtain the matching results between each candidate obstacle and each association relationship, and further associate the candidate obstacle with the historical obstacles corresponding to the matching association relationship to obtain the updated obstacle association relationship.

[0140] By adopting the above method, a process of associating obstacles in sensor results collected by multiple sensors at the same time is realized, and this process simplifies the fusion process of obstacles in sensor results collected by multiple sensors.

[0141] It should be noted that the first frame sensor result mentioned in the embodiment of the present disclosure can be any frame in a continuous sensor result, for example, it can be any frame in a video at one end. In this case, the obstacles in the continuous sensor results are associated starting from this frame.

[0142] Combine the following Figure 2 The structural diagram of the multi-obstacle association model shown in the figure illustrates the multi-target obstacle association method for images collected by a single image sensor in an embodiment of the present disclosure in combination with an autonomous driving scenario.

[0143] Please refer to Figure 2 , Figure 2: This is a schematic diagram of the structure of an exemplary multi-obstacle association model of the present disclosure, which includes a self-attention mechanism encoder 201, a self-attention mechanism encoder 202, and an attention mechanism decoder 203. The detailed structures of the self-attention mechanism encoder 201, the self-attention mechanism encoder 202, and the attention mechanism decoder 203 can refer to the aforementioned embodiments, and will not be repeated here.

[0144] For the first frame:

[0145] The image data corresponding to the obstacle detection result in the first frame image is obtained, and the CNN model is used to perform feature extraction on the image data corresponding to the obstacle detection result to obtain information of each candidate obstacle in the first frame image.

[0146] The information of each candidate obstacle in the first frame image is encoded using the self-attention mechanism encoder 201 to obtain first interaction information between each candidate obstacle, and a track query corresponding to each candidate obstacle is created according to the first interaction information between each candidate obstacle.

[0147] For the second frame image:

[0148] The image data corresponding to the obstacle detection result in the second frame image is obtained, and the CNN model is used to perform feature extraction on the image data corresponding to the obstacle detection result to obtain information of each candidate obstacle in the second frame image.

[0149] The information of each candidate obstacle in the second frame image is encoded using the self-attention mechanism encoder 201 to obtain the first interactive information between the candidate obstacles, and the track query obtained by processing the first frame image is encoded using the self-attention mechanism encoder 202 to obtain the second interactive information between the association relationships.

[0150] The first interaction information output by the self-attention mechanism encoder 201 is input into the attention mechanism decoder 203 for processing to obtain a key matrix and a value matrix, and the second interaction information output by the self-attention mechanism encoder 202 is input into the attention mechanism decoder 203 for processing to obtain a query matrix. The attention mechanism decoder 203 continues to obtain a similarity calculation result based on the key matrix and the query matrix. In addition, the attention mechanism decoder 203 can also continue to obtain the third interaction information based on the similarity calculation result and the value matrix, and use the third interaction information as the historical association result information corresponding to the third frame image, namely, the track query.

[0151] In some embodiments, before the second interaction information outputted from the self-attention mechanism encoder 202 is inputted into the attention mechanism decoder 203 for processing, the second interaction information may be processed using a pre-trained state prediction model to obtain the state information of each candidate obstacle in the second frame image, and then the candidate obstacle information of the second frame image is predicted based on the state information of each candidate obstacle in the second frame image, and further based on the predicted candidate obstacle information of the second frame image, the second interaction information obtained in the current cycle is updated, and the updated second interaction information is inputted into the attention mechanism decoder 203 for processing to obtain a query matrix, so as to further improve the accuracy of multi-obstacle association. Then, the similarity calculation result may be processed by a differentiable Hungarian matching algorithm to obtain the matching result between each candidate obstacle and each association relationship.

[0152] Then, the candidate obstacles may be associated with the historical obstacles corresponding to the matching association relationships to obtain updated obstacle association relationships.

[0153] For the third frame image:

[0154] The image data corresponding to the obstacle detection result in the third frame image is obtained, and the CNN model is used to perform feature extraction on the image data corresponding to the obstacle detection result to obtain information of each candidate obstacle in the third frame image.

[0155] The information of each candidate obstacle in the third frame image is encoded using the self-attention mechanism encoder 201 to obtain the first interaction information between the candidate obstacles, and the track query obtained by processing the second frame image is encoded using the self-attention mechanism encoder 202 to obtain the second interaction information between the association relationships.

[0156] The first interaction information output by the self-attention mechanism encoder 201 is input into the attention mechanism decoder 203 for processing to obtain a key matrix and a value matrix, and the second interaction information output by the self-attention mechanism encoder 202 is input into the attention mechanism decoder 203 for processing to obtain a query matrix. The attention mechanism decoder 203 continues to obtain a similarity calculation result based on the key matrix and the query matrix. In addition, the attention mechanism decoder 203 can also continue to obtain the third interaction information based on the similarity calculation result and the value matrix, and use the third interaction information as the historical association result information corresponding to the third frame image, namely, the track query.

[0157] In some embodiments, before the second interaction information output by the self-attention mechanism encoder 202 is input into the attention mechanism decoder 203 for processing, the second interaction information can also be processed using a pre-trained state prediction model to obtain state information of each candidate obstacle in the third frame image, and then the candidate obstacle information of the third frame image is predicted based on the state information of each candidate obstacle in the third frame image, and further based on the predicted candidate obstacle information of the third frame image, the second interaction information obtained in the current cycle is updated, and the updated second interaction information is input into the attention mechanism decoder 203 for processing to obtain a query matrix, so as to further improve the accuracy of multi-obstacle association.

[0158] Then, the similarity calculation results can be processed by the differentiable Hungarian matching algorithm to obtain the matching results between each candidate obstacle and each association relationship.

[0159] Then, the candidate obstacles may be associated with the historical obstacles corresponding to the matching association relationships to obtain updated obstacle association relationships.

[0160] Therefore, by continuously repeating the above process, obstacles in multiple frames of the video can be associated.

[0161] Combine the following Figure 3 The structural diagram of the multi-obstacle association model shown in the figure illustrates the multi-target obstacle association method of the sensor results collected by three sensors (one image sensor plus two lidars) in the embodiment of the present disclosure in combination with the autonomous driving scenario.

[0162] Please refer to Figure 3 , Figure 3 : This is a schematic diagram of the structure of an exemplary multi-obstacle association model of the present disclosure, which includes three self-attention mechanism encoders 301, one self-attention mechanism encoder 302, and three sequentially stacked sub-attention mechanism decoders 303. The detailed structures of the self-attention mechanism encoder 301, the self-attention mechanism encoder 302, and the sub-attention mechanism decoder 303 can refer to the aforementioned embodiment, and will not be repeated here.

[0163] Sensor results for the first frame:

[0164] The image data corresponding to the obstacle detection result in the first frame image captured by the image sensor is obtained, and the CNN model is used to perform feature extraction on the image data corresponding to the obstacle detection result corresponding to the image sensor to obtain information of each candidate obstacle in the first frame image corresponding to the image sensor.

[0165] The first self-attention mechanism encoder 301 is used to encode the information of each candidate obstacle in the first frame image corresponding to the first sensor to obtain the first interaction information between the candidate obstacles, and a track query corresponding to each candidate obstacle is created according to the first interaction information between the candidate obstacles.

[0166] Sensor results for the second frame:

[0167] The image data corresponding to the obstacle detection result in the second frame image captured by the image sensor is obtained, and the CNN model is used to perform feature extraction on the image data corresponding to the obstacle detection result corresponding to the image sensor to obtain information of each candidate obstacle in the second frame image corresponding to the image sensor.

[0168] The point cloud data corresponding to the obstacle detection results in the second frame point cloud data collected by the two laser radars are obtained, and the PointNet model is used to extract features from the point cloud data corresponding to the obstacle detection results of the two laser radars to obtain the information of each candidate obstacle in the second frame point cloud data corresponding to the two laser radars.

[0169] Three self-attention mechanism encoders 301 are used to encode the information of each candidate obstacle in the second frame sensor results corresponding to the three sensors respectively, so as to obtain the first interaction information between the candidate obstacles corresponding to the three sensors respectively. The self-attention mechanism encoder 302 is used to encode the track query obtained by processing the first frame image, so as to obtain the second interaction information between the association relationships.

[0170] The first interaction information output by the first self-attention mechanism encoder 301 is input into the first sub-attention mechanism decoder 303 for processing to obtain a key matrix and a value matrix, and the second interaction information output by the self-attention mechanism encoder 302 is input into the first sub-attention mechanism decoder 303 for processing to obtain a query matrix. The first sub-attention mechanism decoder 303 continues to obtain a similarity calculation result based on the key matrix and the query matrix, and continues to obtain the third interaction information based on the similarity calculation result and the value matrix.

[0171] In some embodiments, before the second interaction information output by the self-attention mechanism encoder 302 is input into the first sub-attention mechanism decoder 303 for processing, the second interaction information can also be processed using a pre-trained state prediction model to obtain state information of each candidate obstacle in the second frame sensor result, and then the candidate obstacle information in the second frame sensor result is predicted based on the state information of each candidate obstacle in the second frame sensor result, and the second interaction information obtained in the current cycle is further updated based on the predicted candidate obstacle information in the second frame sensor result, and the updated second interaction information is input into the attention mechanism decoder 203 for processing to obtain a query matrix, so as to further improve the accuracy of multi-obstacle association.

[0172] The first interaction information output by the second self-attention mechanism encoder 301 is input into the second sub-attention mechanism decoder 303 for processing to obtain a key matrix and a value matrix, and the third interaction information output by the first sub-attention mechanism decoder 303 is input into the second sub-attention mechanism decoder 303 for processing to obtain a query matrix. The second sub-attention mechanism decoder 303 continues to obtain a similarity calculation result based on the key matrix and the query matrix, and continues to obtain the third interaction information based on the similarity calculation result and the value matrix.

[0173] The first interaction information output by the third self-attention mechanism encoder 301 is input into the third sub-attention mechanism decoder 303 for processing to obtain a key matrix and a value matrix, and the third interaction information output by the second sub-attention mechanism decoder 303 is input into the third sub-attention mechanism decoder 303 for processing to obtain a query matrix. The third sub-attention mechanism decoder 303 continues to obtain a similarity calculation result based on the key matrix and the query matrix, and continues to obtain the third interaction information based on the similarity calculation result and the value matrix.

[0174] Thus, by continuously repeating the above process, the similarity calculation result output by the last sub-attention mechanism decoder 303 stacked in sequence can be obtained, and the similarity calculation result can be determined as the similarity calculation result between each association relationship and each candidate obstacle.

[0175] Then, the similarity calculation results can be processed by the differentiable Hungarian matching algorithm to obtain the matching results between each candidate obstacle and each association relationship.

[0176] Then, the candidate obstacles may be associated with the historical obstacles corresponding to the matching association relationships to obtain updated obstacle association relationships.

[0177] Therefore, by continuously repeating the similar process for the second frame of sensor results, it is possible to associate obstacles in multiple frames of sensor results collected by the three sensors.

[0178] Figure 4 is a block diagram of a multi-target obstacle tracking device according to an exemplary embodiment of the present disclosure, such as Figure 4 As shown, the multi-target obstacle tracking device 400 includes:

[0179] A first encoding module 410 is configured to encode each candidate obstacle information in the current frame sensor result by using a self-attention mechanism encoder to obtain first interaction information between the candidate obstacles;

[0180] The second encoding module 420 is configured to encode the historical association result information corresponding to the current frame sensor result using a self-attention mechanism encoder to obtain second interaction information between the various association relationships, wherein the historical association result information represents the association relationship between the historical obstacles in the historical frame sensor results before the current frame sensor result on the historical frame sensor results;

[0181] A decoding module 430 is configured to decode the first interaction information and the second interaction information using an attention mechanism decoder to obtain a decoding result, wherein the decoding result includes a similarity calculation result between each of the association relationships and each of the candidate obstacles;

[0182] A matching module 440 is configured to process the similarity calculation result based on a preset matching algorithm to obtain a matching result between each candidate obstacle and each association relationship;

[0183] The association module 450 is configured to associate the candidate obstacle with the historical obstacle corresponding to the matching association relationship to obtain an updated obstacle association relationship, and the updated obstacle association relationship is used to determine the tracking result of the multi-target obstacle.

[0184] Optionally, the decoding result further includes third interaction information between each of the association relationships and each of the candidate obstacles, wherein the third interaction information serves as historical association result information corresponding to a next frame sensor result of the current frame sensor result.

[0185] Optionally, the apparatus 400 further includes:

[0186] The prediction module is configured to process the second interaction information using a pre-trained state prediction model to obtain state information of each candidate obstacle in the current frame sensor result, wherein the state prediction model is trained by multiple groups of sample data including the second interaction information and state information labels of the candidate obstacles, and the state information of each candidate obstacle in the current frame sensor result is used to determine each candidate obstacle information in the current frame sensor result.

[0187] Optionally, the decoding result includes the similarity calculation result and the third interaction information between each of the association relationships and each of the candidate obstacles, and the decoding module 430 includes:

[0188] A first processing submodule is configured to process the first interaction information based on the attention mechanism decoder to obtain a key matrix and a value matrix;

[0189] A second processing submodule is configured to process the second interaction information based on the attention mechanism decoder to obtain a query matrix;

[0190] A third processing submodule is configured to obtain the similarity calculation result based on the key matrix and the query matrix;

[0191] The fourth processing submodule is configured to obtain the third interaction information based on the similarity calculation result and the value matrix.

[0192] Optionally, the apparatus 400 further includes:

[0193] The creation module is configured to create an association relationship corresponding to a target candidate obstacle, where the target candidate obstacle is a candidate obstacle for which there is no matching association relationship.

[0194] Optionally, creating a module includes:

[0195] The first creation submodule is configured to create initial association result information according to each candidate obstacle information in the current frame sensor result when there is no historical association result information corresponding to the current frame sensor result, wherein the initial association result information serves as the historical association result information corresponding to the next frame sensor result of the current frame sensor result, and the initial association result information includes initial association relationships corresponding to each candidate obstacle.

[0196] Optionally, creating a module includes:

[0197] The second creation submodule is configured to create an initial association relationship corresponding to the target candidate obstacle according to the target candidate obstacle information when it is determined according to the matching result that there is no target candidate obstacle with a matching association relationship.

[0198] Optionally, the apparatus 400 further includes:

[0199] The deletion module is configured to delete any obstacle association relationship when it is determined that there is no candidate obstacle associated with the target obstacle in the obstacle association relationship in the most recent preset number of frame sensor results according to the obstacle association relationship.

[0200] Optionally, the matching module 440 is further configured to use the similarity calculation result as an input of a differentiable Hungarian matching algorithm, and obtain a matching result between each candidate obstacle and each association relationship output by the differentiable Hungarian matching algorithm.

[0201] Optionally, each candidate obstacle information is candidate obstacle information in the current frame sensor result collected by a preset number of sensors, and the attention mechanism decoder includes the preset number of sub-attention mechanism decoders stacked in sequence. In this case, the encoding module 420 includes: an encoding submodule, configured to respectively use the self-attention mechanism encoder corresponding to each sensor to encode the candidate obstacle information corresponding to each sensor, and obtain the first interactive information between the candidate obstacles corresponding to each sensor. The decoding module 430 includes:

[0202] The decoding submodule is configured to use the Mth sub-attention mechanism decoder to process the corresponding first input parameter and the corresponding second input parameter to obtain a decoding result output by the Mth sub-attention mechanism, wherein the decoding result includes a similarity calculation result and third interaction information; and determine the similarity calculation result included in the decoding result output by the last sub-attention mechanism decoder stacked in sequence as the similarity calculation result between each of the association relationships and each of the candidate obstacles;

[0203] Among them, when M is equal to 1, the corresponding first input parameter is the second interaction information, and the corresponding second input parameter is the first interaction information between the candidate obstacles corresponding to the first sensor. When M is a positive integer greater than 1, the corresponding first input parameter is the third interaction information included in the decoding result output by the M-1th sub-attention mechanism decoder, and the corresponding second input parameter is the first interaction information between the candidate obstacles corresponding to the Mth sensor.

[0204] Optionally, the apparatus 400 further includes:

[0205] An acquisition module is configured to acquire detection result data of each candidate obstacle in the current frame sensor result;

[0206] a feature extraction module configured to perform feature extraction on the detection result data of each candidate obstacle using a feature extraction model corresponding to the detection result data, and obtain feature vectors corresponding to each candidate obstacle under the feature extraction model;

[0207] The candidate obstacle information determination module is configured to obtain each candidate obstacle information in the current frame sensor result based on the feature vectors corresponding to each candidate obstacle under the feature extraction model.

[0208] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0209] Figure 5 FIG. 5 is a block diagram of an electronic device 500 according to an exemplary embodiment. Figure 5 As shown, the electronic device 500 may include: a processor 501 and a memory 502. The electronic device 500 may also include one or more of a multimedia component 503, an input / output (I / O) interface 504, and a communication component 505.

[0210] The processor 501 is used to control the overall operation of the electronic device 500 to complete all or part of the steps in the above-mentioned multi-target obstacle tracking method. The memory 502 is used to store various types of data to support the operation of the electronic device 500, and these data may include instructions for any application or method used to operate on the electronic device 500, as well as application-related data, such as graphic symbols, model data of information collection equipment, etc. The memory 502 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk. The multimedia component 503 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 502 or sent through the communication component 505. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 504 provides an interface between the processor 501 and other interface modules, and the other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 505 is used for wired or wireless communication between the electronic device 500 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 505 may include: Wi-Fi module, Bluetooth module, NFC module, etc.

[0211] In an exemplary embodiment, the electronic device 500 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned multi-target obstacle tracking method.

[0212] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned multi-target obstacle tracking method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 502 including program instructions, and the above-mentioned program instructions can be executed by the processor 501 of the electronic device 500 to complete the above-mentioned multi-target obstacle tracking method.

[0213] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above-mentioned multi-target obstacle tracking method when executed by the programmable device.

[0214] Figure 6 is a block diagram of an automatic driving vehicle 600 according to an exemplary embodiment. For example, the vehicle 600 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles.

[0215] Reference Figure 6 , the vehicle 600 may include various subsystems, for example, an infotainment system 610, a perception system 620, a decision control system 630, a drive system 640, and a computing platform 650. The vehicle 600 may also include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and each component of the vehicle 600 may be interconnected by wire or wireless means.

[0216] In some embodiments, the infotainment system 610 may include a communication system, an entertainment system, and a navigation system, among others.

[0217] The perception system 620 may include several sensors for sensing information about the environment around the vehicle 600. For example, the perception system 620 may include a global positioning system (the global positioning system may be a GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera.

[0218] The decision control system 630 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0219] The drive system 640 may include components that provide powered motion for the vehicle 600. In one embodiment, the drive system 640 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of an internal combustion engine, an electric motor, and an air compression engine. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0220] Some or all functions of the vehicle 600 are controlled by a computing platform 650. The computing platform 650 may include at least one processor 651 and a memory 652, and the processor 651 may execute instructions 653 stored in the memory 652.

[0221] The processor 651 may be any conventional processor, such as a commercially available CPU. The processor may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof.

[0222] The memory 652 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0223] In addition to the instructions 653 , the memory 652 may also store data, such as road maps, route information, and data such as the location, direction, and speed of the vehicle. The data stored in the memory 652 may be used by the computing platform 650 .

[0224] In the embodiment of the present disclosure, the processor 651 may execute instruction 653 to complete all or part of the steps of the above-mentioned multi-target obstacle tracking method.

[0225] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0226] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0227] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A multi-target obstacle tracking method, characterized in that: The method comprises: Encode each candidate obstacle information in the current frame sensor result by using a self-attention mechanism encoder to obtain first interaction information between the candidate obstacles; Encoding the historical association result information corresponding to the current frame sensor result using a self-attention mechanism encoder to obtain second interactive information between the association relationships, wherein the historical association result information represents the association relationship between historical obstacles in the historical frame sensor results before the current frame sensor result on the historical frame sensor results; Decoding the first interaction information and the second interaction information using an attention mechanism decoder to obtain a decoding result, wherein the decoding result includes a similarity calculation result between each of the association relationships and each of the candidate obstacles; Processing the similarity calculation result based on a preset matching algorithm to obtain a matching result between each candidate obstacle and each association relationship; The candidate obstacles are associated with the historical obstacles corresponding to the matching association relationships to obtain updated obstacle association relationships, and the updated obstacle association relationships are used to determine the tracking results of the multi-target obstacles.

2. The method according to claim 1, characterized in that The decoding result further includes third interaction information between each of the association relationships and each of the candidate obstacles, wherein the third interaction information serves as historical association result information corresponding to a next frame sensor result of the current frame sensor result.

3. The method according to claim 1, characterized in that The method further comprises: The second interaction information is processed by using a pre-trained state prediction model to obtain state information of each candidate obstacle in the current frame sensor result, wherein the state prediction model is trained by multiple groups of sample data including the second interaction information and state information labels of the candidate obstacles, and the state information of each candidate obstacle in the current frame sensor result is used to determine each candidate obstacle information in the current frame sensor result.

4. The method according to claim 2, characterized in that: The decoding result includes the similarity calculation result and the third interaction information between each of the association relationships and each of the candidate obstacles. The attention mechanism decoder is used to decode the first interaction information and the second interaction information to obtain a decoding result, including: Processing the first interaction information based on the attention mechanism decoder to obtain a key matrix and a value matrix; Processing the second interaction information based on the attention mechanism decoder to obtain a query matrix; Based on the key matrix and the query matrix, obtaining the similarity calculation result; The third interaction information is obtained based on the similarity calculation result and the value matrix.

5. The method according to claim 1, characterized in that The method further comprises: An association relationship corresponding to a target candidate obstacle is created, wherein the target candidate obstacle is a candidate obstacle for which there is no matching association relationship.

6. The method according to claim 5, characterized in that The creating an association relationship corresponding to the target candidate obstacle includes: In the absence of historical association result information corresponding to the current frame sensor result, initial association result information is created based on each candidate obstacle information in the current frame sensor result. The initial association result information serves as the historical association result information corresponding to the next frame sensor result of the current frame sensor result. The initial association result information includes initial association relationships corresponding to each candidate obstacle.

7. The method according to claim 5, characterized in that The creating an association relationship corresponding to the target candidate obstacle includes: In the case where it is determined according to the matching result that there is no target candidate obstacle with a matching association relationship, an initial association relationship corresponding to the target candidate obstacle is created according to the target candidate obstacle information.

8. The method according to claim 1, characterized in that The method further comprises: For any obstacle association relationship, if it is determined according to the obstacle association relationship that there is no candidate obstacle associated with the target obstacle in the obstacle association relationship in the most recent preset number of frame sensor results, the obstacle association relationship is deleted.

9. The method according to claim 1, characterized in that: The processing of the similarity calculation result based on a preset matching algorithm to obtain a matching result between each candidate obstacle and each association relationship includes: The similarity calculation result is used as an input of a differentiable Hungarian matching algorithm to obtain a matching result between each candidate obstacle and each association relationship output by the differentiable Hungarian matching algorithm.

10. The method according to any one of claims 1 to 8, characterized in that: The information of each candidate obstacle is information of each candidate obstacle in the current frame sensor result collected by a preset number of sensors, the attention mechanism decoder includes the preset number of sub-attention mechanism decoders stacked in sequence, and the self-attention mechanism encoder is used to encode the information of each candidate obstacle to obtain the first interaction information between the candidate obstacles, including: Encode the candidate obstacle information corresponding to each sensor by using the self-attention mechanism encoder corresponding to each sensor, and obtain the first interaction information between the candidate obstacles corresponding to each sensor; The using the attention mechanism decoder to decode the first interaction information and the second interaction information to obtain a decoding result includes: Using the M-th sub-attention mechanism decoder to process the corresponding first input parameter and the corresponding second input parameter to obtain a decoding result output by the M-th sub-attention mechanism, wherein the decoding result includes a similarity calculation result and the third interaction information; Determine the similarity calculation result included in the decoding result output by the last sub-attention mechanism decoder stacked in sequence as the similarity calculation result between each of the association relationships and each of the candidate obstacles; Among them, when M is equal to 1, the corresponding first input parameter is the second interaction information, and the corresponding second input parameter is the first interaction information between the candidate obstacles corresponding to the first sensor. When M is a positive integer greater than 1, the corresponding first input parameter is the third interaction information included in the decoding result output by the M-1th sub-attention mechanism decoder, and the corresponding second input parameter is the first interaction information between the candidate obstacles corresponding to the Mth sensor.

11. The method according to claim 1, characterized in that: The method further comprises: Obtaining detection result data of each candidate obstacle in the current frame sensor result; Using a feature extraction model corresponding to the detection result data, extract features from the detection result data of each candidate obstacle to obtain feature vectors corresponding to each candidate obstacle under the feature extraction model; Based on the feature vectors of the candidate obstacles respectively corresponding to the feature extraction model, information of the candidate obstacles in the current frame sensor result is obtained.

12. An autonomous driving vehicle, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 11.

Images