End-to-end pedestrian perception method and device based on spatiotemporal data association

Abstract

This application relates to the field of autonomous driving technology, and particularly to an end-to-end pedestrian perception method and apparatus based on spatiotemporal data association. The method includes: generating a target spatiotemporal change original feature map that fuses features from the current and historical moments based on sensor data at the current moment; generating specialized feature maps applicable to pedestrian detection, target tracking, intent recognition, and trajectory prediction sub-tasks based on the target spatiotemporal change original feature map; constructing feature association and interaction optimization mechanisms between sub-tasks based on the task-specific features in the specialized feature maps; and generating the final pedestrian perception result using the feature association and interaction optimization mechanisms. This solves the problems of lacking an effective spatiotemporal feature extraction method when serializing sub-tasks, leading to duplicate extracted target features and poor real-time accuracy, while using a single model lacks effective interaction optimization and conflict mitigation mechanisms.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an end-to-end pedestrian perception method and device based on spatiotemporal data correlation. Background Technology

[0002] Currently, with the continuous increase in the number of road users, road condition information is becoming increasingly complex. In complex and dynamic traffic environments, pedestrians are important road users, and efficient and accurate perception of pedestrians is crucial to the safety of intelligent connected vehicles.

[0003] In related technologies, the pedestrian perception task mainly includes four sub-tasks: pedestrian detection, target tracking, intent recognition, and trajectory prediction. The current main implementation methods are: (1) using independent models and methods to complete these four sub-tasks respectively; (2) completing multiple sub-tasks in a single model based on target features.

[0004] However, in related technologies, there is a lack of effective spatiotemporal feature extraction methods for multiple sub-tasks in sequence, making it difficult to combine short-term data tracking and long-term data prediction. This results in duplicated and underutilized target features, poor real-time performance and accuracy. Furthermore, different sub-tasks are completed using a single model, lacking an effective interaction optimization mechanism between different tasks. In addition, there are conflicts in the features required by different tasks, and there is a lack of effective conflict mitigation mechanisms. These issues urgently need improvement. Summary of the Invention

[0005] This application provides an end-to-end pedestrian perception method and apparatus based on spatiotemporal data association to solve the problems in related technologies, such as the lack of effective spatiotemporal feature extraction methods when serializing subtasks, resulting in repeated extraction of target features and poor real-time accuracy, while using a single model lacks effective interaction optimization mechanisms and conflict mitigation mechanisms.

[0006] The first aspect of this application provides an end-to-end pedestrian perception method based on spatiotemporal data association, comprising the following steps: generating a target spatiotemporal change original feature map that fuses features from the current time and historical time based on sensor data at the current time; generating specialized feature maps applicable to pedestrian detection sub-tasks, target tracking sub-tasks, intent recognition sub-tasks, and trajectory prediction sub-tasks based on the target spatiotemporal change original feature map; constructing a feature association and interaction optimization mechanism between sub-tasks based on the task-specific features in the specialized feature map; and generating the final pedestrian perception result using the feature association and interaction optimization mechanism.

[0007] Optionally, in one embodiment of this application, generating the target spatiotemporal change original feature map based on the sensor data at the current moment, which integrates the features of the current moment and historical moments, includes: using a preset end-to-end pedestrian perception spatiotemporal feature extraction module and a spatiotemporal attention module built based on a recurrent neural network, so that the network adaptively selects key moment and key location features from historical features to generate spatiotemporal features, thereby generating the target spatiotemporal change original feature map.

[0008] Optionally, in one embodiment of this application, generating a specialized feature map applicable to the pedestrian detection subtask, target tracking subtask, intent recognition subtask, and trajectory prediction subtask based on the original feature map of the target time-space change includes: generating task-specific features based on the end-to-end pedestrian perception spatiotemporal features of the original feature map of the target spatiotemporal change, wherein the generated task-specific features include pedestrian detection features, target tracking features, intent recognition features, and trajectory prediction features.

[0009] Optionally, in one embodiment of this application, the step of constructing a feature association and interaction optimization mechanism between subtasks based on the task-specific features in the specialized feature map, and generating the final pedestrian perception result using the feature association and interaction optimization mechanism, includes: setting a first feature and a second feature; using a preset formula to perform feature association and interaction optimization on the first feature and the second feature to generate pedestrian detection results, target tracking results, intent recognition results, and trajectory prediction results, so as to obtain the final pedestrian perception result.

[0010] Optionally, in one embodiment of this application, the preset formula is:

[0011] g h =σ(CL) h (F1))

[0012] g w =σ(CL) w (F1))

[0013] IA(i,j)=F1(i,j)×g h (i)×g w (j)

[0014] γ(F1, F2) = F2 + IA

[0015] Among them, g h Let σ(·) be the attention weight in the height direction, and CL be the Sigmoid activation function. h and CL w All are corresponding 1×1 convolutional layers, where h represents the height direction, w represents the width direction, F1 is the first feature, F2 is the second feature, and g is the second feature. wy is the attention weight in the width direction, IA(i,j) is the interaction attention weight at position (i,j), i and j are the indices of the positions in the feature map, and γ(F1,F2) is the feature association and interaction optimization process.

[0016] A second aspect of this application provides an end-to-end pedestrian perception device based on spatiotemporal data association, comprising: a first generation module, configured to generate an original feature map of target spatiotemporal changes that fuses features from the current time and historical times based on sensor data at the current time; a second generation module, configured to generate specialized feature maps applicable to pedestrian detection sub-tasks, target tracking sub-tasks, intent recognition sub-tasks, and trajectory prediction sub-tasks based on the original feature map of target spatiotemporal changes; and a perception module, configured to construct a feature association and interaction optimization mechanism between sub-tasks based on the task-specific features in the specialized feature map, and generate the final pedestrian perception result using the feature association and interaction optimization mechanism.

[0017] Optionally, in one embodiment of this application, the first generation module is further configured to utilize a preset end-to-end pedestrian perception spatiotemporal feature extraction module, and a spatiotemporal attention module constructed based on a recurrent neural network, so that the network adaptively selects key moment and key location features in historical features to generate spatiotemporal features, thereby generating the target spatiotemporal change original feature map.

[0018] Optionally, in one embodiment of this application, the second generation module is further configured to generate task-specific features based on the end-to-end pedestrian perception spatiotemporal features of the original feature map of the target spatiotemporal change, wherein the generated task-specific features include pedestrian detection features, target tracking features, intent recognition features, and trajectory prediction features.

[0019] Optionally, in one embodiment of this application, the perception module includes: a feature unit for setting a first feature and a second feature; and a generation unit for performing feature association and interaction optimization on the first feature and the second feature using a preset formula to generate pedestrian detection results, target tracking results, intent recognition results, and trajectory prediction results, so as to obtain the final pedestrian perception result.

[0020] Optionally, in one embodiment of this application, the preset formula is:

[0021] g h =σ(CL) h (F1))

[0022] g w =σ(CL) w (F1))

[0023] IA(i,j)=F1(i,j)×g h (i)×gw (j)

[0024] γ(F1, F2) = F2 + IA

[0025] Among them, g h Let σ(·) be the attention weight in the height direction, and CL be the Sigmoid activation function. h and CL w All are corresponding 1×1 convolutional layers, where h represents the height direction, w represents the width direction, F1 is the first feature, F2 is the second feature, and g is the second feature. w y is the attention weight in the width direction, IA(i,j) is the interaction attention weight at position (i,j), i and j are the indices of the positions in the feature map, and γ(F1,F2) is the feature association and interaction optimization process.

[0026] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the end-to-end pedestrian sensing method based on spatiotemporal data association as described in the above embodiments.

[0027] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described end-to-end pedestrian perception method based on spatiotemporal data association.

[0028] This application's embodiments generate a target spatiotemporal change original feature map that integrates features from the current and historical moments, establishing a spatiotemporal feature extraction method that can be shared among different sub-tasks. This solves the problem of feature sharing among multiple tasks with different time domain length requirements, improving the real-time performance and accuracy of detection. By generating specialized feature maps for different sub-tasks, an adaptive feature decoupling method for generating task-specific features is established, resolving the problem of conflicting features required by different tasks in a single network and reducing mutual influence between different tasks in the same network. By constructing feature association and interaction optimization mechanisms between sub-tasks, the feature association and interaction optimization problems between different tasks are solved, ultimately generating the final pedestrian perception result. Therefore, this addresses the problems in related technologies where serializing sub-tasks lacks an effective spatiotemporal feature extraction method, leading to duplicate extracted target features and poor real-time accuracy, while using a single model lacks effective interaction optimization and conflict mitigation mechanisms.

[0029] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0030] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0031] Figure 1 This is a flowchart of an end-to-end pedestrian perception method based on spatiotemporal data association provided according to an embodiment of this application;

[0032] Figure 2 A flowchart illustrating the construction of a task-specific feature decoupling module according to one embodiment of this application;

[0033] Figure 3 This is a schematic diagram illustrating the structure for constructing feature association and interaction optimization according to an embodiment of this application;

[0034] Figure 4 This is a flowchart illustrating the generation results of each subtask based on a feature association and interaction optimization mechanism according to an embodiment of this application;

[0035] Figure 5 This is a block diagram illustrating the working principle of an end-to-end pedestrian perception method based on spatiotemporal data association according to an embodiment of this application.

[0036] Figure 6 This is a schematic diagram of the structure of an end-to-end pedestrian sensing device based on spatiotemporal data association according to an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0038] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0039] The following description, with reference to the accompanying drawings, describes an end-to-end pedestrian perception method and apparatus based on spatiotemporal data association according to embodiments of this application. Addressing the issues mentioned in the background art, such as the lack of effective spatiotemporal feature extraction methods when serializing subtasks, leading to repetitive extracted target features and poor real-time accuracy, and the lack of effective interaction optimization and conflict mitigation mechanisms when using a single model, this application provides an end-to-end pedestrian perception method based on spatiotemporal data association. In this method, by generating an original feature map of target spatiotemporal changes that integrates features from the current and historical moments, a spatiotemporal feature extraction method that can be shared among different subtasks is established to solve the problem of feature sharing among multiple tasks with different time domain length requirements, improving the real-time performance and accuracy of detection. By generating specialized feature maps for different subtasks, an adaptive feature decoupling method for generating task-specific features is established to solve the problem of conflicting feature requirements for different tasks in a single network, reducing mutual influence between different tasks in the same network. By constructing feature association and interaction optimization mechanisms between subtasks, the feature association and interaction optimization problems between different tasks are solved, thereby generating the final pedestrian perception result. This solves the problems in related technologies, such as the lack of effective spatiotemporal feature extraction methods when serializing subtasks, resulting in repeated extraction of target features and poor real-time accuracy, while using a single model lacks effective interactive optimization mechanisms and conflict mitigation mechanisms.

[0040] Specifically, Figure 1 This is a flowchart illustrating an end-to-end pedestrian perception method based on spatiotemporal data association, provided in an embodiment of this application.

[0041] like Figure 1 As shown, this end-to-end pedestrian perception method based on spatiotemporal data association includes the following steps:

[0042] In step S101, based on the sensor data at the current moment, an original feature map of the target's spatiotemporal changes is generated, which integrates the features of the current moment and historical moments.

[0043] It is understood that the sensors can include multiple different types of sensors, including but not limited to cameras, camcorders, LiDAR, and millimeter-wave radar. Furthermore, based on the sensor data at the current moment, the data can be from a single camera, a single radar, a fused dataset of a single camera and a single radar, or data from multiple cameras, multiple radars, or fused datasets of one or more cameras and one or more radars. This application does not impose specific limitations.

[0044] As one possible approach, embodiments of this application can generate an original feature map of the spatiotemporal changes of the target by fusing sensor data features at the current moment and sensor data features at historical moments, thereby solving the problem of feature sharing for different time domain length requirements.

[0045] For example, in this application embodiment, camera A can be used to continuously capture images. For the image data obtained at the current moment, the image data of the previous ten frames at the current moment can be used as the image data of the previous moment. It can be, but is not limited to, the image data of the previous ten frames. This application does not impose any restrictions. Furthermore, the original feature map of the spatiotemporal changes of the target can be generated by fusing the features of the image data at the current moment and the features of the image data of the previous ten frames at the previous moment.

[0046] This application embodiment solves the problem of feature sharing among multiple tasks for different time domain length requirements by generating a target spatiotemporal change original feature map that integrates features of the current time and historical time, thereby improving the real-time performance and accuracy of pedestrian detection.

[0047] Optionally, in one embodiment of this application, a target spatiotemporal change original feature map is generated based on sensor data at the current moment, which integrates features from the current moment and historical moments. This includes: using a preset end-to-end pedestrian perception spatiotemporal feature extraction module and a spatiotemporal attention module built on a recurrent neural network, so that the network adaptively selects key moment and key location features from historical features to generate spatiotemporal features, thereby generating the target spatiotemporal change original feature map.

[0048] It is understood that the process of constructing the spatiotemporal attention module in the embodiments of this application can be as follows:

[0049] r t =σ(W r ·[f t h t-1 ])

[0050] z t =σ(W z ·[f t h t-1 ])

[0051] n t =tanh(W n ·[f t r t *h t-1 ])

[0052] h t =(1-z) t )*n t +z t *h t-1

[0053] Where r is the reset gate, z is the update gate, n is the add gate, h is the hidden state, and r t Let σ(·) be the update value of the gate reset at time t, and W be the Sigmoid activation function. r W Z W n These are the weight parameters for resetting the door, updating the door, and adding the door, respectively. t-1 Let z be the hidden state at time t-1. t Let n be the value of the gate updated at time t. t h represents the update value of the gate added at time t. t Let f be the hidden state updated at time t. t Let be the sensor data characteristics at time t.

[0054] Furthermore, let δ(·) be the feature extractor for sensor data at a single moment. Then, for sensor data x at the current moment t... t The sensor data features f at the current moment are generated by the feature extractor. t .

[0055] This application embodiment iterates from time t-TI to time t-1, and finally generates the end-to-end pedestrian perception spatiotemporal features s corresponding to the current time t. t f t =h t Where TI is the temporal length of the spatiotemporal attention module.

[0056] This application addresses the lack of effective spatiotemporal feature extraction methods in related technologies, which makes it difficult to combine tracking tasks that rely on short-term data with prediction tasks that rely on long-term data. It constructs a pre-defined end-to-end pedestrian perception spatiotemporal feature extraction module. Specifically, through a spatiotemporal attention module built based on a recurrent neural network, the network adaptively selects key moment and key location features from historical features to generate spatiotemporal features, generating the original feature map of the target spatiotemporal changes, and realizing the modeling of the spatiotemporal changes of pedestrian targets in complex traffic environments.

[0057] In step S102, a specialized feature map suitable for pedestrian detection subtask, target tracking subtask, intent recognition subtask, and trajectory prediction subtask is generated based on the original feature map of target time change.

[0058] Understandably, the pedestrian perception task can currently be divided into four subtasks: pedestrian detection, target tracking, intent recognition, and trajectory prediction.

[0059] As one possible implementation method, embodiments of this application can generate four subtasks applicable to pedestrian detection, target tracking, intent recognition, and trajectory prediction based on the original feature map of the target time-varying event.

[0060] Optionally, in one embodiment of this application, generating a specialized feature map applicable to the pedestrian detection subtask, target tracking subtask, intent recognition subtask, and trajectory prediction subtask based on the original feature map of target time-space change includes: generating task-specific features based on the end-to-end pedestrian perception spatiotemporal features of the original feature map of target spatiotemporal change, wherein the generated task-specific features include pedestrian detection features, target tracking features, intent recognition features, and trajectory prediction features.

[0061] It is understood that, in the embodiments of this application, an end-to-end pedestrian perception spatiotemporal feature (STF) is generated after step S101. t According to the generated STF t Lightweight task-specific feature decoupling modules based on self-attention mechanisms can be constructed, such as... Figure 2 As shown.

[0062] Furthermore, this module can generate task-specific features. Specifically, pedestrian detection features Target tracking features Intent recognition features Trajectory prediction features This solves the problem of competition between different task feature extractions.

[0063] Specifically, the process of constructing the task-specific feature decoupling module in this embodiment of the application can be as follows:

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] in, Let be the initial features for task j at time t, Q be the task query features, K be the task key features, V be the task value features, and conv be the convolution function. Let t be the task query features for task j. The weight parameters for task j at time t are the task query features, task key features, and task value features. Let t be the task key feature for task j. for The corresponding task attention for task j at time t, where softmax is a normalized exponential function, and c K for The number of channels, Let t be the task value feature for task j. Let t represent the task decoupling characteristics for task j at time t.

[0071] This application embodiment can solve the problem of conflicting features of different tasks in a single network by establishing a lightweight task-specific feature decoupling module based on a self-attention mechanism, thereby reducing the mutual influence between different tasks in the same network and improving the accuracy of pedestrian perception.

[0072] In step S103, a feature association and interaction optimization mechanism between subtasks is constructed based on the task-specific features in the specialized feature map, and the final pedestrian perception result is generated using the feature association and interaction optimization mechanism.

[0073] In some embodiments, a feature association and interaction optimization mechanism between subtasks can be constructed based on the task-specific features in the specialized feature map, thereby generating the final pedestrian perception result.

[0074] The embodiments of this application can perform feature association based on the shared features of each sub-task, and optimize the feature interaction of each sub-task to realize feature sharing and feature-based interaction optimization between different tasks, thereby achieving end-to-end optimization of the entire pedestrian perception system.

[0075] Optionally, in one embodiment of this application, a feature association and interaction optimization mechanism between subtasks is constructed based on the task-specific features in the specialized feature map, and the final pedestrian perception result is generated using the feature association and interaction optimization mechanism. This includes: setting a first feature and a second feature; using a preset formula to perform feature association and interaction optimization on the first feature and the second feature to generate pedestrian detection results, target tracking results, intent recognition results, and trajectory prediction results, so as to obtain the final pedestrian perception result.

[0076] As one possible implementation method, embodiments of this application can use a constructed multi-task interaction optimization module to perform feature association and interaction optimization on different features one and feature two, thereby generating the results of each sub-task and outputting the final pedestrian perception result.

[0077] For example, the feature association and interaction optimization process in the embodiments of this application can be as follows: Figure 3 As shown, the constructed multi-task interaction optimization module will be described in detail below. Furthermore, based on feature association and interaction optimization mechanisms, the generation process of each subtask can be as follows: Figure 4 As shown, the detailed steps are as follows:

[0078] Step S1: Pedestrian detection subtask branch results. Based on the pedestrian detection features at time t. After processing by the fully connected layer, the pedestrian detection result DR is obtained. (t) ={dr1 (t) dr2 (t) , ..., dr n (t)} Where n represents the number of targets in the image at time t. Let be the bounding box of pedestrian target i at time t. Let be the confidence level of pedestrian target i at time t.

[0079] Step S2: Target tracking subtask branch results. Pedestrian detection features based on time t. Target tracking features Features were obtained through feature association and interaction optimization. The target tracking result is obtained after processing by a fully connected layer. in, Let t be the ID (Identity Document) of the trajectory at time t. Let be the historical trajectory point corresponding to pedestrian target i in the image at time t.

[0080] Step S3: Consciousness recognition subtask branch results. Based on target tracking features at time t. Intent recognition features Features were obtained through feature association and interaction optimization. The intent recognition result is obtained after processing by the fully connected layer. in, Let t represent the intention of the i-th pedestrian target.

[0081] Step S4: Trajectory prediction subtask branch results. Based on target tracking features at time t. Intent recognition features Trajectory prediction features Features were obtained through feature association and interaction optimization. The trajectory prediction result is obtained after processing by the fully connected layer. Among them, pr i (t+s) Let be the predicted position of the i-th pedestrian target at time t+s, where s is the prediction duration.

[0082] Optionally, in one embodiment of this application, the preset formula is:

[0083] g h=σ(CL) h (F1))

[0084] g w =σ(CL) w (F1))

[0085] IA(i,j)=F1(i,j)×g h (i)×g w (j)

[0086] γ(F1, F2) = F2 + IA

[0087] Among them, g h Let σ(·) be the attention weight in the height direction, and CL be the Sigmoid activation function. h and CL w All are corresponding 1×1 convolutional layers, where h represents the height direction, w represents the width direction, F1 is the first feature, F2 is the second feature, and g is the second feature. w y is the attention weight in the width direction, IA(i,j) is the interaction attention weight at position (i,j), i and j are the indices of the positions in the feature map, and γ(F1,F2) is the feature association and interaction optimization process.

[0088] As one possible implementation method, the multi-task interaction optimization module constructed in this application is described in detail here, and can be specifically represented as follows:

[0089] g h =σ(CL) h (F1))

[0090] g w =σ(CL) w (F1))

[0091] IA(i,j)=F1(i,j)×g h (i)×g w (j)

[0092] γ(F1, F2) = F2 + IA

[0093] Among them, g h Let σ(·) be the attention weight in the height direction, and CL be the Sigmoid activation function. h and CL w All are corresponding 1×1 convolutional layers, where h represents the height direction, w represents the width direction, F1 is the first feature, F2 is the second feature, and g is the second feature. w y is the attention weight in the width direction, IA(i,j) is the interaction attention weight at position (i,j), i and j are the indices of the positions in the feature map, and γ(F1,F2) is the feature association and interaction optimization process.

[0094] Combination Figure 5 As shown, the working principle of an end-to-end pedestrian perception method based on spatiotemporal data association according to an embodiment of this application will be described in detail below:

[0095] like Figure 5 As shown, the embodiments of this application comprise three parts, the specific contents of which are as follows:

[0096] Step S1: Construct an end-to-end pedestrian perception spatiotemporal feature extraction module. Specifically, a spatiotemporal attention module is constructed based on a recurrent neural network, which is used to adaptively select key moment and key location features from historical features to generate spatiotemporal features, thereby realizing the modeling of spatiotemporal changes of pedestrian targets in complex traffic environments.

[0097] Specifically, if the feature extractor for sensor data at a single moment is σ(·), then for the sensor data x at the current moment... t The current time feature generated by the feature extractor is f. t The spatiotemporal attention module integrates current-moment features and historical features to generate end-to-end pedestrian perception spatiotemporal features (STF) for the current moment. t The specific process of the spatiotemporal attention module is as follows:

[0098] r t =σ(W r ·[f t h t-1 ])

[0099] z t =σ(W z ·[f t h t-1 ])

[0100] n t =tanh(W n ·[f t r t *h t-1 ])

[0101] h t =(1-z) t )*n t +z t *h t-1

[0102] Where r is the reset gate, z is the update gate, n is the add gate, h is the hidden state, and r t Let σ(·) be the update value of the gate reset at time t, and W be the Sigmoid activation function. r W z W n These are the weight parameters for resetting the door, updating the door, and adding the door, respectively.t-1 Let z be the hidden state at time t-1. t Let n be the value of the gate updated at time t. t h represents the update value of the gate added at time t. t Let f be the hidden state updated at time t. t Let be the sensor data characteristics at time t.

[0103] This application embodiment iterates from time t-TI to time t-1, and finally generates the end-to-end pedestrian perception spatiotemporal feature (STF) corresponding to the current time t. t =h t Where TI is the temporal length of the spatiotemporal attention module.

[0104] Step S2: Construct a task-specific feature decoupling module. Specifically, this is a lightweight task-specific feature decoupling module based on a self-attention mechanism, which generates task-specific features, including pedestrian detection features, target tracking features, intent recognition features, and trajectory prediction features, thereby resolving the competition problem between feature extraction from different tasks. The specific process is as follows: Figure 2 As shown, the detailed process is as follows:

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111] in, Let be the initial features for task j at time t, Q be the task query features, K be the task key features, V be the task value features, and conv be the convolution function. Let t be the task query features for task j. The weight parameters for task j at time t are the task query features, task key features, and task value features. Let t be the task key feature for task j. for The corresponding task attention for task j at time t, where softmax is a normalized exponential function and cK is... The number of channels, Let t be the task value feature for task j. Let t represent the task decoupling characteristics for task j at time t.

[0112] Step S3: Construct a multi-task interaction optimization module. This application proposes a multi-task interaction optimization module for pedestrian perception, wherein the process of feature association and interaction optimization is as follows: Figure 3 As shown. Through the multi-task interaction optimization module, adaptive selection of required features from F1 and association with F2 can be achieved. The multi-task interaction optimization module constructed in this application is described in detail below, and can be specifically represented as follows:

[0113] g h =σ(CL) h (F1))

[0114] g w =σ(CL) w (F1))

[0115] IA(i,j)=F1(i,j)×g h (i)×g w (j)

[0116] γ(F1, F2) = F2 + IA

[0117] Among them, g h Let σ(·) be the attention weight in the height direction, and CL be the Sigmoid activation function. h and CL w All are corresponding 1×1 convolutional layers, where h represents the height direction, w represents the width direction, F1 is the first feature, F2 is the second feature, and g is the second feature. w y is the attention weight in the width direction, IA(i,j) is the interaction attention weight at position (i,j), i and j are the indices of the positions in the feature map, and γ(F1,F2) is the feature association and interaction optimization process.

[0118] Based on the aforementioned feature association and interaction optimization mechanism, the process for generating results for each branch is as follows: Figure 4 As shown, the detailed process is as follows:

[0119] Step S1: Pedestrian detection subtask branch results. Based on the pedestrian detection features at time t. After processing by the fully connected layer, the pedestrian detection result DR is obtained. (t) ={dr1 (t) dr2 (t) , ..., dr n (t)} Where n represents the number of targets in the image at time t. Let be the bounding box of pedestrian target i at time t. Let be the confidence level of pedestrian target i at time t.

[0120] Step S2: Target tracking subtask branch results. Pedestrian detection features based on time t. Target tracking features Features were obtained through feature association and interaction optimization. The target tracking result is obtained after processing by a fully connected layer. in, Let t be the ID of the trajectory. Let be the historical trajectory point corresponding to pedestrian target i in the image at time t.

[0121] Step S3: Consciousness recognition subtask branch results. Based on target tracking features at time t. Intent recognition features Features were obtained through feature association and interaction optimization. The intent recognition result is obtained after processing by the fully connected layer. in, Let t represent the intention of the i-th pedestrian target.

[0122] Step S4: Trajectory prediction subtask branch results. Based on target tracking features at time t. Intent recognition features Trajectory prediction features Features were obtained through feature association and interaction optimization. The trajectory prediction result is obtained after processing by the fully connected layer. Among them, pr i (t+s) Let be the predicted position of the i-th pedestrian target at time t+s, where s is the prediction duration.

[0123] In summary, the embodiments of this application first input the sensor data at the current moment into the spatiotemporal feature extraction module to generate an original feature map of the target spatiotemporal change that integrates features from the current moment and historical moments; then input the original feature map into the task-specific feature decoupling module to generate specialized feature maps applicable to four sub-tasks: pedestrian detection, target tracking, intent recognition, and trajectory prediction; finally, based on the task-specific features, a feature association and interaction optimization mechanism between the four sub-tasks is constructed to generate the final pedestrian perception result.

[0124] The end-to-end pedestrian perception method based on spatiotemporal data association proposed in this application generates an original feature map of target spatiotemporal changes that integrates features from the current and historical moments. It establishes a spatiotemporal feature extraction method that can be shared among different sub-tasks, solving the problem of feature sharing among multiple tasks with different time domain length requirements, thus improving the real-time performance and accuracy of detection. Furthermore, it generates specialized feature maps for different sub-tasks and establishes an adaptive feature decoupling method for generating task-specific features, resolving the conflict between features required by different tasks in a single network and reducing mutual influence between different tasks within the same network. Finally, it constructs a feature association and interaction optimization mechanism between sub-tasks to address the feature association and interaction optimization problems between different tasks, thereby generating the final pedestrian perception result. This solves the problems in related technologies where serializing sub-tasks lacks an effective spatiotemporal feature extraction method, leading to duplicate extracted target features and poor real-time accuracy, while using a single model lacks an effective interaction optimization mechanism and conflict mitigation mechanism.

[0125] Next, referring to the accompanying drawings, an end-to-end pedestrian sensing device based on spatiotemporal data association proposed according to an embodiment of this application is described.

[0126] Figure 6 This is a block diagram of an end-to-end pedestrian sensing device based on spatiotemporal data association, according to an embodiment of this application.

[0127] like Figure 6 As shown, the end-to-end pedestrian sensing device 10 based on spatiotemporal data association includes: a first generation module 100, a second generation module 200, and a sensing module 300.

[0128] Specifically, the first generation module 100 is used to generate an original feature map of the spatiotemporal change of the target that integrates features from the current moment and historical moments based on the sensor data at the current moment.

[0129] The second generation module 200 is used to generate specialized feature maps suitable for pedestrian detection subtasks, target tracking subtasks, intent recognition subtasks, and trajectory prediction subtasks based on the original feature map of target time-varying changes.

[0130] The perception module 300 is used to construct a feature association and interaction optimization mechanism between subtasks based on the task-specific features in the specialized feature map, and to generate the final pedestrian perception result using the feature association and interaction optimization mechanism.

[0131] Optionally, in one embodiment of this application, the first generation module 100 is further configured to utilize a preset end-to-end pedestrian perception spatiotemporal feature extraction module, and a spatiotemporal attention module constructed based on a recurrent neural network, so that the network adaptively selects key moment and key location features in historical features to generate spatiotemporal features and generate the target spatiotemporal change original feature map.

[0132] Optionally, in one embodiment of this application, the second generation module 200 is further configured to generate task-specific features based on the end-to-end pedestrian perception spatiotemporal features of the original feature map of the target spatiotemporal change, wherein the generated task-specific features include pedestrian detection features, target tracking features, intent recognition features and trajectory prediction features.

[0133] Optionally, in one embodiment of this application, the sensing module 300 includes a feature unit and a generation unit.

[0134] The feature unit is used to set the first feature and the second feature.

[0135] The generation unit is used to perform feature association and interaction optimization on the first feature and the second feature using a preset formula to generate pedestrian detection results, target tracking results, intent recognition results and trajectory prediction results, so as to obtain the final pedestrian perception results.

[0136] Optionally, in one embodiment of this application, the preset formula is:

[0137] g h =σ(CL) h (F1))

[0138] g w =σ(CL) w (F1))

[0139] IA(i,j)=F1(i,j)×g h (i)×g w (j)

[0140] γ(F1, F2) = F2 + IA

[0141] Among them, g h Let σ(·) be the attention weight in the height direction, and CL be the Sigmoid activation function. h and CL w All are corresponding 1×1 convolutional layers, where h represents the height direction, w represents the width direction, F1 is the first feature, F2 is the second feature, and g is the second feature. w y is the attention weight in the width direction, IA(i,j) is the interaction attention weight at position (i,j), i and j are the indices of the positions in the feature map, and γ(F1,F2) is the feature association and interaction optimization process.

[0142] It should be noted that the foregoing explanation of the end-to-end pedestrian sensing method embodiment based on spatiotemporal data association also applies to the end-to-end pedestrian sensing device based on spatiotemporal data association in this embodiment, and will not be repeated here.

[0143] The end-to-end pedestrian sensing device based on spatiotemporal data association proposed in this application generates an original feature map of target spatiotemporal changes that integrates features from the current and historical moments. It establishes a spatiotemporal feature extraction method that can be shared among different sub-tasks, solving the problem of feature sharing among multiple tasks with different time domain length requirements, thus improving the real-time performance and accuracy of detection. Furthermore, it generates specialized feature maps for different sub-tasks and establishes an adaptive feature decoupling method for generating task-specific features, resolving the conflict between features required by different tasks in a single network and reducing mutual influence between different tasks within the same network. Finally, it constructs a feature association and interaction optimization mechanism between sub-tasks to address the feature association and interaction optimization problems between different tasks, thereby generating the final pedestrian sensing result. This solves the problems in related technologies where serializing sub-tasks lacks an effective spatiotemporal feature extraction method, leading to duplicate extracted target features and poor real-time accuracy, while using a single model lacks an effective interaction optimization mechanism and conflict mitigation mechanism.

[0144] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0145] The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0146] When the processor 702 executes the program, it implements the end-to-end pedestrian perception method based on spatiotemporal data association provided in the above embodiments.

[0147] Furthermore, electronic devices also include:

[0148] Communication interface 703 is used for communication between memory 701 and processor 702.

[0149] The memory 701 is used to store computer programs that can run on the processor 702.

[0150] The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0151] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized into address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0152] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0153] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0154] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described end-to-end pedestrian perception method based on spatiotemporal data association.

[0155] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0156] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0157] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0158] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0159] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0160] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0161] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0162] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. An end-to-end pedestrian perception method based on spatiotemporal data correlation, characterized in that, Includes the following steps: Based on sensor data at the current moment, including cameras and radar, a target spatiotemporal change original feature map is generated by fusing features from the current moment and historical moments. Based on the original feature map of the spatiotemporal changes of the target, generate specialized feature maps suitable for pedestrian detection subtask, target tracking subtask, intent recognition subtask, and trajectory prediction subtask; Based on the task-specific features in the specialized feature map, a feature association and interaction optimization mechanism is constructed between sub-tasks, and the final pedestrian perception result is generated using the feature association and interaction optimization mechanism. The generation of a target spatiotemporal change original feature map based on sensor data at the current moment, fusing features from the current moment and historical moments, includes: Using a pre-defined end-to-end pedestrian perception spatiotemporal feature extraction module and a spatiotemporal attention module built on a recurrent neural network, the network adaptively selects key moment and key location features from historical features to generate spatiotemporal features, thereby generating the original feature map of the target spatiotemporal change. The step of generating specialized feature maps suitable for pedestrian detection, target tracking, intent recognition, and trajectory prediction subtasks based on the original spatiotemporal variation feature map of the target includes: Based on the end-to-end pedestrian perception spatiotemporal features of the original feature map of the target spatiotemporal change, task-specific features are generated, wherein the generated task-specific features include pedestrian detection features, target tracking features, intent recognition features and trajectory prediction features. The mechanism for constructing feature association and interaction optimization between subtasks based on task-specific features in the specialized feature map, and using the feature association and interaction optimization mechanism to generate the final pedestrian perception result, includes: Let there be a first feature and a second feature; The first feature and the second feature are features of any two different tasks among the pedestrian detection feature, the target tracking feature, the intent recognition feature, and the trajectory prediction feature; The first feature and the second feature are correlated and interacted using a preset formula to generate pedestrian detection results, target tracking results, intent recognition results and trajectory prediction results, so as to obtain the final pedestrian perception result; The preset formula is: in, For attention weights in the height direction, For Sigmod activation function, and All are corresponding Convolutional layer Represents the height direction. Represents the width direction. As the first feature, As the second feature, The attention weights are in the width direction. for Location-based interactive attention weights, This is the index of the position in the feature map. For the feature association and interaction optimization process, For the first feature map at position The eigenvalue at that location.

2. An end-to-end pedestrian sensing device based on spatiotemporal data correlation, characterized in that, The end-to-end pedestrian perception method based on spatiotemporal data association as described in claim 1 includes: The first generation module is used to generate an original feature map of target spatiotemporal change that fuses features from the current moment and historical moments based on sensor data at the current moment, including cameras and radar. The second generation module is used to generate specialized feature maps suitable for pedestrian detection subtasks, target tracking subtasks, intent recognition subtasks, and trajectory prediction subtasks based on the original spatiotemporal change feature map of the target. The perception module is used to construct a feature association and interaction optimization mechanism between subtasks based on the task-specific features in the specialized feature map, and to generate the final pedestrian perception result using the feature association and interaction optimization mechanism.

3. The apparatus of claim 2, wherein, The first generation module is further used to utilize a preset end-to-end pedestrian perception spatiotemporal feature extraction module and a spatiotemporal attention module built based on a recurrent neural network to enable the network to adaptively select key moment and key location features from historical features to generate spatiotemporal features and generate the original feature map of the target spatiotemporal change.

4. The apparatus of claim 2, wherein, The second generation module is further used to generate task-specific features based on the end-to-end pedestrian perception spatiotemporal features of the original feature map of the target spatiotemporal change, wherein the generated task-specific features include pedestrian detection features, target tracking features, intent recognition features and trajectory prediction features.

5. An electronic device, comprising: include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the end-to-end pedestrian perception method based on spatiotemporal data association as described in claim 1.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the end-to-end pedestrian perception method based on spatiotemporal data association as described in claim 1.

Images