Cooperative joint perception prediction method based on continuous space-time deduction

Abstract

The application relates to a collaborative joint perception prediction method based on continuous space-time deduction, wherein the method comprises the following steps: acquiring local scene observation characteristics collected by multiple intelligent agents and corresponding collection time points; constructing an asynchronous observation event sequence according to the order of the collection time points, and constructing a global continuous scene state, continuously performing space-time deduction on the global continuous scene state between adjacent observation events to obtain a global continuous scene deduction state; when the next event time point is reached, generating a candidate correction state according to the corresponding local scene observation characteristics and the global continuous scene deduction state, constructing a time-aware update weight, and performing time-gated weighted updating on the global continuous scene deduction state until the global continuous scene state at the current decision time point is obtained, generating a scene state representation at a future time point, and obtaining a collaborative perception and motion prediction result. Therefore, the problems that related technologies are difficult to cope with real asynchronous input and affect the collaborative perception and prediction effect are solved.

Description

Technical Field

[0001] This application relates to the field of cooperative perception and prediction technology for autonomous driving, and in particular to a cooperative joint perception and prediction method based on continuous spatiotemporal extrapolation. Background Technology

[0002] Existing single-vehicle autonomous driving systems often face challenges in complex traffic scenarios, such as limited sensor range and perception occlusion, making it difficult to establish accurate environmental awareness and thus restricting the safety and reliability of subsequent decision-making. Collaborative technologies, however, can improve the completeness and accuracy of autonomous vehicles' perception of the overall scene by sharing perception information from other vehicles or roadside facilities. Building on this, further research has combined temporal modeling methods to achieve collaborative perception and prediction, providing a more comprehensive and accurate model of the current and future evolution of the global scene.

[0003] In related technologies, the collaborative joint perception and prediction method is based on the time alignment assumption of synchronous sampling and fixed frame rate. It requires different agents to provide perception information at the same time point and complete feature space fusion at the frame level, so as to realize the perception and prediction of the global scene.

[0004] However, in related technologies, due to the inconsistency in the sensor sampling frequency, acquisition time, and communication transmission delay of different agents in actual collaborative scenarios, the observation information of each agent is naturally asynchronous. This makes the collaborative joint perception and prediction method based on the time alignment assumption prone to information mismatch when processing asynchronous perception information, leading to semantic understanding errors and motion inference errors, which seriously affects the collaborative perception and prediction effect and urgently needs to be improved. Summary of the Invention

[0005] This application provides a collaborative joint perception prediction method based on continuous spatiotemporal inference to solve the problems in related technologies, such as information mismatch when processing real asynchronous perception information, which leads to semantic understanding errors and motion inference errors, seriously affecting the collaborative perception and prediction effect.

[0006] This application provides a collaborative joint perception and prediction method based on continuous spatiotemporal extrapolation, comprising the following steps: acquiring local scene observation representations collected by multiple agents and their corresponding acquisition times; constructing an asynchronous observation event sequence according to the order of the acquisition times, and constructing a global continuous scene state based on the asynchronous observation event sequence and the local scene observation representations, and performing continuous spatiotemporal extrapolation on the global continuous scene state between adjacent observation events to obtain a global continuous scene extrapolation state; when the next event time is reached, generating a candidate correction state based on the corresponding local scene observation representation and the global continuous scene extrapolation state, and constructing a time-aware update weight by combining a time mask, performing time-gated weighted update on the global continuous scene extrapolation state to obtain the global continuous scene state after the event update, until all events of the asynchronous observation event sequence are processed to obtain the global continuous scene state at the current decision time; and generating a scene state representation for future times based on the global continuous scene state at the current decision time to obtain collaborative perception and motion prediction results.

[0007] Through the above-mentioned technical means, the embodiments of this application can obtain local scene observation representations collected by multiple intelligent agents, and construct an asynchronous observation event sequence according to the order of collection time, so as to construct a global continuous scene state under a unified reference coordinate system, and perform continuous spatiotemporal inference on the global continuous scene state between adjacent observation events. When the next event time is reached, the global continuous scene inference state is updated by time-gated weighting by combining the candidate correction state and the time-aware update weight, so as to obtain the scene state representation at the current decision time and the future time. After decoding, the collaborative perception and motion prediction results are obtained, which can be used to process collaborative joint perception prediction under the asynchronous input conditions of multiple intelligent agents, improve the accuracy and asynchronous robustness of the collaborative perception results at the current time and the prediction results at the future time, and effectively adapt to collaborative scenarios with communication delay, asynchronous sampling and dynamic target interaction.

[0008] Optionally, in one embodiment of this application, the step of continuously extrapolating the global continuous scene state between adjacent observation events to obtain the global continuous scene extrapolation state includes: generating a extrapolation driving force input based on the global continuous scene state at the start of each integration time step; calculating the unit time change of the global continuous scene state based on the extrapolation driving force input, performing residual state evolution using the unit time change, determining the global continuous scene state at the end of the integration step, until reaching the next event time, and determining the global continuous scene extrapolation state based on the global continuous scene state at the next event time.

[0009] Through the above-mentioned technical means, the embodiments of this application can dynamically generate the inference driving force input according to the current scene state in each integral time step between adjacent observation events, and calculate the unit time change of the scene state. The continuous scene inference state of the whole domain is obtained through residual evolution. Thus, through continuous spatiotemporal inference, the continuous evolution process of the collaborative scene in the time dimension is characterized, the accuracy of the whole domain scene inference results is improved, and a reliable state benchmark is provided for the subsequent observation and correction process.

[0010] Optionally, in one embodiment of this application, the expression for the state evolution at each integration time step is: ,

[0011] in, The global continuous scene state at the start of the integration step. This represents the change in the state of the entire continuous scene per unit time. The driving force input for the deduction is... For state deduction function, For the integration time step, This refers to the global continuous scene state at the end of the integration step.

[0012] Through the above-mentioned technical means, the embodiments of this application can continuously extrapolate the state of the entire continuous scene through residual state evolution to obtain the extrapolated state of the entire continuous scene, thereby ensuring the stability of the extrapolation process and adapting to different event time intervals, thus having the ability to handle various asynchronous observation forms.

[0013] Optionally, in one embodiment of this application, generating a candidate correction state based on the corresponding local scene observation representation and the global continuous scene deduction state includes: determining the corresponding local scene observation representation based on the next event time; and inputting the local scene observation representation and the global continuous scene deduction state into the target convolutional network to output the candidate correction state.

[0014] Through the above technical means, the embodiments of this application can input the local scene observation representation and the global continuous scene inference state corresponding to the next event moment into the convolutional network for semantic space alignment, thereby obtaining candidate correction states with spatial semantic consistency, making the generated correction quantity more accurately reflect scene changes, and providing a high-quality correction basis for subsequent updates.

[0015] Optionally, in one embodiment of this application, the formula for time-gated weighted updating of the global continuous scene deduction state is: , , in, The time-aware update weights, A time-aware function for calculating update weights; For the sigmoid function, The time mask, This refers to the candidate correction state. This refers to the state of the continuous scenario across the entire domain. To utilize The state of the entire continuous scene is updated at any given moment.

[0016] Through the above technical means, the embodiments of this application can introduce a time-gated update mechanism based on time intervals, and adaptively adjust the fusion ratio of candidate correction states and global continuous scene states through time-aware update weights, thereby improving the rationality of state updates under asynchronous conditions, and thus effectively correcting scene states when asynchronous observations arrive, improving the accuracy and asynchronous robustness of collaborative perception prediction.

[0017] This application's embodiments can acquire local scene observation representations collected by multiple intelligent agents and construct an asynchronous observation event sequence according to the order of acquisition time. This allows for the construction of a global continuous scene state under a unified reference coordinate system. Continuous spatiotemporal extrapolation of the global continuous scene state is performed between adjacent observation events. When the next event time is reached, the global continuous scene extrapolation state is updated with time-gated weighting by combining candidate correction states and time-aware update weights, resulting in scene state representations for the current decision time and future times. Decoding yields collaborative perception and motion prediction results, enabling the processing of collaborative joint perception and prediction under asynchronous input conditions of multiple intelligent agents. This improves the accuracy and asynchronous robustness of the current collaborative perception results and future prediction results, effectively adapting to complex collaborative scenarios with communication delays, asynchronous sampling, and dynamic target interactions. Therefore, it solves the problems in related technologies where information mismatch easily occurs when processing real asynchronous perception information, leading to semantic understanding errors and motion inference errors, severely affecting collaborative perception and prediction performance.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0019] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1This is a flowchart of a collaborative joint sensing and prediction method based on continuous spatiotemporal inference provided in an embodiment of this application; Figure 2 This is a schematic diagram of the overall framework of a collaborative joint sensing and prediction method provided according to an embodiment of this application; Figure 3 This is a schematic diagram of two urban complex working condition perception and prediction results provided according to an embodiment of this application. Detailed Implementation

[0020] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0021] The following describes, with reference to the accompanying drawings, a collaborative joint sensing and prediction method based on continuous spatiotemporal extrapolation, according to embodiments of this application. In response to the aforementioned issues in the related technologies mentioned in the background section, such as the tendency for information mismatch to occur when processing real asynchronous sensing information, leading to semantic understanding errors and motion inference errors, which seriously affect the collaborative sensing and prediction performance, this application provides a collaborative joint sensing and prediction method based on continuous spatiotemporal extrapolation. This method acquires local scene observation representations collected by multiple agents and constructs an asynchronous observation event sequence according to the order of acquisition time. This allows for the construction of a global continuous scene state under a unified reference coordinate system. Continuous spatiotemporal extrapolation of the global continuous scene state is performed between adjacent observation events. When the next event time is reached, the global continuous scene extrapolation state is updated with time-gated weights by combining candidate correction states and time-aware update weights, obtaining scene state representations for the current decision time and future times. Decoding yields collaborative sensing and motion prediction results, enabling the processing of collaborative joint sensing and prediction under asynchronous input conditions of multiple agents. This improves the accuracy and asynchronous robustness of the current collaborative sensing results and future prediction results, effectively adapting to complex collaborative scenarios with communication delays, asynchronous sampling, and dynamic target interactions. This solves the problem in related technologies where information mismatch easily occurs when processing real asynchronous sensing information, leading to semantic understanding errors and motion inference errors, which seriously affect the collaborative sensing and prediction effects.

[0022] Specifically, Figure 1 This is a flowchart illustrating a collaborative joint sensing and prediction method based on continuous spatiotemporal extrapolation, provided in an embodiment of this application.

[0023] like Figure 1 As shown, the collaborative joint sensing and prediction method based on continuous spatiotemporal inference includes the following steps: In step S101, the local scene observation representations collected by multiple intelligent agents and their corresponding collection times are obtained.

[0024] It is understood that in the embodiments of this application, "multiple intelligent agents" refers to nodes with environmental perception capabilities, such as intelligent connected vehicles and roadside computing units. Local scene observation representations can be understood as feature representations extracted by each intelligent agent after perceiving the surrounding environment using its onboard sensors, such as feature maps or feature vectors. The acquisition time refers to the actual time when each local observation representation is sampled and recorded.

[0025] In actual implementation, the embodiments of this application can acquire local scene observation representations collected by multiple intelligent agents. and their corresponding acquisition time The local scene observation representation is subjected to coordinate transformation to be unified to the same reference coordinate system, and an asynchronous observation event sequence is constructed according to the acquisition time order. .

[0026] Specifically, after multiple intelligent agents establish communication connections within a certain range, one agent is selected as the primary agent to receive, fuse, and process information. The primary agent can be an intelligent connected vehicle, a roadside computing unit, etc. Each agent extracts local scene observation representations based on its own perception information, which can come from lidar, cameras, or a combination of multiple sensors. For the local scene observation representations uploaded by different agents, the primary agent transforms them to the same reference coordinate system based on the relative pose relationships between the agents.

[0027] The embodiments of this application can acquire local scene observation representations and their corresponding acquisition times collected by multiple intelligent agents, and transform them to the same reference coordinate system, thereby providing a unified spatiotemporal coordinate benchmark for subsequent processing. This avoids semantic misalignment and motion distortion introduced by time-aligned resampling, and eliminates differences in sensor models and sampling formats among different intelligent agents, ensuring the feasibility of subsequent global state construction and feature fusion.

[0028] In step S102, an asynchronous observation event sequence is constructed according to the order of acquisition time, and a global continuous scene state is constructed based on the asynchronous observation event sequence and local scene observation representation. The global continuous scene state is then continuously spatiotemporally extrapolated between adjacent observation events to obtain the global continuous scene extrapolation state.

[0029] It is understood that the asynchronous observation event sequence in this application embodiment refers to the set of events formed by arranging the local observations of all agents in the order of their corresponding acquisition times. The global continuous scene state can be understood as a unified state representation covering the global perception range and containing the semantics and motion information of the entire scene target. The global continuous scene state is the global state obtained by continuous deduction before each event update, utilizing the observations. Continuous spatiotemporal deduction refers to the temporal continuous evolution calculation of the global scene state based on motion laws within the time interval between two observation events, filling the state gaps in the observation time interval.

[0030] In actual implementation, the embodiments of this application can construct a global continuous scene state in a unified reference coordinate system based on an asynchronous observation event sequence, and perform continuous spatiotemporal extrapolation of the global continuous scene state between adjacent observation events to obtain the global continuous scene extrapolation state.

[0031] In the embodiments of this application, the union of the perception coverage areas of each agent can be selected as the spatial range corresponding to the global continuous scene state, or a fixed range around the main agent can be specified to construct the global continuous scene state.

[0032] Specifically, embodiments of this application can perform continuous spatiotemporal extrapolation of the global continuous scene state between adjacent observation events. Within each integration time step, a extrapolation driving force input is generated based on the current global continuous scene state, and the unit time change of the global continuous scene state is calculated. Residual state evolution is performed, and the next global continuous scene state obtained through spatiotemporal extrapolation is obtained. Spatiotemporal extrapolation continues until the next event moment in the asynchronous observation event sequence is reached.

[0033] The embodiments of this application can construct an asynchronous observation event sequence and an initial global continuous scene state, and perform continuous spatiotemporal inference within the interval between adjacent observation events to obtain the inference state at the next event moment, thereby establishing a global continuous scene state and performing continuous spatiotemporal inference. This adapts to the asynchronous characteristics of inconsistent sampling frequencies and acquisition times of various agents in real scenarios, without the need for forced alignment of sampling times, breaking through the dependence on synchronous sampling and fixed frame rates, and maintaining the continuity of the global scene state in the time dimension.

[0034] Optionally, in one embodiment of this application, continuous spatiotemporal extrapolation of the global continuous scene state between adjacent observation events is performed to obtain the global continuous scene extrapolation state, including: generating extrapolation driving force input based on the global continuous scene state at the start of each integration time step; calculating the unit time change of the global continuous scene state based on the extrapolation driving force input, using the unit time change to perform residual state evolution, determining the global continuous scene state at the end of the integration step, until the next event time is reached, and determining the global continuous scene extrapolation state based on the global continuous scene state at the next event time.

[0035] It is understood that the integration time step in the embodiments of this application... Fixed, small time steps can be selected, such as 50 ms or 100 ms in a typical sampling frequency, for numerical integration. Alternatively, the time difference between adjacent events can be used as the time step for one iteration. The integration time step can be set by those skilled in the art according to the actual situation, and no specific restrictions are imposed here. The driving force input refers to the feature input determined by the current scene state and used to drive the temporal evolution of the scene state, such as the movement trends of various traffic participants in the scene. Residual state evolution refers to updating the state by superimposing state changes on the original state to retain the basic information of the original state.

[0036] In actual implementation, the embodiments of this application can be performed at each integration time step. Within, based on the current state of the entire continuous scene. Generate driving force input for deduction In this embodiment, the deduced driving force input can be achieved by... The input is fed into a convolutional neural network to model the expectation at each spatial location. With variance Therefore, a Gaussian distribution is established for each spatial location, and random sampling is performed from it to obtain... : , , in, Representing the expectation of each spatial location With variance A set whose shape is similar to Consistency can be viewed as expected characteristics and variance characteristics; Represents a convolutional neural network; This indicates that a Gaussian distribution is established based on the expected value and variance; This represents a random sampling operation. From this, we can obtain... Consistent shape of driving force input .

[0037] Furthermore, embodiments of this application can be obtained Then, the change in the state of the entire continuous scene per unit time is calculated. The residual state evolution is performed to determine the global continuous scene state at the end of the integration step. This global continuous scene state is then used as the global continuous scene state at the start of the next integration step, and the state evolution continues. Since the next event time may not be reached after one integration step, the embodiments of this application can extend the time from the end of the integration step to the next event time in the asynchronous observation event sequence. The state of the global continuous scene at the next event moment is determined as the global continuous scene deduction state.

[0038] The state derivation function can be ConvGRU or its variants, or other neural networks with sequence modeling capabilities, with an integration time step. Fixed, small time steps can be selected, such as 50 ms or 100 ms in a typical sampling frequency, for numerical integration, or the time difference between adjacent events can be used as the time step for one iteration.

[0039] The embodiments of this application can dynamically generate the inference driving force input based on the current scene state within each integral time step between adjacent observation events, and calculate the unit time change of the scene state. The continuous scene inference state of the whole domain is obtained through residual evolution. Thus, through continuous spatiotemporal inference, the continuous evolution process of the collaborative scene in the time dimension is characterized, the accuracy of the whole domain scene inference results is improved, and a reliable state benchmark is provided for subsequent observation and correction.

[0040] Optionally, in one embodiment of this application, the expression for the state evolution at each integration time step is: ,

[0041] in, This represents the global continuous scene state at the start of the integration step. This represents the change in the state of the entire continuous scene per unit time. To simulate the driving force input, For state deduction function, For the integration time step, This represents the global continuous scene state at the end of the integration step.

[0042] In actual implementation, this embodiment of the application can calculate the unit-time change of the global continuous scene state after obtaining a driving force input with a shape consistent with the current global continuous scene state. Perform residual state evolution: , , in, Represents the state deduction function. The global continuous scene state at the end of the integration step, since the next event moment may not be reached after one integration step, can be considered in this embodiment until the end of the integration step reaches the next event moment in the asynchronous observation event sequence. The state of the global continuous scene at the next event moment is determined as the global continuous scene inference state. The state inference function can be ConvGRU and its variants, or other neural networks with sequence modeling capabilities; the integral time step... Fixed, small time steps can be selected, such as 50 ms or 100 ms in a typical sampling frequency, for numerical integration, or the time difference between adjacent events can be used as the time step for one iteration.

[0043] The embodiments of this application can continuously extrapolate the state of the entire continuous scene through residual state evolution to obtain the extrapolated state of the entire continuous scene, thereby ensuring the stability of the extrapolation process and adapting to different event time intervals, thus having the ability to handle various asynchronous observation forms.

[0044] In step S103, when the next event time is reached, a candidate correction state is generated based on the corresponding local scene observation representation and the global continuous scene inference state. The time-aware update weight is constructed by combining the time mask, and the global continuous scene inference state is updated by time-gated weighting to obtain the global continuous scene state after the event is updated. The global continuous scene state at the current decision time is obtained after all events of the asynchronous observation event sequence are processed.

[0045] It is understood that, in the embodiments of this application, the candidate correction state refers to the candidate state representation generated by combining newly arrived local observation information and the inferred state, which is used to correct the state of the continuous scene in the whole domain.

[0046] In this embodiment, the time mask is used to characterize the time elapsed since the last observation update at different spatial locations within the global continuous scene state. For locations older than the last update, the uncertainty of their corresponding states is generally higher; therefore, a higher correction weight should be assigned to the current event observation during the update. For locations closer to the last update, their corresponding states retain more of the continuous deduction results. By introducing the time mask, the time-aware update weights can adaptively adjust the fusion ratio of candidate corrected states and global continuous scene states according to the time freshness of different locations, thereby improving the rationality of state updates under asynchronous conditions. Time-gated weighted update refers to an update mechanism that weights and fuses the global continuous scene state and candidate corrected states according to the time-aware update weights.

[0047] In actual implementation, the embodiments of this application can be characterized based on local scene observations corresponding to the event time. With the continuous scenario simulation status of the whole domain Generate candidate correction state Further stitching together candidate correction states and the state of continuous scenario simulation across the entire domain Simultaneously, a time mask is constructed based on the time interval between the state of each location and the last update. A time-aware update weight is constructed for time-gated weighted updates. Continuous spatiotemporal extrapolation and time-gated weighted updates are repeatedly executed until all events in the asynchronous observation event sequence are updated, resulting in the global continuous scene state at the current decision moment. This global continuous scene state integrates local scene observation representations collected by multiple agents at different times, and through continuous spatiotemporal extrapolation and time-gated updates, forms a unified spatiotemporal state representation of the overall scene of the current collaborative region.

[0048] The embodiments of this application can complete the local observation update of an event moment in the observation sequence, continuously extrapolate the global state to the next event moment, and then perform time-weighted updates based on the latest extrapolated state and the observation representation of the next moment. After alternating through all events in the sequence, the global continuous scene state at the current decision moment is obtained.

[0049] The embodiments of this application can generate candidate correction states by fusing local observations and global inference states, and use time masks to construct update weights for time-gated weighted updates until the global continuous scene state at the current decision moment is obtained. This effectively suppresses semantic understanding errors and motion inference biases introduced by time mismatch in asynchronous input, and ensures the high accuracy and robustness of the final output collaborative perception and motion prediction results in complex asynchronous scenarios.

[0050] Optionally, in one embodiment of this application, generating a candidate correction state based on the corresponding local scene observation representation and the global continuous scene deduction state includes: determining the corresponding local scene observation representation based on the next event time; and inputting the local scene observation representation and the global continuous scene deduction state into the target convolutional network to output the candidate correction state.

[0051] It is understood that the target convolutional network in the embodiments of this application can be a convolutional GRU (Gated Recurrent Unit) network. The target convolutional network can be set by those skilled in the art according to the actual situation, and no specific restrictions are made here.

[0052] In actual implementation, the embodiments of this application can be characterized based on local scene observations corresponding to the event time. The latest simulation results show the state of the continuous scenario across the entire domain. A common input convolutional GRU network is used for semantic space alignment to obtain candidate correction states. : .

[0053] In this embodiment, the local scene observation representation corresponding to the next event moment and the global continuous scene inference state can be input into the convolutional network for semantic space alignment, thereby obtaining candidate correction states with spatial semantic consistency. This makes the generated correction quantity more accurately reflect scene changes and provides a high-quality correction basis for subsequent updates.

[0054] Optionally, in one embodiment of this application, the formula for time-gated weighted updating of the global continuous scene deduction state is: , , in, For time-aware update weights, A time-aware function for calculating update weights; For the sigmoid function, For time mask, In candidate correction state, For the continuous scenario simulation across the entire domain, To utilize The state of the entire continuous scene is updated at any given moment.

[0055] In actual implementation, the embodiments of this application can splice candidate correction states. and the state of continuous scenario simulation across the entire domain Simultaneously, a time mask is constructed based on the time interval between the state of each location and the last update. Construct time-aware update weights for time-gated weighted updates: , , Where [] represents the feature concatenation operation; Represents the time-aware update weight calculation function; Represents the sigmoid function; Represents the time-aware update weights; This indicates element-wise multiplication. For the continuous scenario simulation across the entire domain, To utilize The state of the entire continuous scene is updated at any given moment.

[0056] The embodiments of this application can introduce a time-gated update mechanism based on time intervals, and adaptively adjust the fusion ratio of candidate correction states and global continuous scene states through time-aware update weights, thereby improving the rationality of state updates under asynchronous conditions, and thus effectively correcting scene states when asynchronous observations arrive, improving the accuracy and asynchronous robustness of collaborative perception prediction.

[0057] In step S104, based on the global continuous scene state at the current decision moment, a scene state representation for future moments is generated to obtain collaborative perception and motion prediction results.

[0058] In actual implementation, the embodiments of this application use the current decision-making moment's global continuous scene state as the initial condition to continue continuous spatiotemporal extrapolation, generating future scene state representations for multiple future moments, and then decoding them to obtain collaborative perception and motion prediction results.

[0059] In the embodiments of this application, the collaborative joint perception prediction results may include occupancy semantic results and motion flow field prediction results at future times, and may also be extended to other forms of environmental perception and motion prediction results according to actual needs. Since asynchronous observation events are organized under a unified reference coordinate system, and continuous modeling and time-gated updates are performed with the continuous scene state of the entire domain as the core, it can more effectively adapt to collaborative scenarios with communication latency, asynchronous sampling, and complex dynamic target interactions.

[0060] The embodiments of this application can perform continuous spatiotemporal extrapolation based on the global continuous scene state at the current decision moment to generate scene state representations for future moments, so as to obtain collaborative perception and motion prediction results, thereby outputting more accurate and complete perception results, better suppressing missed detections, false detections, semantic confusion and motion prediction errors, and improving the accuracy and asynchronous robustness of collaborative perception prediction.

[0061] Specifically, it can be combined with Figure 2 and Figure 3 As shown, the working principle of the collaborative joint perception prediction method based on continuous spatiotemporal inference in this application is explained in detail with a specific embodiment.

[0062] like Figure 2 As shown, embodiments of this application may include local observation representation extraction, global continuous state deduction and updating, and perception and prediction.

[0063] In this application embodiment, local scene observation representations collected by multiple intelligent agents can be obtained. and their corresponding acquisition time The local scene observation representation is subjected to coordinate transformation to be unified to the same reference coordinate system, and an asynchronous observation event sequence is constructed according to the acquisition time order. .

[0064] The embodiments of this application can combine continuous spatiotemporal extrapolation with time-gated updates to uniformly model the current scene state and the future scene state using asynchronous time information.

[0065] Specifically, a global continuous scene state is constructed based on an asynchronous observation event sequence, and the global continuous scene state is continuously spatiotemporally extrapolated between adjacent observation events. When the event time is reached, a candidate correction state is generated based on the corresponding local scene observation representation and the latest extrapolated global continuous scene state. A time-aware update weight is constructed by combining a time mask, and the global continuous scene extrapolation state is updated with time-gated weighting.

[0066] The embodiments of this application can repeatedly perform inference and update to obtain the scene state representation at the current decision moment and future moments, and obtain the perception prediction result after decoding.

[0067] To verify the effectiveness of the proposed method, this embodiment uses the V2XPnP-Seq real-world vehicle-road cooperative dataset for training and validation. The experiment sets the intelligent connected vehicle as the primary agent, selects two roadside facilities as cooperative agents, and sets the asynchronous information time between agents to 400 ms, representing a relatively severe asynchronous situation in real-world cooperative scenarios. Figure 3 As shown, the comparison of perception prediction results between the proposed method and existing typical methods under two complex urban conditions is presented. The black blocks represent the current position of the perceived instance, and the gray lines in front of the instance represent the predicted future motion trajectory. It can be seen that when faced with a severely asynchronous input of 400 ms, the proposed method can better suppress missed detections (a3), false detections (a1, b3), semantic confusion (a2, b2), and motion prediction errors (a2, b1, b2) compared with the comparison algorithm, thus improving the accuracy and asynchronous robustness of collaborative perception prediction.

[0068] The collaborative joint perception and prediction method based on continuous spatiotemporal extrapolation proposed in this application can acquire local scene observation representations collected by multiple agents and construct an asynchronous observation event sequence according to the order of acquisition time to build a global continuous scene state under a unified reference coordinate system. Continuous spatiotemporal extrapolation of the global continuous scene state is performed between adjacent observation events. When the next event time is reached, the global continuous scene extrapolation state is updated with time-gated weighting by combining candidate correction states and time-aware update weights to obtain scene state representations at the current decision time and future times. Decoding yields collaborative perception and motion prediction results, which can be used to process collaborative joint perception and prediction under asynchronous input conditions of multiple agents. This improves the accuracy and asynchronous robustness of the current collaborative perception results and future prediction results, effectively adapting to collaborative scenarios with communication delays, asynchronous sampling, and complex dynamic target interactions. Therefore, it solves the problem in related technologies where information mismatch easily occurs when processing real asynchronous perception information, leading to semantic understanding errors and motion inference errors, which seriously affect the collaborative perception and prediction performance.

[0069] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0070] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0071] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0072] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0073] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0074] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0075] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0076] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A collaborative joint sensing and prediction method based on continuous spatiotemporal extrapolation, characterized in that, Includes the following steps: Acquire local scene observation representations collected by multiple intelligent agents and their corresponding acquisition times; An asynchronous observation event sequence is constructed according to the order of the acquisition time, and a global continuous scene state is constructed based on the asynchronous observation event sequence and the local scene observation representation. The global continuous scene state is continuously spatiotemporally extrapolated between adjacent observation events to obtain the global continuous scene extrapolation state. When the next event moment is reached, a candidate correction state is generated based on the corresponding local scene observation representation and the global continuous scene inference state. A time-aware update weight is constructed by combining the time mask, and the global continuous scene inference state is updated with time-gated weighting to obtain the global continuous scene state after the event is updated. The global continuous scene state at the current decision moment is obtained after all events of the asynchronous observation event sequence are processed. Based on the current continuous scene state at the current decision moment, a scene state representation for future moments is generated to obtain collaborative perception and motion prediction results.

2. The method according to claim 1, characterized in that, The step of continuously extrapolating the state of the entire continuous scene between adjacent observation events to obtain the extrapolated state of the entire continuous scene includes: Within each integration time step, the inference driving force input is generated based on the global continuous scene state at the start of the integration step; Based on the driving force input, the unit time change of the global continuous scene state is calculated, and the unit time change is used to perform residual state evolution to determine the global continuous scene state at the end of the integral step, until the next event time is reached, and the global continuous scene deduction state is determined according to the global continuous scene state at the next event time.

3. The method according to claim 2, characterized in that, The expression for the state evolution at each integration time step is: ， in, This refers to the global continuous scene state at the start of the integration step. This represents the change in the state of the entire continuous scene per unit time. The driving force input for the deduction is... For state deduction function, For the integration time step, This refers to the global continuous scene state at the end of the integration step.

4. The method according to claim 1, characterized in that, The step of generating candidate correction states based on the corresponding local scene observation representation and the global continuous scene deduction state includes: The corresponding local scene observation representation is determined based on the next event time. The local scene observation representation and the global continuous scene deduction state are input into the target convolutional network to output the candidate correction state.

5. The method according to claim 1, characterized in that, The formula for performing time-gated weighted updates on the global continuous scene deduction state is as follows: ， ， in, The time-aware update weights, A time-aware function for calculating update weights; For the sigmoid function, The time mask, This refers to the candidate correction state. This refers to the state of the continuous scenario across the entire domain. To utilize The state of the entire continuous scene is updated at any given moment.

Images