Time sequence fusion method for road element detection results, intelligent equipment and medium

Abstract

The invention relates to the field of automatic driving, in particular to a time sequence fusion method for road element detection results, intelligent equipment for implementing the method and a computer readable storage medium for implementing the method. The method comprises the steps that continuous multi-frame road element detection results are obtained, the continuous multi-frame road element detection results comprise a current frame and a historical frame, and each frame comprises one or more road elements and attribute information of the road elements; establishing a corresponding relation of the same road element among different frames by utilizing an attention mechanism; and performing time sequence fusion on the current frame and the historical frame according to the corresponding relationship to obtain a road element detection result of the current frame after time sequence fusion.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and more specifically to a temporal fusion method for road element detection results, an intelligent device for implementing the method, and a computer-readable storage medium for implementing the method. Background Art

[0002] In the field of autonomous driving, the reliability of autonomous driving functions depends on stable and accurate environmental perception information, among which the perception quality of road elements plays a decisive role in downstream prediction and planning tasks.

[0003] At present, traditional road element perception solutions usually take a single frame of observation data captured by on-board sensors as input and output road element information directly associated with the frame of observation data. However, this method is easily affected by environmental factors, such as dynamic occlusion, poor lighting conditions, and extreme weather, which may interfere with the detection process of road elements, thereby reducing the integrity and accuracy of the detection results. These environmental factors may also cause jumps and jitters in the detection results of road elements, which not only increases the incoherence of the data, but also increases the complexity of downstream application processing.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] In order to solve or at least alleviate one or more of the above problems, an embodiment of the present application provides a temporal fusion method for road element detection results, an intelligent device for implementing the method, and a computer-readable storage medium for implementing the method, which realizes the temporal fusion of road element detection results by combining multi-frame information and the attention mechanism in the deep learning framework.

[0006] According to a first aspect of the present application, a temporal fusion method for road element detection results is provided, the method comprising the following steps: obtaining road element detection results of multiple consecutive frames, wherein the road element detection results of the multiple consecutive frames include a current frame and a historical frame, and each frame includes one or more road elements and their attribute information; using an attention mechanism to establish a correspondence between different frames for the same road element; and temporally fusing the current frame and the historical frame according to the correspondence to obtain the road element detection result of the current frame after temporal fusion.

[0007] As an alternative or supplement to the above solution, the time series fusion method according to an embodiment of the present application further includes: encoding the road element detection results of the continuous multiple frames to generate a multi-dimensional feature vector containing time series information.

[0008] As an alternative or supplement to the above scheme, in a temporal fusion method according to an embodiment of the present application, using an attention mechanism to establish a correspondence between the same road element in different frames includes: for each road element in the current frame, using the attention mechanism to calculate the similarity score between the feature vector of the road element and the feature vector of the candidate road element in the historical frame; determining the matching relationship between the road element and the candidate road element based on the calculated similarity score; and converting the matching relationship into structured data.

[0009] As an alternative or supplement to the above scheme, in a temporal fusion method according to an embodiment of the present application, using an attention mechanism to establish a correspondence between the same road element in different frames also includes: taking each road element in the historical frame as the candidate road element; or selecting a set of candidate road elements that have a potential correspondence with the road element from the historical frame based on the attribute information of the road element, wherein the attribute information includes the semantic type, position, color, shape, and confidence of the road element.

[0010] As an alternative or supplement to the above scheme, in a temporal fusion method according to an embodiment of the present application, the structured data includes one or more of the following items: an identifier of the road element in the current frame; a position or index of a candidate road element matching the road element in the historical frame; and a similarity score between feature vectors of the matching road element and the candidate road element.

[0011] As an alternative or supplement to the above scheme, in a temporal fusion method according to an embodiment of the present application, temporal fusion of the current frame and the historical frame according to the corresponding relationship includes: extracting feature vectors of road elements in the current frame and feature vectors of road elements in historical frames that have a corresponding relationship therewith; and using a cross-attention mechanism to temporally fuse the extracted feature vectors to achieve feature enhancement of the road elements in the current frame.

[0012] As an alternative or supplement to the above scheme, in a temporal fusion method according to an embodiment of the present application, temporal fusion of the current frame and the historical frame according to the corresponding relationship also includes: inputting the feature vector after temporal fusion into a decoder; and the decoder converts the feature vector after temporal fusion into attribute information of road elements in the current frame.

[0013] As an alternative or supplement to the above scheme, the temporal fusion method according to an embodiment of the present application also includes: determining the scene association relationship between different road elements based on the road element detection results of the current frame after temporal fusion, wherein the scene association relationship includes the relative position and / or functional connection between different road elements.

[0014] As an alternative or supplement to the above scheme, the temporal fusion method according to an embodiment of the present application also includes: extracting a feature subset related to location information and functional connections from the feature vectors of the road elements in the current frame after temporal fusion; and inputting the feature subset into a fully connected network or a graph network to predict the scene association relationship between different road elements.

[0015] As an alternative or supplement to the above scheme, the timing fusion method according to an embodiment of the present application also includes: outputting one or more of the road element detection results of the current frame after timing fusion, the correspondence between the same road element in different frames, and the scene association relationship between different road elements, as data input for the autonomous driving control logic and the cockpit display logic.

[0016] According to the second aspect of the present application, an intelligent device is provided, comprising: a memory; a processor; and a computer program stored in the memory and executable on the processor, wherein the execution of the computer program enables any one of the timing fusion methods described in the first aspect of the present application to be executed.

[0017] According to a third aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes instructions, and the instructions, when run, execute any one of the timing fusion methods described in the first aspect of the present application.

[0018] The core of the temporal fusion solution according to one or more embodiments of the present application is to use the road element detection results of multiple consecutive frames, combined with the attention mechanism for temporal fusion, so as to optimize the detection results of the current frame. Specifically, firstly, the detection results of multiple consecutive frames are obtained, which provides rich contextual information for optimization and makes up for the defect that single-frame detection is susceptible to interference; secondly, the attention mechanism can effectively establish the correspondence between the same road element in different frames, overcome the influence of factors such as dynamic occlusion and illumination changes, solve the problem of timing jumps and jitters in the detection results in traditional methods, and realize continuous tracking of the target; finally, through temporal fusion, the reliable information in the historical frame is integrated with the detection results of the current frame, thereby weakening the influence of noise and interference and improving the accuracy and completeness of the detection results. Overall, the scheme uses deep learning technology, especially attention mechanism and temporal fusion, to achieve comprehensive optimization of road element detection, improve the accuracy and stability of detection, and provide more reliable environmental perception capabilities for autonomous driving systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or other aspects and advantages of the present application will become clearer and easier to understand through the following description of various aspects in conjunction with the accompanying drawings, in which the same or similar units are represented by the same reference numerals. In the accompanying drawings:

[0020] Figure 1 is a schematic flow chart of the road element temporal fusion process based on rule and nonlinear optimization algorithm;

[0021] Figure 2 is a schematic flow chart of a timing fusion method 20 according to an embodiment of the present application;

[0022] Figure 3 is a schematic flow chart of a time series fusion process based on a deep learning framework according to an embodiment of the present application; and

[0023] Figure 4 is a schematic block diagram of a smart device 40 according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The description of the following specific embodiments is merely exemplary in nature and is not intended to limit the disclosed technology or the application and use of the disclosed technology. In addition, it is not intended to be bound by any express or implied theory presented in the aforementioned technical field, background technology or the following specific embodiments.

[0025] In the following detailed description of the embodiments, many specific details are set forth in order to provide a more thorough understanding of the disclosed technology. However, it is apparent to one of ordinary skill in the art that the disclosed technology can be practiced without these specific details. In other instances, well-known features are not described in detail to avoid unnecessarily complicating the description.

[0026] Terms such as "comprising" and "including" indicate that in addition to the units and steps directly and clearly stated in the specification, the technical solution of the present application does not exclude the situation of having other units and steps that are not directly or clearly stated. Terms such as "first" and "second" do not indicate the order of units in terms of time, space, size, etc., but are only used to distinguish each unit. The technology of the present application is generally used in electric vehicles, which include but are not limited to pure electric vehicles (BEV), hybrid electric vehicles (HEV), fuel cell vehicles (FCEV), etc.

[0027] An automated driving system (ADS) refers to a system that will continuously perform all dynamic driving tasks (DDT) within its own operational domain design (ODD). That is, the machine system is allowed to fully assume the task of automatic vehicle control only under the specified appropriate driving scenario conditions, that is, the system is activated when the vehicle meets the ODD conditions, and the system replaces the human driver as the driving subject of the vehicle. Among them, the dynamic driving task DDT refers to the continuous lateral (left and right steering) and longitudinal motion control (acceleration, deceleration, constant speed) of the vehicle, as well as the detection and response to targets and events in the vehicle's driving environment. The design operating range ODD refers to the conditions under which the automated driving system can operate safely, and the conditions set may include geographic location, road type, speed range, weather, time, national and local traffic laws and regulations, etc.

[0028] As mentioned above, traditional road element perception solutions usually use single-frame observation data captured by vehicle-mounted sensors as input. To address this drawback, an embodiment of the present application proposes to input multiple frames of data, decouple detection and time series tracking, and implement time series fusion based on filtering algorithms and / or nonlinear optimization algorithms. Figure 1 This type of timing fusion solution is introduced by way of example.

[0029] Figure 1 A schematic flow chart of the road element temporal fusion process based on rule and nonlinear optimization algorithm.

[0030] like Figure 1As shown, the road element detection result of the current frame is input; based on the road element detection result of the current frame and the control point chain information of the previous frame fed back, predefined rules are used to associate the same road elements detected in the current frame and the previous frame, and these rules may be based on the relative position, size, shape and other features of the elements; the control point chain is added and deleted to maintain the continuity of the elements in time sequence; next, the attributes of the road elements are filtered and voted to determine the final attributes, which may involve statistical analysis of the element status; the time series data is fused using a nonlinear optimization algorithm, and the time series information is used to track the control point chain, so that the control point chain information of the current frame can be used for fusion with the next frame.

[0031] It should be pointed out that this solution is based on filtering algorithms and nonlinear optimization algorithms for time series association and fusion, and relies on predefined rules to associate elements. This approach limits the flexibility and robustness of the association, and has poor generalization ability, especially in complex scenarios or abnormal situations, which may cause the rules to fail, thereby affecting tracking accuracy and continuity. In addition, fixed rules themselves are difficult to adapt to the changing road environment and sensor data characteristics, have no adaptive capabilities, and are difficult to iterate and update.

[0032] In view of its drawbacks, according to one or more embodiments of the present application, a time series fusion solution for road element detection results is proposed, which performs time series fusion in a data-driven manner. Compared with predefined rules, deep learning technology has better generalization ability and can adapt to a wider range of scenarios. In addition, this solution can be end-to-end integrated with upstream detection models to achieve an overall upgrade of the perception solution. Figure 2-4 This type of temporal fusion solution for road element detection results is introduced in detail.

[0033] Figure 2 is a schematic flow chart of a timing fusion method 20 according to one or more embodiments of the present application.

[0034] like Figure 2 As shown, in step 210, road element detection results of multiple consecutive frames are obtained.

[0035] The road element detection results obtained in step 210 refer to relevant information of road elements identified and located from images, videos or point cloud data using computer vision technology or perception technology. In one embodiment, the road element detection results output by the upstream scene parsing model can be obtained. The scene parsing model refers to a model based on computer vision and perception technology. The model automatically identifies, locates and classifies different elements in the scene by analyzing the original sensor data. Exemplarily, the model can identify vehicles, traffic signs, lane lines, etc. in the scene, and determine their positions and categories in the scene. To achieve this goal, the scene parsing model can adopt a variety of technical architectures, such as target detection technology or image segmentation technology.

[0036] It should be noted that this application only focuses on the detection results of the scene parsing model for static road elements. These static road elements include but are not limited to lane lines, curbs, navigation lanes, ground lane markers, and road signs. This application does not involve the processing and optimization of the detection results of dynamic elements in the scene (for example, pedestrians, vehicles). Figure 1 What is different from the scheme in is that the input data in this application is not single-frame data, but the road element detection results of multiple consecutive frames. Among them, the multiple consecutive frames include the current frame and at least one historical frame, and each frame includes one or more detected road elements and their corresponding attribute information (for example, the semantic type, position, color, shape, confidence, etc. of the road element). In one possible embodiment, the present application adopts a sliding time window to obtain the road element detection results. The sliding time window intercepts data with a fixed-size window in the data stream of continuous road element detection results, and slides forward with the arrival of new data. For example, the sliding time window can be set to include the current frame and the previous 14 frames of data, that is, a total of 15 frames of data, as the input of the deep learning model. This method can ensure the consistency and predictability of the model input.

[0037] Optionally, after obtaining the road element detection results of multiple consecutive frames, the encoder network in the deep learning framework can be used to encode these results to generate a multi-dimensional feature vector containing temporal information. This step is intended to convert the heterogeneous road element detection results into numerical representations that are easy to process for the deep learning model. Different attributes of different road elements will be encoded into feature vectors, including but not limited to semantic types (such as lane lines, curbs), positions (such as bounding box coordinates), colors, shapes (such as bounding box aspect ratios), confidence, etc. Exemplarily, discrete attributes (such as semantic types) can be converted into binary vectors using one-hot encoding; continuous attributes (such as position coordinates) can be directly represented using numerical values; shape information can be numerically represented by geometric feature extraction, such as the aspect ratio of the bounding box, center point coordinates, direction angles, etc. The output after encoding is a high-dimensional feature vector, the dimension of which depends on the selected encoding scheme and feature engineering technology. The feature vector represents the necessary information of all road elements in all frames in a numerical form that can be processed by the deep learning model, for example, it can be expressed in the form of (frame number, road element identifier, attribute 1, attribute 2, ...).

[0038] In step 220, the attention mechanism is used to establish the correspondence between the same road element in different frames.

[0039] The attention mechanism is an important component of the deep learning model, which allows the deep learning model to dynamically focus on key information in the input data and adjust subsequent calculations based on this information. Step 220 aims to use the attention mechanism to track the same road element in consecutive frames, which is significantly different from the association algorithm based on fixed rules.

[0040] Specifically, as mentioned above, some solutions rely on predefined rules (such as position, shape similarity, etc.) to associate road elements in different frames. These rules are often difficult to deal with complex scenes, such as occlusion, partial disappearance or mutation, and are prone to association errors. The attention mechanism allows the model to adaptively learn these change patterns, and predict more accurate correspondences by learning the intrinsic connections between road element features between different frames, such as motion trajectories and shape evolution. Therefore, unlike the rule-based association algorithm, the attention mechanism-driven model does not need to predefine fixed association rules, but adaptively learns and predicts the correspondence between road elements in different frames. The model can learn the similarity of road element features in different frames based on the input continuous frame data, and dynamically adjust its attention weights to focus on more critical information. Through this adaptive learning mechanism, when processing the road element detection results of continuous frames, the attention mechanism-driven model can capture the dynamic change pattern of road element features between different frames without relying on prior knowledge of the scene, providing reliable correspondences for subsequent temporal fusion, and ultimately improving the accuracy and robustness of road element detection results.

[0041] Exemplarily, step 220 may be performed according to the following three sub-steps: calculating a similarity score, determining a matching relationship, and converting into structured data. These three sub-steps will be introduced one by one below.

[0042] First, for each road element in the current frame, the attention mechanism is used to calculate the similarity score between the feature vector of the road element and the feature vectors of all candidate road elements in the historical frame. Here, there are two strategies for selecting candidate road elements: one is to use all road elements in the historical frame as candidate road elements, which is a full-quantity matching strategy with the advantage of high recall rate; the other is to pre-screen a set of candidate elements with potential corresponding relationships based on the attribute information of the current road element (e.g., semantic type, position, color, shape, confidence, etc.), thereby improving computational efficiency and reducing mismatches. It should be noted that the calculation of the similarity score is not a simple calculation of the feature vector distance, but is based on the attention mechanism (e.g., the self-attention mechanism in Transformer), which can consider contextual information and calculate the similarity between feature vectors to form a similarity matrix, which can better capture the complex relationship between features than simple distance metrics, especially when processing sequence data.

[0043] Then, the matching relationship between the road element and the candidate road element is determined based on the calculated similarity score. Exemplarily, the matching relationship can be extracted from the similarity matrix based on the following strategies: based on Top-k sorting selection, that is, for each road element in the current frame, select the k (k is a positive integer) candidate road elements with the highest similarity score; based on softmax probability distribution, that is, convert the similarity score into a probability distribution through the softmax function, where each candidate road element will obtain a probability value, representing the possibility of it becoming a matching object, and finally select one or more elements with the highest probability as matching objects; based on the global optimization of the graph matching algorithm, if the relationship between multiple elements needs to be considered, the graph matching algorithm (such as the Hungarian algorithm) can be used to find the optimal match in the global scope, and the cost function of the algorithm can be based on the similarity score obtained previously, combined with other constraints (such as spatial consistency).

[0044] Finally, the determined matching relationship is converted into structured data. This structured data contains the association information between the road elements in the current frame and the corresponding elements in the historical frame, providing a basis for subsequent temporal fusion. Exemplarily, the structured data may include the following information: the identifier of the road element in the current frame, which is used to uniquely identify each road element in the current frame; the position or index of the matching candidate element in the historical frame, which is used to indicate the position (e.g., coordinates) or index of the candidate element matching the current road element in the historical frame; the similarity score, which is used to represent the similarity between the current road element and the candidate road element matching it, which can be used as weight information in the subsequent fusion process. These structured data can be represented in a variety of data formats, such as tables, JSON, or other formats suitable for deep learning model processing.

[0045] In step 230, the current frame and the historical frames are temporally fused according to the corresponding relationship to obtain a road element detection result of the current frame after temporal fusion.

[0046] Step 230 is intended to perform temporal fusion on the road element detection results of the current frame and the historical frames according to the correspondence established in step 220. Temporal fusion refers to utilizing the temporal correlation between consecutive frames and integrating multi-frame information to improve the accuracy and robustness of single-frame detection results. Compared with the solution that only relies on single-frame information, this can better handle complex situations such as occlusion and illumination changes. In the framework of deep learning, a special temporal fusion layer can be designed, which receives the feature vectors of the current frame and the historical frame from the encoder, as well as the correspondence established in step 220, and integrates these features to form a temporal fusion output through a convolutional neural network, an attention mechanism (for example, a cross-attention mechanism), or other fusion strategies.

[0047] In the implementation method of using the cross-attention mechanism for temporal fusion, the feature vector of each road element in the current frame and the feature vector of the road element that has a corresponding relationship with it in the historical frame are first extracted. These feature vectors can be directly output by the encoder, or they can be the result of other processing (for example, feature screening or dimensionality reduction). Afterwards, the extracted feature vectors can be temporally fused using the cross-attention mechanism to achieve feature enhancement of the road elements in the current frame. Through the cross-attention mechanism, the model can adaptively learn the contribution of the corresponding elements of the historical frame to the feature enhancement of the road elements in the current frame, effectively utilize the temporal context information, and thus enhance the robustness and accuracy of the feature representation of the road elements in the current frame.

[0048] After the time series fusion process is completed, the enhanced feature vector of the road element of the current frame is obtained. Further, the enhanced feature vector can be sent to the decoder for processing. In the deep learning framework, the decoder is used to convert the high-dimensional feature vector into interpretable road element attributes. The structure of the decoder can be selected according to the specific application scenario and requirements, such as a fully connected layer, a convolutional layer, or a recurrent neural network. The decoder outputs optimized road element attributes, such as more accurate location coordinates, more reliable semantic categories, and more accurate confidence. These optimized attribute information are integrated with the original detection results of the road elements of the current frame to form the road element detection results of the current frame after time series fusion.

[0049] The road element detection results after temporal fusion outputted in step 230 can be used for subsequent downstream applications. However, for downstream applications with tasks such as autonomous driving and map building, it is not enough to simply obtain the road element detection results. To achieve a higher level of scene understanding and application, it is also necessary to understand the relationship between these road elements. In this regard, method 20 may further include step 240: based on the road element detection results of the current frame after temporal fusion, determining the scene association relationship between different road elements.

[0050] Step 240 aims to analyze and identify scene association relationships between road elements under the deep learning framework. The scene association relationship in this article refers to the spatial relationship and functional connection between road elements. Spatial relationship refers to the geometric position and relative layout between road elements, such as the distance between lane lines, the relative position of road signs and lanes, etc., which helps the autonomous driving vehicle to accurately locate its position on the road and understand the geometric structure of the surrounding environment. Functional connection refers to the logical and rule connection between road elements, such as lanes indicated by traffic signs, driving directions indicated by road signs, etc. The determination of scene association relationships requires the deep learning model to have the ability to understand the scene context, so as to go beyond simple object detection and perform a higher level of scene understanding, which is crucial for downstream functions (for example, vehicle navigation, path planning, and decision making).

[0051] First, feature subsets related to location information and functional connections can be extracted from the feature vectors of the road elements in the current frame after time series fusion. These features may include the shape, location, semantic type, etc. of the elements. Then, the extracted feature subsets are input into a fully connected network or a graph neural network to predict the scene association relationship between different road elements. The fully connected network can learn the nonlinear mapping relationship between feature subsets and scene association relationships, and its output can be a vector representing the probability distribution of different types of scene association relationships. This scheme is suitable for scenes where the relationship between road elements is relatively simple. If the relationship between road elements is more complex, such as multiple relationships between multiple road elements, it is more suitable to use a graph neural network for prediction. Specifically, each road element can be regarded as a node in the graph, and the feature subset can be regarded as the feature vector of the node. The graph neural network can learn the topological structure and feature information between nodes and predict the relationship between nodes, which is more effective for modeling complex spatial relationships and functional connections between road elements. For example, the output of step 240 may include relative positional relationships between different road elements (e.g., "A is on the left side of B", "A and B are 1 meter apart"), functional connections (e.g., lane lines L1 and L3 form the boundary of the first lane, and the ground arrow indicates the left turn lane), etc. Such information can provide a higher level of scene understanding and a more complete road network structure for subsequent applications.

[0052] Exemplarily, one or more of the output of step 220 (correspondence between different frames of the same road element), step 230 (road element detection results of the current frame after time series fusion), and the output of step 240 (scene association relationship between different road elements) can be used as the output of the deep learning model and provided to the autonomous driving control logic and the cockpit display logic as data input. Exemplarily, the autonomous driving system can perform the following operations based on these data inputs: adjust the vehicle driving trajectory based on the identified spatial relationship between lane lines and lanes; remind the driver to pay attention to traffic rules based on the identified relationship between stop lines and traffic lights; optimize vehicle positioning based on the understanding of the spatial relationship of road elements. The cockpit display logic can use the road element detection results of the current frame after time series fusion and the scene association relationship to provide the driver with real-time road information and warnings, thereby enhancing the driver's perception of the vehicle's surroundings.

[0053] Figure 3 A schematic flowchart of a time series fusion process based on a deep learning framework according to one or more embodiments of the present application. Figure 3 Used with Figure 1By contrast, the difference between the timing fusion solution based on the deep learning framework in this application and the timing fusion process of road elements based on rules and nonlinear optimization algorithms is clarified.

[0054] like Figure 3 As shown, the road element detection results of multiple consecutive frames are input into the deep learning model; under the deep learning framework, the encoder is used to encode the input data to generate a multi-dimensional feature vector containing temporal information; then, the attention mechanism is used to establish the correspondence between the same road element in different frames, and the correspondence is used as part of the output of the deep learning model; then, the current frame and the historical frame are temporally fused according to the correspondence to obtain the road element detection result of the current frame after temporal fusion, and the road element detection result after temporal fusion is used as part of the output of the deep learning model; finally, based on the road element detection result of the current frame after temporal fusion, the scene association relationship between different road elements is determined, and the scene association relationship is used as part of the output of the deep learning model.

[0055] The above time series fusion solution ( Figure 2 and Figure 3 ) uses deep learning technology, especially attention mechanism and time series fusion, to optimize the detection of road elements, improve the accuracy and stability of detection, and provide more reliable environmental perception capabilities for autonomous driving systems. In addition, some implementations of this solution can analyze and identify the scene association relationship between road elements, providing a higher level of scene understanding and a more complete road network structure to the downstream.

[0056] Figure 4 4 is a schematic block diagram of an intelligent device 40 according to one or more embodiments of the present application. The intelligent device 40 includes a memory 410, a processor 420, and a computer program 430 stored in the memory 410 and executable on the processor 420. The execution of the computer program 430 enables the above-mentioned time series fusion method 20 to be executed. Exemplarily, the intelligent device 40 may be a driving device, a smart car, an electric car, a robot, or other device having an automatic driving function or an automatic driving system.

[0057] In addition, as described above, the present application can also be implemented as a computer storage medium, in which a program for causing a computer to execute the method described in any of the above embodiments is stored. Here, as a computer storage medium, various computer storage media such as disks (e.g., magnetic disks, optical disks, etc.), cards (e.g., memory cards, optical cards, etc.), semiconductor memories (e.g., ROMs, non-volatile memories, etc.), and tapes (e.g., magnetic tapes, cassette tapes, etc.) can be used.

[0058] In the applicable situation, hardware, software or a combination of hardware and software can be used to realize the various embodiments provided by the application. Moreover, in the applicable situation, without departing from the scope of the application, the various hardware components and / or software components set forth herein can be combined into a composite component comprising software, hardware and / or both. In the applicable situation, without departing from the scope of the application, the various hardware components and / or software components set forth herein can be divided into subcomponents comprising software, hardware or both. In addition, in the applicable situation, it is contemplated that the software component can be implemented as a hardware component, and vice versa.

[0059] Software (such as program code and / or data) according to the present application can be stored on one or more computer storage media. It is also contemplated that the software identified herein can be implemented using one or more general or special-purpose computers and / or computer systems, networked and / or otherwise. Where applicable, the order of the various steps described herein can be changed, combined into composite steps, and / or divided into sub-steps to provide the features described herein.

[0060] The embodiments and examples set forth herein are provided to best illustrate embodiments according to the present application and its specific applications, and thereby enable those skilled in the art to implement and use the present application. However, those skilled in the art will appreciate that the above description and examples are provided only for ease of illustration and example. The description set forth is not intended to cover all aspects of the present application or to limit the present application to the precise form disclosed.

Claims

1. A temporal fusion method for road element detection results, characterized in that: The method comprises the following steps: Acquire road element detection results of multiple consecutive frames, wherein the road element detection results of the multiple consecutive frames include a current frame and a historical frame, and each frame includes one or more road elements and attribute information thereof; Using the attention mechanism, the correspondence between the same road elements in different frames is established; and The current frame and the historical frame are temporally fused according to the corresponding relationship to obtain a road element detection result of the current frame after temporal fusion.

2. The method according to claim 1, characterized in that The method further comprises: The road element detection results of the continuous multiple frames are encoded to generate a multi-dimensional feature vector containing time sequence information.

3. The method according to claim 1, characterized in that Using the attention mechanism, the correspondence between the same road elements in different frames is established, including: For each road element in the current frame, using the attention mechanism to calculate a similarity score between a feature vector of the road element and a feature vector of a candidate road element in the historical frame; Determining a matching relationship between the road element and the candidate road element according to the calculated similarity score; and The matching relationship is converted into structured data.

4. The method according to claim 3, characterized in that Using the attention mechanism, the corresponding relationship between the same road elements in different frames is also established: Taking each road element in the historical frame as the candidate road element; or According to the attribute information of the road element, a set of candidate road elements having a potential corresponding relationship therewith is selected from the historical frame, wherein the attribute information includes the semantic type, position, color, shape, and confidence of the road element.

5. The method according to claim 3 or 4, characterized in that: The structured data includes one or more of the following: An identifier of a road element in the current frame; The position or index of the candidate road element matching the road element in the historical frame; The similarity score between the feature vectors of the matched road element and the candidate road element.

6. The method according to claim 1, characterized in that Performing temporal fusion of the current frame and the historical frame according to the corresponding relationship includes: Extracting feature vectors of road elements in the current frame and feature vectors of road elements in historical frames that have a corresponding relationship therewith; and The extracted feature vectors are temporally fused using a cross-attention mechanism to achieve feature enhancement for the road elements in the current frame.

7. The method according to claim 6, characterized in that Performing temporal fusion of the current frame and the historical frame according to the corresponding relationship further includes: Input the time-series fused feature vector into the decoder; The decoder converts the time-series fused feature vector into attribute information of the road element in the current frame.

8. The method according to claim 1, characterized in that The method further comprises: Based on the road element detection result of the current frame after time series fusion, the scene association relationship between different road elements is determined, wherein the scene association relationship includes the relative position and / or functional connection between different road elements.

9. The method according to claim 1, characterized in that: The method further comprises: Extracting a feature subset related to location information and functional connection from the feature vectors of the road elements in the current frame after time series fusion; and The feature subset is input into a fully connected network or a graph network to predict scene association relationships between different road elements.

10. The method according to claim 8 or 9, characterized in that: The method further comprises: Output one or more of the road element detection results of the current frame after time series fusion, the correspondence between the same road element in different frames, and the scene association relationship between different road elements as data input for the autonomous driving control logic and cockpit display logic.

11. A smart device, characterized in that: The method comprises: a memory; a processor; and a computer program stored in the memory and executable on the processor, wherein the execution of the computer program causes the method according to any one of claims 1 to 10 to be executed.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions which, when executed, perform the method according to any one of claims 1-10.

Images