Multi-sensor fusion automatic driving perception method and device and electronic equipment

Abstract

The embodiment of the invention provides a multi-sensor fusion automatic driving sensing method and device and electronic equipment, and the sensing precision and robustness of an automatic driving system are remarkably improved by efficiently integrating and utilizing rich sensing data collected by multiple sensors. Specifically, firstly, multi-source sensor data are orderly arranged to form unified multi-sensor fusion data, then feature extraction is performed to construct a comprehensive scene perception feature set, and a plurality of fusion scene perception feature sets capable of reflecting spatio-temporal mode features are generated through a multi-dimensional privilege domain fusion strategy. The method enhances the understanding capability of the driving environment, finally achieves the precise prediction of the driving state through a machine learning network model, timely outputs the driving safety assessment, carries out the early warning and reminding based on the above, effectively reduces the safety risk in the automatic driving process, improves the intelligent level of driving decision, and improves the driving safety. And safer and more reliable automatic driving experience is provided for the user.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a multi-sensor fusion autonomous driving perception method, device and electronic equipment. Background Art

[0002] With the rapid development of autonomous driving technology, how to ensure the safe driving of autonomous vehicles in various complex road environments has become a key issue that needs to be solved urgently. Autonomous vehicles are usually equipped with a variety of sensors, such as radar, camera, laser radar (LiDAR), etc., to collect perception data around the vehicle in real time. However, how to effectively integrate these data from different sensors and accurately identify and understand the driving environment is a major challenge facing autonomous driving technology.

[0003] Traditional autonomous driving perception methods often focus on processing data from a single sensor, or simply superimpose multi-sensor data, but fail to fully explore and utilize the complementarity and correlation between sensors. This method is prone to perception blind spots or misjudgments when dealing with complex scenarios, thus affecting the safety and reliability of autonomous driving.

[0004] In addition, most existing multi-sensor fusion methods focus on low-level fusion of data, such as fusion at the data level or simple splicing at the feature level, but fail to deeply explore the characteristics of each sensor data in spatiotemporal patterns. This shallow fusion method is difficult to capture subtle changes and potential risks in the driving environment, limiting the perception ability of autonomous vehicles in complex environments. Summary of the invention

[0005] In order to at least overcome the above-mentioned deficiencies in the prior art, the purpose of the embodiments of the present application is to provide a multi-sensor fusion autonomous driving perception method, device and electronic device.

[0006] According to one aspect of the present application, a multi-sensor fusion autonomous driving perception method is provided, the method comprising: Acquire first multi-sensor fusion data, where the first multi-sensor fusion data is generated by arranging sensor perception data collected by a plurality of sensors of the first autonomous driving vehicle in a set driving segment based on an autonomous driving perception timing sequence; Extracting features from the first multi-sensor fusion data according to the perception knowledge space of each sensor perception data in the first multi-sensor fusion data to generate a first scene perception feature set; Perform privileged domain fusion of multiple fusion dimensions on the first scene perception feature set to generate multiple fusion scene perception feature sets of the first autonomous driving vehicle; wherein one fusion dimension corresponds to one fusion scene perception feature set, and each fusion scene perception feature set represents the spatiotemporal pattern characteristics of the first multi-sensor fusion data in the corresponding fusion dimension; Predicting a driving state of a plurality of fused scene perception feature sets of the first autonomous driving vehicle according to a machine learning network model to generate a driving state prediction result, wherein the driving state prediction result represents the driving safety of the first autonomous driving vehicle in terms of perceived driving safety; Based on the driving state prediction result, a warning reminder is given to the first automatic driving vehicle.

[0007] According to one aspect of an embodiment of the present application, a multi-sensor fusion autonomous driving perception device is provided, in which machine executable instructions are stored, and the machine executable instructions are loaded by a processor to execute the method provided above.

[0008] According to one aspect of an embodiment of the present application, an electronic device is provided, which includes a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and the machine-executable instructions are loaded by the processor and execute the method provided above.

[0009] In the technical solutions provided in some embodiments of the present application, the embodiments of the present application significantly improve the perception accuracy and robustness of the autonomous driving system by efficiently integrating and utilizing the rich perception data collected by multiple sensors. Specifically, the multi-source sensor data is first arranged in an orderly manner to form unified multi-sensor fusion data, and then feature extraction is performed to construct a comprehensive scene perception feature set. Through a multi-dimensional privileged domain fusion strategy, multiple fusion scene perception feature sets that can reflect spatiotemporal pattern characteristics are generated, which enhances the ability to understand the driving environment. Finally, a machine learning network model is used to achieve accurate prediction of the driving state, and a driving safety assessment is output in a timely manner. Based on this, early warning reminders are given, which effectively reduces the safety risks in the autonomous driving process, improves the intelligence level of driving decisions, and provides users with a safer and more reliable autonomous driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be extracted in combination with these drawings without creative work.

[0011] Figure 1 A schematic diagram of a process flow of a multi-sensor fusion autonomous driving perception method provided in an embodiment of the present application; Figure 2 A schematic block diagram of the architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0012] The following description is intended to enable one of ordinary skill in the art to implement and combine the present application, and the description is provided in the context of a specific application scenario and its requirements. It will be apparent to one of ordinary skill in the art that various changes may be made to the disclosed embodiments, and that the general principles defined in the present application may be applied to other embodiments and application scenarios without departing from the principles and scope of the present application. Therefore, the present application is not limited to the described embodiments, but should be given the broadest scope consistent with the claims.

[0013] Figure 1 This is a flow chart of an autonomous driving perception method based on multi-sensor fusion provided by an embodiment of the present application. The autonomous driving perception method based on multi-sensor fusion is introduced in detail below.

[0014] Step S110, obtaining first multi-sensor fusion data, where the first multi-sensor fusion data is generated by arranging sensor perception data collected by multiple sensors of the first autonomous driving vehicle within a set driving segment based on the autonomous driving perception timing.

[0015] For a first autonomous driving vehicle to conduct a test drive on a snowy scene, the set driving section is from point A to point B on the snow, a total distance of approximately 20 kilometers, passing through multiple intersections, different types of roads (such as main roads, branch roads) and different traffic scenes.

[0016] The first autonomous driving vehicle is equipped with a variety of sensors, including cameras, laser radars, millimeter wave radars, etc. In the embodiment of the present application, the autonomous driving vehicle may be an electric snowmobile, for example, the snowmobile may adopt a wheeled structure, and may adopt an emb brake as a braking system, the camera on it is responsible for capturing visual images around the vehicle, the laser radar is used to accurately measure the distance and shape of surrounding objects, and the millimeter wave radar mainly detects information such as the speed and distance of moving objects in front of, behind, and to the side of the vehicle.

[0017] During driving, each sensor continuously collects data according to the autonomous driving perception sequence. For example, the camera collects image data at a rate of 30 frames per second, the lidar emits a laser beam every 0.1 seconds and receives reflected data to build a point cloud map of the surrounding environment, and the millimeter wave radar also updates the speed and distance information of surrounding vehicles at a certain time interval.

[0018] These perception data from different sensors are continuously transmitted to the server. After receiving these data, the server arranges them according to the autonomous driving perception sequence. For example, based on the timestamp, the camera image data, lidar point cloud data, and millimeter wave radar target detection data collected at the same time or at a similar time are combined to form a fusion data unit. In this way, after driving the entire set driving section, the server obtains the first multi-sensor fusion data generated by arranging the sensor perception data collected by multiple sensors of the first autonomous driving vehicle in this section based on the autonomous driving perception sequence. This first multi-sensor fusion data contains rich environmental information at different times during the driving process of the vehicle, providing a basis for subsequent analysis and processing.

[0019] Step S120: extracting features from the first multi-sensor fusion data according to the perception knowledge space of each sensor perception data in the first multi-sensor fusion data to generate a first scene perception feature set.

[0020] In this embodiment, after the server obtains the first multi-sensor fusion data, it starts to extract features. Taking the driving scene mentioned above as an example, for each data unit in the multi-sensor fusion data (including data from different sensors at the same time or at similar times), the server first determines the set arrangement nodes between Y perception domains, assuming that Y=5, and the five perception domains are spatial positioning, object recognition, dynamic capture, environmental understanding, and road condition analysis.

[0021] For the spatial positioning perception domain, taking the data of the lidar as an example, the perception knowledge vector of the lidar contains the coordinate information of objects around the vehicle relative to the vehicle's own coordinate system. For example, there is an object 10 meters in front of the vehicle and 3 meters to the left (this is the coordinate representation in different coordinate systems), as well as the relative positioning relationship with other surrounding objects, such as the distance of 5 meters from the street lamp pole next to it and other structural elements. This constitutes the first substructure of the perception knowledge vector under the spatial positioning perception domain.

[0022] In the object recognition perception domain, the image data collected by the camera is processed by the image recognition algorithm to form a second substructure of the perception knowledge vector in terms of describing the characteristics of the object. For example, the vehicle in front is identified as a red car, and its basic geometric shape is described as a cuboid. The deep semantic features include hierarchical structures such as the brand of the vehicle may be a certain brand (inferred by the appearance characteristics of the vehicle in the image).

[0023] In terms of dynamic capture perception domain, millimeter wave radar data provides dynamic features about vehicles and other moving objects. For example, if the speed of the vehicle in front is 60 kilometers per hour, the vector representation structure of its speed is a vector with size (60 kilometers per hour) and direction (driving direction), and the time series structure of acceleration may be that the acceleration gradually increases from 0 to 2 meters per second squared in the past 5 seconds, etc., which constitutes the third substructure.

[0024] In the environmental understanding perception domain, the server combines information from multiple sensors to determine. For example, the camera image shows that the sky is dark and there are raindrops, which belongs to the classification structure of weather conditions. The lidar detects that the ambient light reflection is weak, indicating that the light intensity is low, and this light intensity is associated with the weather condition (rain causes dim light), which is the fourth substructure.

[0025] In the road condition analysis perception domain, through map data and vehicle sensor information, it is determined that the road type is a snowy main road and the road condition is good (no potholes or construction, etc.), which reflects the structural relationship between road type and road condition, that is, the fifth substructure.

[0026] Based on the set arrangement nodes between these five perception domains, the server integrates any perception knowledge vector included in each perception domain. For example, first integrate the spatial positioning information with the object recognition information according to the sequential relationship in the perception domain relationship model to determine the object identified at a specific spatial position. Then further integrate the dynamic capture information with the previous integration results to obtain dynamic information of a specific object at a specific location. Then integrate the environmental understanding information, consider the impact of environmental factors on objects and dynamic information, and finally combine the road condition analysis information. Through such integration operations, a perception knowledge space is generated for each sensor perception data.

[0027] The server generates a feature identification bit for each perception knowledge space based on the number of all perception knowledge spaces. For example, if there are 100 different perception knowledge spaces, a unique identification bit (from 00 to 99) is assigned to each space. Then the perception knowledge space encoding bits of each sensor perception data in the first multi-sensor fusion data are mapped to the feature identification bits of the corresponding perception knowledge space, thereby generating a first scene perception feature set. This first scene perception feature set contains the features related to scene perception extracted from the original multi-sensor fusion data, which is an abstraction and refinement of the original data, and is prepared for subsequent fusion processing.

[0028] Step S130 performs privileged domain fusion on the first scene perception feature set in multiple fusion dimensions to generate multiple fused scene perception feature sets for the first autonomous vehicle. Among them, one fusion dimension corresponds to one fused scene perception feature set, and each fused scene perception feature set represents the spatio-temporal pattern features of the first multi-sensor fusion data in the corresponding fusion dimension.

[0029] In this embodiment, assuming the fusion dimension X = 3, the server starts to perform privileged domain fusion on the first scene perception feature set.

[0030] First, set i = 1 and label the first scene perception feature set as the first fused scene perception feature set corresponding to the first fusion dimension. This is like making an initial classification and marking of the data, taking the first scene perception feature set as the basic data under the first fusion dimension.

[0031] When i < X (at this time i = 1, X = 3, which meets the condition), set i = i + 1, that is, i becomes 2. Perform fusion processing on the first scene perception feature set in the 2nd fusion dimension. The 2nd fusion dimension is configured such that the members fused into the same feature cluster during the fusion process in the 2nd fusion dimension include 2. The server groups the first scene perception feature set based on the order of each member in the first scene perception feature set according to this setting.

[0032] For example, if there are 100 feature members in the first scene perception feature set, every 2 members are grouped in order, thus generating the 2nd fusion set. This 2nd fusion set contains 50 feature clusters, and each feature cluster fuses 2 members with their order sorted.

[0033] Then, the server connects the members within each feature cluster in the 2nd fusion set to generate the 2nd reference scene perception feature set. For example, for two members in a feature cluster, one is the speed feature of the vehicle ahead and the other is the position feature of the vehicle ahead. Connect them to form an aggregated feature. In this way, the 2nd reference scene perception feature set includes 50 aggregated features, and each aggregated feature is generated by aggregating the members within a feature cluster in the 2nd fusion set.

[0034] Next, the server extracts features from the aggregated features in the 2nd reference scene perception feature set to generate the 2nd fused scene perception feature set corresponding to the 2nd fusion dimension. This set can represent the spatio-temporal pattern features of the first multi-sensor fusion data in the 2nd fusion dimension. It abstracts and generalizes the original data from another perspective, reflecting the internal relationship of the data under this specific fusion dimension.

[0035] Since i=2 is still less than X=3, i=i+1 is set again, that is, i becomes 3. The first scene perception feature set is fused in the third fusion dimension. The third fusion dimension is configured in the fusion process set in the third fusion dimension, and the members fused into the same feature cluster include 3.

[0036] In the same way, the server groups the first scene-aware feature set according to the order of the members to generate the third fusion set. Assuming that the first scene-aware feature set has 90 feature members left after the previous processing (because of the previous grouping operation), 30 feature clusters can be obtained by grouping 3 members.

[0037] After that, the members in each feature cluster in the third fusion set are connected to generate the third reference scene perception feature set. For example, the three members in a feature cluster are the speed and position of the vehicle in front and the light intensity of the surrounding environment, and they are connected to form a new aggregate feature. In this way, the third reference scene perception feature set contains 30 aggregate features.

[0038] Finally, feature extraction is performed on the aggregated features in the third reference scene perception feature set to generate a third fused scene perception feature set corresponding to the third fusion dimension. This set represents the spatiotemporal pattern features of the first multi-sensor fusion data in the third fusion dimension.

[0039] When i=X (i=3, X=3), the server obtains three fusion scene perception feature sets corresponding to the three fusion dimensions. These fusion scene perception feature sets mine and refine the original data from different fusion dimensions, providing more comprehensive and targeted feature information for subsequent driving state prediction.

[0040] Step S140, predicting the driving state of the first autonomous driving vehicle based on the multiple fused scene perception feature sets according to the machine learning network model, and generating a driving state prediction result, wherein the driving state prediction result represents the driving safety perceived by the first autonomous driving vehicle.

[0041] In this embodiment, the machine learning network model includes a prediction unit and three focus weight embedding units (because the previous fusion dimension X=3, one fusion scene perception feature set corresponds to one focus weight embedding unit).

[0042] The server loads the three fused scene perception feature sets of the first autonomous driving vehicle into the corresponding focus weight embedding units for spatiotemporal pattern feature learning.

[0043] Taking the first fused scene perception feature set as an example, it is loaded into the first focus weight embedding unit. This focus weight embedding unit includes an encoding subunit and multiple cascaded focus weight allocation subunits. First, the first fused scene perception feature set is loaded into the encoding subunit for encoding representation. For example, there are 50 features in this fused scene perception feature set. The encoding subunit converts these features into a coded representation data according to a certain algorithm. This data may be in the form of a vector, in which each element represents a certain numerical representation of the original feature after encoding.

[0044] Then, this encoded representation data is loaded into multiple cascaded focus weight allocation subunits for focus weight fusion learning.

[0045] Initialize the multiple cascaded focus weight allocation subunits in the first focus weight embedding unit. For example, determine the initial state parameters of each focus weight allocation subunit, where the initial value of the internal parameter of one focus weight allocation subunit may be set to some default weight values, and the initial mode of data processing is to process the input data in sequence. At the same time, analyze the structural information of each focus weight allocation subunit to determine the internal data flow, storage structure and connection relationship with other focus weight allocation subunits. For example, there is a hierarchical structure inside a certain focus weight allocation subunit, and the data processing method between different levels is that the upper layer passes the processing results to the lower layer, and in the interactive relationship, the lower layer will make further weight adjustments based on the results of the upper layer. Determine the cascade parameters between each focus weight allocation subunit, for example, the transmission format in the cascade process is a specific vector format, and the transmission order is to pass from the first focus weight allocation subunit to the back in sequence.

[0046] According to the structural information of the focus weight allocation subunit, the coded representation data is adaptively adjusted. For example, if a focus weight allocation subunit requires the input data to be in the range of 0 to 1, and some elements in the original coded representation data are not in this range, these elements are normalized to generate adaptively adjusted coded representation data and initialized focus weight allocation subunits.

[0047] The coded representation data is loaded into the first focus weight allocation subunit. During the loading process, an initial weight is assigned to each feature in the coded representation data based on the initial state parameters of the first focus weight allocation subunit. For example, according to the predefined rules, if a feature has a high correlation with driving safety (such as the distance feature of the obstacle in front of the vehicle), a higher initial weight, such as 0.8, is assigned; if a feature is relatively unimportant (such as the color feature of some minor objects in the surrounding environment), a lower initial weight, such as 0.1, is assigned. In this way, the coded representation data after the initial weight is assigned forms a feature representation with the initial weight in the first focus weight allocation subunit, and the coded representation data after the initial weight allocation in the first focus weight allocation subunit is obtained.

[0048] The coded representation data in the first focus weight allocation subunit is transferred to the next focus weight allocation subunit. During the transfer process, the coded representation data is adjusted according to the connection relationship and data processing rules between the two focus weight allocation subunits. For example, if the weight adjustment strategy of the next focus weight allocation subunit is to make further adjustments based on the importance of the output features of the previous focus weight allocation subunit, and the internal state of the current focus weight allocation subunit is to pay more attention to speed-related features, then the weight of the speed-related features in the coded representation data will be further increased. In this way, it is carried out sequentially between the cascaded focus weight allocation subunits to generate coded representation data after the weights and data attributes are adjusted sequentially by multiple cascaded focus weight allocation subunits.

[0049] After the last focus weight allocation subunit is processed, the fusion feature is extracted from the weight-adjusted coded representation data. This fusion feature reflects the feature importance distribution of the coded representation data in the focus weight allocation process. For example, after processing, it is found that the weights of the obstacle distance feature in front of the vehicle and the vehicle speed feature become more prominent in the entire weight allocation process, which means that in this fusion scene perception feature set, these two features are more critical for describing the driving state. Based on this fusion feature, the coded representation data is updated to generate iterative coded representation data, which is the first focus weight coded representation data.

[0050] In the same way, the server processes the other two fused scene perception feature sets to obtain the second and third focus weight encoding representation data respectively.

[0051] Then, the server integrates the three focus weight encoding representation data and loads them into the prediction unit for driving state prediction. The prediction unit analyzes the integrated feature data based on the pre-learned model parameters and algorithms. For example, if the integrated feature data shows that there is an obstacle in front of the vehicle and the vehicle itself is moving at a high speed, and the surrounding light is dim, the prediction unit may judge the driving state as dangerous based on this information and generate a driving state prediction result. This result represents the driving safety of the first autonomous driving vehicle's perception of driving safety, and provides a basis for subsequent early warning reminders.

[0052] Step S150: Based on the driving state prediction result, a warning reminder is issued to the first automatic driving vehicle.

[0053] When the server obtains the driving state prediction result, if the prediction result shows that the driving state is dangerous, such as the aforementioned situation where there is an obstacle in front of the vehicle and the speed is high, the server will send a warning reminder instruction to the first autonomous driving vehicle.

[0054] After the vehicle receives this early warning reminder instruction, a warning message can be popped up through the display system inside the car, such as "There is an obstacle ahead, please slow down". At the same time, the voice system in the car will also issue a corresponding voice warning to remind the driver or the automatic driving system to take corresponding measures, such as slowing down or avoiding.

[0055] If the prediction results show that the driving status is safe, the vehicle can continue to drive normally. The server will continue to monitor the status of the vehicle, continuously acquire new multi-sensor fusion data, and repeat the above steps to predict and monitor the driving status to ensure the safety of the vehicle throughout the driving process.

[0056] Based on the above steps, the embodiment of the present application significantly improves the perception accuracy and robustness of the autonomous driving system by efficiently integrating and utilizing the rich perception data collected by multiple sensors. Specifically, the multi-source sensor data is first arranged in an orderly manner to form a unified multi-sensor fusion data, and then feature extraction is performed to construct a comprehensive scene perception feature set. Through a multi-dimensional privileged domain fusion strategy, multiple fusion scene perception feature sets that can reflect spatiotemporal pattern characteristics are generated, which enhances the ability to understand the driving environment. Finally, the machine learning network model is used to achieve accurate prediction of the driving state, and the driving safety assessment is output in a timely manner. Based on this, early warning reminders are given, which effectively reduces the safety risks in the autonomous driving process, improves the intelligence level of driving decision-making, and provides users with a safer and more reliable autonomous driving experience.

[0057] In a possible implementation, the fusion dimensions include X, and any one of the X fusion dimensions is represented as the i-th fusion dimension, where i is a positive integer and i is not greater than X.

[0058] Step S130 includes: Step S131 , setting i=1, marking the first scene-aware feature set as the first fused scene-aware feature set corresponding to the first fusion dimension.

[0059] Step S132: if i<X, set i=i+1, perform fusion processing of the i+1th fusion dimension on the first scene perception feature set, and generate the i+1th fusion set, where i+1 is an integer greater than 1 and not greater than X.

[0060] Step S133, performing feature extraction on the i+1th fusion set to generate the i+1th fusion scene perception feature set corresponding to the i+1th fusion dimension.

[0061] Step S134: if i=X, obtain X fusion scene perception feature sets corresponding to the X fusion dimensions respectively.

[0062] In a possible implementation, the fusion dimension is configured to set the scale of members that are fused into the same feature cluster during the fusion process of the corresponding fusion dimension, and the i-th fusion dimension is configured to set the number of members that are fused into the same feature cluster during the fusion process of the i-th fusion dimension.

[0063] Step S132 includes: According to the setting of the i+1 fusion dimension, the first scene-aware feature set is grouped based on the order of each member in the first scene-aware feature set to generate an i+1 fusion set. The i+1 fusion set includes one or more feature clusters, each of which is fused with the ordered i+1 members.

[0064] Step S133 includes: Step S1331, connecting the members in each feature cluster in the i+1th fused set to generate the i+1th reference scene-aware feature set. The i+1th reference scene-aware feature set includes one or more aggregated features, each of which is generated by aggregating the members in a feature cluster in the i+1th fused set.

[0065] Step S1332 , performing feature extraction on the aggregated features in the i+1th reference scene-aware feature set to generate the i+1th fused scene-aware feature set corresponding to the i+1th fusion dimension.

[0066] In this embodiment, the aforementioned scenario of an autonomous driving vehicle is continued as an example. Assume that the first autonomous driving vehicle is driving on a complex road with multiple road conditions, and the server is responsible for processing and analyzing the data collected by the vehicle.

[0067] First, set the fusion dimension X=5. When starting to perform privileged domain fusion of multiple fusion dimensions on the first scene perception feature set, set i=1. At this time, the server directly marks the first scene perception feature set as the first fused scene perception feature set corresponding to the first fusion dimension. This first fused scene perception feature set contains various scene perception-related features extracted from the previous sensor data after a series of processing, such as the location, type, speed and other information of objects around the vehicle.

[0068] Next, because i=1 is less than X=5, i=i+1 is set, that is, i becomes 2. Now the first scene-aware feature set is to be fused in the second fusion dimension. According to the setting, the second fusion dimension is configured so that the members fused into the same feature cluster in the fusion process of the second fusion dimension include 2. According to this setting, the server groups the first scene-aware feature set based on the order of each member in the first scene-aware feature set.

[0069] For example, the first scene perception feature set includes many feature members such as the distance, speed, and type of the vehicle in front, the location and shape of the nearby buildings, the curvature and slope of the road, etc. According to the order of the members, every two members are grouped together. For example, the distance and speed of the vehicle in front are grouped together, the location and shape of the nearby buildings are grouped together, and so on. In this way, the second fusion set is generated, which contains multiple feature clusters, and each feature cluster is fused with two members in order.

[0070] Then, the server needs to perform feature extraction on the second fusion set to generate the second fusion scene perception feature set corresponding to the second fusion dimension. First, connect the members in each feature cluster in the second fusion set. For example, for the feature cluster of the distance and speed of the vehicle in front, connect the distance value and the speed value in some way to form a new aggregate feature. For the feature cluster of the position and shape of the nearby buildings, they are also connected to form another aggregate feature. In this way, the second reference scene perception feature set is generated, which contains multiple such aggregate features, each of which is generated by aggregating the members in a feature cluster in the second fusion set.

[0071] Afterwards, feature extraction is performed on the aggregated features in the second reference scene perception feature set. For example, for aggregated features that include the distance and speed of the vehicle ahead, the server may further analyze the relationship between this aggregated feature and other relevant information, and extract some more advanced features, such as the safety factor of the vehicle's distance from the vehicle ahead at the current speed. By performing such processing on each aggregated feature, a second fusion scene perception feature set corresponding to the second fusion dimension is generated. This set can reflect the spatiotemporal pattern characteristics of the first multi-sensor fusion data in the second fusion dimension from another perspective. It integrates the information after the previous grouping and connection operations, and further explores the potential relationships in the data through feature extraction.

[0072] Since i=2 is still less than X=5, i=i+1 is set again, and i becomes 3. Then, the first scene perception feature set is fused in the third fusion dimension. The third fusion dimension is configured in the fusion process set in the third fusion dimension, and the members fused into the same feature cluster include 3.

[0073] The server groups the members in the first scene perception feature set based on the order of the members according to the setting of the third fusion dimension. For example, from the previously ungrouped feature members, they are divided into groups of three in order. Assuming that there is traffic sign information in front of the vehicle, speed limit information of the current road section, and acceleration information of the vehicle itself, these three members are grouped together. In this way, the third fusion set is generated, which contains multiple such feature clusters, each of which is a fusion of the three members in order.

[0074] Next, the members of each feature cluster in the third fusion set are connected to generate the third reference scene perception feature set. For example, for the feature cluster mentioned above that includes the traffic sign information in front of the vehicle, the speed limit information of the current road section, and the acceleration information of the own vehicle, these three pieces of information are connected in a specific way to form an aggregate feature. The entire third reference scene perception feature set contains multiple such aggregate features.

[0075] Afterwards, feature extraction is performed on the aggregated features in the third reference scene perception feature set. For example, for aggregated features including traffic signs, speed limits, and vehicle acceleration, the server may analyze the degree of compliance with the speed limit and traffic sign requirements under the current vehicle acceleration, and extract features such as whether there is a risk of speeding relative to traffic signs. In this way, the third fusion scene perception feature set corresponding to the third fusion dimension is generated. This set further explores the spatiotemporal pattern characteristics of the data under the third fusion dimension, reflecting some characteristics of the combination of different types of information.

[0076] According to the same process, since i=3 is less than X=5, i=i+1 is set to 4, and the fusion process of the fourth fusion dimension is performed. The fourth fusion dimension is configured in the fusion process set in the fourth fusion dimension, and the members fused into the same feature cluster include 4.

[0077] The server groups the remaining members in the first scene perception feature set in order. For example, the steering angle of the vehicle, the distribution of surrounding vehicles, the current weather conditions (such as rainfall intensity), and the friction information of the road are selected as one group to generate the fourth fusion set. This set contains multiple such feature clusters, each of which fuses 4 members.

[0078] Then, the members of each feature cluster in the fourth fusion set are connected to form the fourth reference scene perception feature set. For example, the steering angle of the vehicle, the distribution of surrounding vehicles, the rainfall intensity and the road friction are connected to generate an aggregate feature. The entire fourth reference scene perception feature set consists of multiple such aggregate features.

[0079] Then, feature extraction is performed on the aggregated features in the fourth reference scene perception feature set. For example, the relationship between the steering angle of the vehicle and the distribution of surrounding vehicles under the current weather conditions and road friction is analyzed to extract features similar to the comprehensive evaluation of steering safety. Thus, the fourth fusion scene perception feature set corresponding to the fourth fusion dimension is generated.

[0080] Because i=4 is still less than X=5, i=i+1 is set again to 5. The fusion process of the fifth fusion dimension is performed. The fifth fusion dimension is configured in the fusion process set in the fifth fusion dimension. The members fused into the same feature cluster include 5.

[0081] The server groups the unprocessed members in the first scene perception feature set according to their order. For example, the color of the vehicle ahead, the number of pedestrians around, the current light intensity, the width of the road, and the braking performance of the vehicle are selected as one group to generate the fifth fusion set. This set contains multiple such feature clusters, each of which fuses five members.

[0082] Next, the members of each feature cluster in the fifth fusion set are connected to generate the fifth reference scene perception feature set. For example, the color of the vehicle in front, the number of pedestrians around, the light intensity, the width of the road, and the braking performance of the vehicle are connected to form an aggregate feature, and the fifth reference scene perception feature set contains multiple such aggregate features.

[0083] Finally, feature extraction is performed on the aggregated features in the fifth reference scene perception feature set. For example, the relationship between the color of the vehicle ahead, the number of pedestrians around, and the braking performance of the vehicle itself is analyzed under the current light intensity and road width, and comprehensive consideration features such as braking in response to vehicles ahead and pedestrians in specific roads and environments are extracted. Thus, the fifth fusion scene perception feature set corresponding to the fifth fusion dimension is generated.

[0084] At this time, because i=X=5, the server obtains 5 fusion scene perception feature sets corresponding to the 5 fusion dimensions. These 5 fusion scene perception feature sets process and refine the original first scene perception feature set from different fusion dimensions. Each set reflects the spatiotemporal pattern characteristics of the first multi-sensor fusion data in the corresponding fusion dimension, providing rich and multi-dimensional feature information for subsequent operations such as driving state prediction.

[0085] Step S120 includes: Step S121, obtaining set arrangement nodes between Y perception domains and a plurality of perception knowledge vectors of each of the perception domains. The Y perception domains include: one or more of: spatial positioning, object recognition, dynamic capture, environment understanding, and road condition analysis.

[0086] Step S122, based on the set arrangement nodes between the Y perceptual domains and the multiple perceptual knowledge vectors of each of the perceptual domains, all perceptual knowledge spaces are determined. Any perceptual knowledge space among all the perceptual knowledge spaces is generated by integrating any perceptual knowledge vector included in each of the perceptual domains based on the set arrangement nodes between the Y perceptual domains.

[0087] Step S123: generating a feature identifier of each of the perceptual knowledge spaces according to the number of the perceptual knowledge spaces.

[0088] Step S124, mapping the perceptual knowledge space encoding bits of the perceptual data of each sensor in the first multi-sensor fusion data to the feature identification bits of the corresponding perceptual knowledge space to generate a first scene perception feature set.

[0089] Taking the first autonomous driving vehicle mentioned above as an example, the server starts to extract features based on the perception knowledge space of each sensor perception data in the first multi-sensor fusion data to generate a first scene perception feature set.

[0090] First, the server obtains the set arrangement nodes between Y perception domains and multiple perception knowledge vectors of each perception domain. Here Y is 5, and the five perception domains are spatial positioning, object recognition, dynamic capture, environmental understanding, and road condition analysis. For example, in the scenario of a vehicle driving on a snowy scene, in the spatial positioning perception domain, the perception knowledge vector corresponding to the data collected by the lidar sensor contains the coordinate information of objects around the vehicle relative to the vehicle's own coordinate system. For example, there is an obstacle 10 meters in front of the vehicle, and its coordinates are (x, y, z), as well as the relative positioning relationship between this obstacle and other surrounding objects, such as the distance from the nearby street lamp pole is 5 meters.

[0091] In the object recognition perception domain, the perception knowledge vector obtained after processing the image data collected by the camera includes the recognition results of the surrounding objects. For example, it is recognized that the vehicle in front is a sedan of a certain brand, its shape is a cuboid, and its body color is white. Some appearance features of the vehicle can be used as information to describe the characteristics of the object. These are all perception knowledge vectors in the object recognition perception domain.

[0092] In terms of the dynamic capture perception domain, the data of the millimeter wave radar provides the perception knowledge vector for this perception domain. For example, it detects that the speed of the vehicle in front is 60 kilometers per hour, the speed direction is along the road, and the acceleration gradually increases from 0 to 2 meters per second in the past 5 seconds, etc. These constitute the perception knowledge vector under the dynamic capture perception domain.

[0093] For the environmental understanding perception domain, the server integrates information from multiple sensors. The camera image shows that the sky is dark and there are raindrops, which is the classification structure information of the weather condition. The lidar detects that the ambient light reflection is weak, indicating that the light intensity is low, and this light intensity is associated with the weather condition (rain causes dim light). These are all perception knowledge vectors under the environmental understanding perception domain.

[0094] In the road condition analysis perception domain, it can be known from map data and vehicle sensor information that the current road is a snowy main road with good road conditions and no potholes or construction. This is the perception knowledge vector under the road condition analysis perception domain.

[0095] Based on the set arrangement nodes between these five perception domains, the server determines all perception knowledge spaces. For example, taking a specific moment as an example, the spatial positioning information and object recognition information are first integrated according to the pre-set arrangement node order. For example, the white car (object recognition) identified at a certain coordinate position (spatial positioning) is associated, which initially forms an integrated information combination. Then the dynamic capture information is incorporated, such as the speed of the white car is 60 kilometers per hour (dynamic capture), to further enrich this information combination. Then the environmental understanding information is added, taking into account the impact of the current rainy weather (environmental understanding) on ​​the driving of the white car, such as the possible obstruction of vision. Finally, combined with the road condition analysis information, such as the driving state of the white car on the snowy main road (road condition analysis) may be different from that on the branch road. By integrating a perception knowledge vector of each perception domain according to the set arrangement nodes, a perception knowledge space is generated. In the same way, the different perception knowledge vectors at each moment are integrated to determine all perception knowledge spaces.

[0096] Afterwards, based on the number of all perceptual knowledge spaces, the server generates a feature identifier for each of the perceptual knowledge spaces. Assuming that after the previous integration operation, a total of 100 different perceptual knowledge spaces are obtained, the server assigns a unique feature identifier from 00 to 99 to each perceptual knowledge space.

[0097] Finally, the server encodes the perceptual knowledge space mapping of each sensor perception data in the first multi-sensor fusion data into the feature identification bits of the corresponding perceptual knowledge space. For example, for a certain sensor perception data, its perceptual knowledge space matches the 30th perceptual knowledge space determined previously after integration, so the perceptual knowledge space mapping of this sensor perception data is encoded as 30. By mapping and encoding the perceptual knowledge space of all sensor perception data in this way, the first scene perception feature set is finally generated. This set contains the feature information of all sensor perception data after the above-mentioned complex processing. This information has undergone operations such as integration of perceptual knowledge space, allocation of identification bits, and mapping encoding. It is a highly abstract and refined version of the original multi-sensor fusion data, and provides a basis for subsequent further fusion processing and other operations on the first scene perception feature set.

[0098] In a possible implementation manner, any sensor perception data in the first multi-sensor fusion data is represented as first sensor perception data. The method further includes: Step A110, obtaining set arrangement nodes between Y perception domains, wherein the Y perception domains include: one or more of: spatial positioning, object recognition, dynamic capture, environmental understanding, and road condition analysis.

[0099] Step A120: Acquire the perception knowledge vector of the first sensor perception data in each of the perception domains.

[0100] Step A130: Based on the set arrangement nodes between the Y perception domains, the perception knowledge vectors of the first sensor perception data in each of the perception domains are integrated to generate a perception knowledge space of the first sensor perception data.

[0101] In a possible implementation, step A130 includes: Step A131, for each perception knowledge vector, determine the structural analysis results of each perception knowledge vector. Among them, for the perception knowledge vector under the spatial positioning perception domain, determine the first substructure, the first substructure includes structural elements with coordinate representation and relative positioning relationship in different coordinate systems. For the perception knowledge vector under the object recognition perception domain, determine the second substructure in describing the object features, the second substructure includes a hierarchical structure of object features, and the hierarchical structure includes basic geometric shape description and deep semantic features. For the perception knowledge vector under the dynamic capture perception domain, determine the third substructure of dynamic features, the third substructure includes the vector representation structure of speed, the time series structure of acceleration, etc. For the perception knowledge vector under the environmental understanding perception domain, determine the fourth substructure about environmental factors, the fourth substructure includes the classification structure of weather conditions, the light intensity is represented by a numerical range structure and the association structure with other environmental factors. For the perception knowledge vector under the road condition analysis perception domain, determine the fifth substructure, the fifth substructure includes the structural relationship between road type and road condition.

[0102] Step A132, determining the perceptual domain relationship of each perceptual domain, and generating a perceptual domain relationship model, wherein the perceptual domain relationship includes the order priority of each perceptual domain in the overall perceptual domain relationship, the dependency relationship and the parallel relationship between each perceptual domain.

[0103] Step A133, based on the structural analysis result and the perceptual domain relationship model, pre-alignment operation is performed on the perceptual knowledge vector to generate a perceptual knowledge vector after the pre-alignment operation, and the perceptual knowledge vector after the pre-alignment operation is adapted to be integrated according to the perceptual domain relationship in terms of structure and content.

[0104] Step A134, construct a preliminary integration framework of the perception knowledge vector, and determine the integrated framework structure according to the sequential relationship, dependency relationship and parallel relationship in the perception domain relationship model. For the perception domain with sequential relationship, build the integration chain in sequence; for the perception domain with dependency relationship, mark the flow direction and conversion relationship of the data in the integration framework, and indicate the conversion operations that need to be performed on the flow direction; for the perception domain with parallel relationship, build a parallel integration module, and build an interactive mechanism for sharing information inside the parallel integration module.

[0105] Step A135, hierarchical integration is performed according to the preliminary integration framework to generate the perception knowledge space of the first sensor perception data, wherein, for the perception domain chain of the sequential relationship, integration is performed layer by layer starting from the starting perception domain, and the integration result is integrated with the perception knowledge vector of the next sequential perception domain. For the perception domain of the dependency relationship, integration is performed according to the data flow and conversion relationship in the preliminary integration framework. For the perception domain of the parallel relationship, integration is performed in the parallel integration module according to the interactive mechanism of shared information.

[0106] In this embodiment, firstly, for any sensor perception data (ie, first sensor perception data) in the first multi-sensor fusion data, the server needs to perform a series of operations to generate its perception knowledge space.

[0107] The server first obtains the set arrangement nodes between Y perception domains, where the Y perception domains include spatial positioning, object recognition, dynamic capture, environmental understanding, and road condition analysis. Taking the data of the first autonomous driving vehicle in the actual driving scene as an example, these set arrangement nodes specify the order or relationship between different perception domains in the data integration process.

[0108] Then, the server obtains the perception knowledge vector of the first sensor perception data in each perception domain.

[0109] From the perspective of spatial positioning perception domain, if the first sensor perception data comes from the laser radar, the structural analysis result corresponding to its perception knowledge vector is the first substructure. For example, during vehicle driving, the laser radar detects a traffic sign 10 meters in front of the vehicle, and its coordinates in the vehicle coordinate system are (x=5 meters, y=0 meters, z=0.5 meters), which is the coordinate representation in different coordinate systems. At the same time, the distance between this traffic sign and the street light pole next to it is 3 meters, which is the structural element of the relative positioning relationship.

[0110] In the object recognition perception domain, if the first sensor perception data is image information collected by the camera, the second substructure of its perception knowledge vector presents a hierarchical structure in describing the object features. For example, a pedestrian is identified on the road, and its basic geometric shape is described as a cylinder (the approximate shape of the human body), and the deep semantic features include the color of the clothes (assuming it is blue), whether it carries items (such as a backpack), etc. This information constitutes a hierarchical description of the object features of the pedestrian.

[0111] For the dynamic capture perception domain, if the first sensor perception data is data collected by the millimeter wave radar, the third substructure of its perception knowledge vector includes the vector representation structure of speed and the time series structure of acceleration. For example, it is detected that the speed of an electric snowmobile in the lane next to it is 50 kilometers per hour, and the speed direction is the same as the direction of the vehicle. This is the vector representation of speed. At the same time, the millimeter wave radar records the change in the acceleration of the electric snowmobile in the past 10 seconds, which gradually increases from 0 to 1 meter / second². This is the time series structure of acceleration.

[0112] In the environmental understanding perception domain, it is assumed that the first sensor perception data is a combination of the data from the camera and other environmental sensors. The fourth substructure of its perception knowledge vector contains the classification structure of weather conditions, the light intensity represented by a numerical range structure, and the association structure with other environmental factors. For example, the camera image shows that there are thin clouds in the sky, and the weather condition is judged to be cloudy (this is the classification structure of the weather condition), and the light sensor detects that the light intensity is 500-800 lux (this is the numerical range structure of the light intensity), and this light intensity is associated with the weather conditions. For example, the light intensity in cloudy weather is moderate, not as strong as on a sunny day, nor as dim as on a cloudy day.

[0113] For the road condition analysis perception domain, the first sensor perception data is combined with map data and vehicle chassis sensor information. The fifth substructure of its perception knowledge vector includes the structural relationship between road type and road condition. For example, through map information, we know that we are currently driving on a snowy branch road, and the chassis sensor detects that the road is relatively flat and has no obvious potholes, which reflects the structural relationship between the road type (snowy branch road) and the road condition (flat and without potholes).

[0114] Next, the server needs to determine the perceptual domain relationship of each perceptual domain and generate a perceptual domain relationship model. In this autonomous vehicle scenario, the spatial positioning perceptual domain may have a higher order priority because the position of an object needs to be determined before it is recognized, so it has a higher order priority in the overall perceptual domain relationship. The object recognition perceptual domain depends on the spatial positioning perceptual domain because objects need to be recognized at a certain position, which is a dependency relationship. The dynamic capture perceptual domain has a certain dependency relationship with the object recognition perceptual domain because the dynamic features of the recognized objects need to be captured. The environmental understanding perceptual domain has a parallel relationship with other perceptual domains. For example, although weather conditions and light intensity are related to object recognition, dynamic capture, etc., they are relatively independent environmental factors and can affect the entire driving scene in parallel. The road condition analysis perceptual domain also has a parallel relationship with other perceptual domains. For example, the road type and condition will affect the driving of the vehicle, but it affects the driving scene in different aspects from perceptual domains such as object recognition. Through such analysis, a perceptual domain relationship model is generated, which includes the order priority of each perceptual domain in the overall perceptual domain relationship, the dependency relationship between each perceptual domain, and the parallel relationship.

[0115] Based on the above structural analysis results and the perceptual domain relationship model, the server performs a pre-alignment operation on the perceptual knowledge vector. Taking the relationship between spatial positioning and object recognition as an example, since object recognition depends on spatial positioning, the pre-alignment operation must ensure that the object position information in the object recognition perceptual knowledge vector matches the coordinate information in the spatial positioning perceptual knowledge vector in terms of structure and content. For example, the coordinate system in spatial positioning must be consistent with the coordinate system of the object's location in object recognition, so that the object identified at a certain location can be accurately associated. For the dynamic capture perceptual knowledge vector, its dynamic information such as speed and acceleration must correspond to the object in object recognition, that is, it must be ensured that it is dynamic information for the same object. For the environmental understanding perceptual knowledge vector, information such as weather conditions and light intensity must be adjusted in a way that can be integrated with other perceptual knowledge vectors. For example, the numerical range of weather conditions and light intensity is formatted according to certain standards so that it can be easily combined with other information in subsequent integration. For the road condition analysis perceptual knowledge vector, the road type and condition information must be organized into a form compatible with other perceptual knowledge vectors, such as expressing the road condition in a quantitative form (such as road roughness index, etc.). After the pre-alignment operation, the perceptual knowledge vector can be adapted to be integrated according to the perceptual domain relationship in terms of structure and content.

[0116] Then, the server builds a preliminary integration framework for the perception knowledge vector. The integration framework structure is determined according to the sequential relationship, dependency relationship, and parallel relationship in the perception domain relationship model. For perception domains with sequential relationships, such as spatial positioning and object recognition, the integration chain is built in sequence. Starting from the spatial positioning perception domain, based on the coordinate information of objects around the vehicle detected by the lidar, the object information at these coordinate positions recognized by the camera is integrated. For perception domains with dependency relationships, such as object recognition and dynamic capture, the flow direction and conversion relationship of the data are marked in the integration framework. For example, the result of object recognition (such as the identified vehicle) should flow to the dynamic capture perception domain, and the conversion operations required in this flow direction are noted, such as associating the vehicle identification in object recognition with the speed and acceleration information in dynamic capture to ensure that the speed and acceleration are for this specific vehicle. For perception domains with parallel relationships, such as environmental understanding and road condition analysis, parallel integration modules are built. Within the parallel integration module of environmental understanding, an interactive mechanism for sharing information is built between information such as weather conditions and light intensity. For example, weather conditions will affect the perception of light intensity. Through this interactive mechanism, environmental factors can be understood more comprehensively. Within the parallel integration module of road condition analysis, there is also information interaction between road type and road condition. For example, road type will affect the evaluation criteria of road condition.

[0117] Finally, hierarchical integration is performed according to the preliminary integration framework to generate the perceptual knowledge space of the first sensor perception data. For the perceptual domain chain of sequential relationship, integration is performed layer by layer starting from the starting perceptual domain. Taking the sequential chain of spatial positioning-object recognition-dynamic capture as an example, the coordinate information of spatial positioning is first integrated with the object information of object recognition to obtain the object information at a specific coordinate position. Then this integration result is integrated with the dynamic information such as speed and acceleration in dynamic capture to obtain the dynamic information of a specific object at a specific position. For the perceptual domain of dependency relationship, integration is performed according to the data flow and conversion relationship in the preliminary integration framework. For example, the objects recognized by the object and the dynamic information in the dynamic capture are integrated according to the previously marked conversion relationship to ensure the accuracy and relevance of the information. For the perceptual domain of parallel relationship, integration is performed in the parallel integration module according to the interactive mechanism of shared information. For example, in the parallel integration module of environmental understanding, weather conditions and light intensity are integrated according to the interactive mechanism between them to obtain comprehensive environmental information. In the parallel integration module of road condition analysis, road type and road condition are integrated according to the internal information interaction method to obtain complete road condition information. Then, the results of sequential relationship integration, dependency relationship integration, and parallel relationship integration are further integrated to generate the perception knowledge space of the first sensor perception data. This perception knowledge space contains information integrated from various perception domains and comprehensively describes the characteristics and relationships of the first sensor perception data in the entire autonomous driving scenario.

[0118] In a possible implementation, the fusion dimensions include X, the machine learning network model includes a prediction unit and X focus weight embedding units, and one fusion scene perception feature set corresponds to one focus weight embedding unit. X is a positive integer.

[0119] Step S140 includes: Step S141: Load the X fused scene perception feature sets of the first autonomous driving vehicle into corresponding focus weight embedding units for spatiotemporal pattern feature learning, and generate X focus weight encoding representation data, each focus weight encoding representation data coming from a fused scene perception feature set.

[0120] Step S142: Integrate the X focusing weight encoding representation data and load them into the prediction unit to perform driving state prediction and generate a driving state prediction result.

[0121] Continuing with the previously mentioned autonomous driving vehicle scenario as an example, the server acts as the executor to operate this part of the content.

[0122] Assuming the fusion dimension X=3, the machine learning network model in the server contains a prediction unit and 3 focus weight embedding units, and each fusion scene perception feature set corresponds to a focus weight embedding unit.

[0123] First, the server loads the three fused scene perception feature sets of the first autonomous driving vehicle into the corresponding focus weight embedding unit for spatiotemporal pattern feature learning. Taking the first fused scene perception feature set as an example, this set contains various feature information related to the driving scene after multiple steps of processing, such as the spatial relationship between the vehicle and surrounding objects, the dynamic characteristics of objects, and environmental factors. After it is loaded into the corresponding focus weight embedding unit, spatiotemporal pattern feature learning begins.

[0124] In this focus weight embedding unit, it is assumed that it includes an encoding subunit and multiple cascaded focus weight allocation subunits. First, the fused scene perception feature set enters the encoding subunit for encoding representation. For example, there are multiple features in this set, such as the distance, speed, type of the object in front of the vehicle, and the light intensity of the surrounding environment. The encoding subunit will convert these features into a coded representation data according to a specific algorithm. This coded representation data may be in the form of a vector, in which each element represents a certain numerical representation of the original feature after encoding. For example, the distance of the object in front of the vehicle may be encoded as a specific value in the vector, which is generated based on the size of the distance and the pre-set encoding rules.

[0125] Then, this encoded representation data is loaded into multiple cascaded focus weight allocation subunits for focus weight fusion learning. These focus weight allocation subunits are initialized and set to determine the initial state parameters of each subunit. For example, the initial value of the internal parameter of one of the focus weight allocation subunits may be set to some default weight values, and the initial mode of data processing is to process the input data in sequence. At the same time, the structural information of each focus weight allocation subunit is analyzed to determine the internal data flow, storage structure, and connection relationship with other focus weight allocation subunits. For example, there is a hierarchical structure inside a focus weight allocation subunit, and the data processing method between different levels is that the upper layer passes the processing results to the lower layer, and in the interactive relationship, the lower layer will make further weight adjustments based on the results of the upper layer. Determine the cascade parameters between each focus weight allocation subunit, for example, the transmission format in the cascade process is a specific vector format, and the transmission order is to pass from the first focus weight allocation subunit to the back in sequence.

[0126] According to the structural information of the focus weight allocation subunit, the coded representation data is adaptively adjusted. For example, if a focus weight allocation subunit requires the input data to be between 0 and 1, and some elements in the original coded representation data are not within this range, these elements are normalized. After that, the coded representation data is loaded into the first focus weight allocation subunit. During the loading process, an initial weight is assigned to each feature in the coded representation data based on the initial state parameters of the first focus weight allocation subunit. For example, according to the predefined rules, if a feature has a high correlation with driving safety (such as the distance feature of the obstacle in front of the vehicle), a higher initial weight is assigned, such as 0.8; if a feature is relatively unimportant (such as the color feature of some minor objects in the surrounding environment), a lower initial weight is assigned, such as 0.1. In this way, the coded representation data after the initial weight is assigned forms a feature representation with an initial weight in the first focus weight allocation subunit, and the coded representation data after the initial weight allocation in the first focus weight allocation subunit is obtained.

[0127] Next, the coded representation data in the first focus weight allocation subunit is transferred to the next focus weight allocation subunit. During the transfer process, the coded representation data is adjusted according to the connection relationship and data processing rules between the two focus weight allocation subunits. For example, if the weight adjustment strategy of the next focus weight allocation subunit is to make further adjustments based on the importance of the output features of the previous focus weight allocation subunit, and the internal state of the current focus weight allocation subunit is to pay more attention to speed-related features, then the weight of the speed-related features in the coded representation data will be further increased. In this way, it is carried out sequentially between the cascaded focus weight allocation subunits to generate coded representation data after the weights and data attributes are adjusted sequentially by multiple cascaded focus weight allocation subunits.

[0128] After the last focus weight allocation subunit is processed, the fusion feature is extracted from the weight-adjusted coded representation data. This fusion feature reflects the feature importance distribution of the coded representation data in the focus weight allocation process. For example, after processing, it is found that the weights of the obstacle distance feature in front of the vehicle and the vehicle speed feature become more prominent in the entire weight allocation process, which means that in this fusion scene perception feature set, these two features are more critical for describing the driving state. Based on this fusion feature, the coded representation data is updated to generate iterative coded representation data, which is the first focus weight coded representation data.

[0129] In the same way, the server processes the other two fused scene perception feature sets to obtain the second and third focus weight encoding representation data respectively.

[0130] Afterwards, the server integrates the three focus weight encoding representation data and loads them into the prediction unit for driving state prediction. For example, the three focus weight encoding representation data may contain different aspects of driving-related information, such as the interaction between the vehicle and the surrounding environment, the vehicle's own dynamic information, and the impact of the environment on driving. The prediction unit analyzes the integrated feature data based on the pre-learned model parameters and algorithms. For example, if the integrated feature data shows that there is an obstacle in front of the vehicle and the vehicle's own speed is fast, and the surrounding light is dim, the prediction unit may judge the driving state as dangerous based on this information and generate a driving state prediction result. This result represents the driving safety of the first autonomous driving vehicle's perception of driving safety, and provides a basis for subsequent possible early warning reminders or other operations.

[0131] In a possible implementation, any one of the X fused scene-aware feature sets is represented as an i-th fused scene-aware feature set, and the X focus weight encoding representation data include the i-th focus weight encoding representation data derived from the i-th fused scene-aware feature set. The i-th fused scene-aware feature set corresponds to the i-th focus weight embedding unit among the X focus weight embedding units, and the i-th focus weight embedding unit includes an encoding subunit and a plurality of cascaded focus weight allocation subunits. i is a positive integer not greater than X.

[0132] Step S141 includes: Step S1411: Load the i-th fused scene-perceptual feature set into the encoding subunit for encoding representation, and generate encoding representation data of the i-th fused scene-perceptual feature set.

[0133] Step S1412: Load the encoded representation data of the i-th fused scene perception feature set into the multiple cascaded focus weight allocation subunits to perform focus weight fusion learning to generate the i-th focus weight encoded representation data.

[0134] In one possible implementation, step S1412 includes; Step S1412-1, initialize the multiple cascaded focus weight allocation subunits in the i-th focus weight embedding unit, and determine the initial state parameters of each focus weight allocation subunit, wherein the initial state parameters include the initial values ​​of internal parameters and determine the initial mode of data processing.

[0135] Step S1412-2, analyze the structural information of each focus weight allocation subunit, determine the internal data flow, storage structure and connection relationship with other focus weight allocation subunits, and specifically analyze whether there is a hierarchical structure inside each focus weight allocation subunit, the data processing method and interaction relationship between different levels. Determine the cascade parameters between each focus weight allocation subunit, and the cascade parameters include the transmission format and transmission order during the cascade process.

[0136] Step S1412-3, adaptively adjusting the coded representation data according to the structural information of the focus weight allocation subunit, generating the adaptively adjusted coded representation data and the initialized focus weight allocation subunit.

[0137] Step S1412-4, load the encoded representation data into the first focusing weight allocation subunit. During the loading process, assign an initial weight to each feature in the encoded representation data based on the initial state parameters of the first focusing weight allocation subunit. The initial weight is assigned based on the predefined rules of the focusing weight allocation subunit. After the initial weight is assigned, the encoded representation data forms a feature representation with the initial weight in the first focusing weight allocation subunit, and the encoded representation data after the initial weight allocation in the first focusing weight allocation subunit is obtained.

[0138] Step S1412-5, transfer the coded representation data in the first focus weight allocation subunit to the next focus weight allocation subunit. During the transfer process, the coded representation data is adjusted according to the connection relationship and data processing rules between the two focus weight allocation subunits. Specifically, the feature weights in the coded representation are adjusted. The adjustment basis is the weight adjustment strategy of the next focus weight allocation subunit. The weight adjustment strategy is used to make adjustments based on the output features of the previous focus weight allocation subunit and the internal state of the current focus weight allocation subunit. This is performed sequentially between the cascaded focus weight allocation subunits to generate coded representation data after the weights and data attributes of multiple cascaded focus weight allocation subunits are adjusted in sequence.

[0139] Step S1412-6, after the last focus weight allocation subunit is processed, a fusion feature is extracted from the weight-adjusted coded representation data. The fusion feature is obtained by weight-adjusting and fusing each feature in the coded representation data through the entire cascaded focus weight allocation subunit, and is used to reflect the feature importance distribution of the coded representation data in the focus weight allocation process.

[0140] Step S1412-7, updating the coding representation data based on the fusion feature to generate iterative coding representation data, wherein the iterative coding representation data is the i-th focusing weighted coding representation data.

[0141] Continuing with the previous example of an autonomous vehicle, this section will be explained with the server acting as the execution entity.

[0142] Assuming that the fusion dimension X=3, the whole process is elaborated in detail by taking one of the fused scene-aware feature sets (represented here as the i-th fused scene-aware feature set, assuming i=1) as an example.

[0143] First, the server loads the first fused scene perception feature set into the encoding subunit in the corresponding first focus weight embedding unit for encoding representation, and generates the encoding representation data of the first fused scene perception feature set. This first fused scene perception feature set contains a lot of feature information related to the driving scene, such as the distribution of objects around the vehicle (including spatial information such as distance and orientation), the types of these objects (vehicles, pedestrians or other obstacles), their dynamic characteristics (speed, acceleration, etc.) and some conditions of the surrounding environment (such as the impact of light and weather on visibility, etc.). The encoding subunit will encode and represent these rich feature information according to the pre-set encoding rules. For example, for the distance information of objects around the vehicle, it may be encoded into a value within a specific range according to the distance, and the closer the distance, the larger the value; for the type of object, a classification encoding method may be used to map different types of objects to different digital codes; for dynamic features such as speed and acceleration, they may be converted into specific encoding values ​​according to a certain proportional relationship. In this way, through the processing of the encoding subunit, the first fused scene perception feature set is converted into a coded representation data, which represents the information in the original fused scene perception feature set in a form suitable for subsequent processing.

[0144] Next, the encoded representation data of the first fused scene perception feature set is loaded into a plurality of cascaded focus weight allocation subunits in the first focus weight embedding unit for focus weight fusion learning to generate the first focus weight encoded representation data.

[0145] The first step is to initialize the multiple cascaded focus weight allocation subunits in the first focus weight embedding unit. Determine the initial state parameters of each focus weight allocation subunit, which include the initial values ​​of internal parameters and the initial mode of determining data processing. For example, the initial values ​​of the internal parameters of the first focus weight allocation subunit may be set to certain default weight values, which may be based on past experience or pre-training. The initial mode of data processing is set to process the input data in sequence, that is, starting from the first feature of the encoded data, and processing them one by one. At the same time, the structural information of each focus weight allocation subunit is analyzed. For example, there is a hierarchical structure inside a focus weight allocation subunit, the upper layer may be a preliminary feature screening and weight allocation of the input data, and the lower layer is further adjusted and optimized according to the results of the upper layer. The data processing method between different levels is that the upper layer passes the processing results to the lower layer, and the lower layer will further adjust the weight according to the results of the upper layer. For the connection relationship between each focus weight allocation subunit, their cascade parameters are determined, including the transmission format and transmission order during the cascade process. Assuming that the transmission format is in vector form, the transmission order is from the first focusing weight allocation subunit to the back in sequence, so that the flow of data between each subunit is clear.

[0146] In the second step, according to the structural information of the focus weight allocation subunit, the coded representation data is adaptively adjusted to generate the adaptively adjusted coded representation data and the initialized focus weight allocation subunit. For example, if a focus weight allocation subunit requires the input data to be between 0 and 1, and some elements in the original coded representation data are not within this range, these elements need to be normalized. At the same time, the focus weight allocation subunit is initialized to be ready to process the adaptively adjusted coded representation data.

[0147] The third step is to load the coded representation data into the first focus weight allocation subunit. During the loading process, an initial weight is assigned to each feature in the coded representation data based on the initial state parameters of the first focus weight allocation subunit. The basis for the allocation of this initial weight is the predefined rule of the focus weight allocation subunit. For example, for features closely related to driving safety, such as the distance of the obstacle in front of the vehicle, if the distance is closer, it means that it has a greater impact on driving safety, then a higher initial weight will be assigned, assuming it is 0.8; while for some relatively minor features, such as the color features of some non-critical objects in the surrounding environment, which have less impact on driving safety, a lower initial weight will be assigned, such as 0.1. In this way, the coded representation data after the initial weight is assigned forms a feature representation with an initial weight in the first focus weight allocation subunit, thereby obtaining the coded representation data after the initial weight allocation in the first focus weight allocation subunit.

[0148] The fourth step is to transfer the coded representation data in the first focus weight allocation subunit to the next focus weight allocation subunit. During the transfer process, the coded representation data is adjusted according to the connection relationship and data processing rules between the two focus weight allocation subunits, specifically, the feature weights in the coded representation are adjusted. The adjustment is based on the weight adjustment strategy of the next focus weight allocation subunit, which is used to adjust based on the output features of the previous focus weight allocation subunit and the internal state of the current focus weight allocation subunit. For example, if the weight of a feature (such as the speed of the vehicle) in the result output by the previous focus weight allocation subunit is high, and the internal state of the current focus weight allocation subunit is more concerned about the risks related to speed (because it may be in a complex road condition at present), then the weight of this speed feature will be further increased during the transfer process. In this way, this adjustment of weights and data attributes is performed in sequence between the cascaded focus weight allocation subunits, and the coded representation data after the weights and data attributes are adjusted in sequence by multiple cascaded focus weight allocation subunits is gradually generated.

[0149] Finally, after the last focus weight allocation subunit is processed, the fusion feature is extracted from the weight-adjusted coded representation data. This fusion feature is obtained by weight adjustment and fusion of each feature in the coded representation data through the entire cascade of focus weight allocation subunits. It reflects the feature importance distribution of the coded representation data in the focus weight allocation process. For example, after a series of weight adjustments, it is found that the distance feature of the obstacle in front of the vehicle and the speed feature of the vehicle itself become more prominent in the entire weight allocation process, which indicates that in this fusion scene perception feature set, these two features are more critical for describing the driving state. Based on this fusion feature, the coded representation data is updated to generate iterative coded representation data, which is the first focus weight coded representation data.

[0150] In the same way, for the fused scene perception feature sets of i=2 and i=3, the server also loads them into the corresponding second and third focus weight embedding units respectively, repeating the above steps to generate the second and third focus weight coding representation data. In this way, by performing the above processing on these three fused scene perception feature sets (when X=3), the server obtains three focus weight coding representation data, which contain the fused scene perception feature information after complex processing, and highlight the importance of different features in driving state judgment through focus weight coding, providing a more targeted and effective data basis for subsequent driving state prediction. The whole process makes full use of the structure and function of the encoding subunit in the focus weight embedding unit and multiple cascaded focus weight allocation subunits, and deeply mines and processes the fused scene perception feature set to meet the needs of accurate prediction of driving state in autonomous driving scenarios.

[0151] In a possible implementation, the method further includes: Step B110, obtaining second multi-sensor fusion data, where the second multi-sensor fusion data is generated by arranging sensor perception data collected by multiple sensors of the second autonomous driving vehicle within a set driving segment based on the autonomous driving perception timing.

[0152] Step B120: extracting features from the second multi-sensor fusion data based on the perception knowledge space of each sensor perception data in the second multi-sensor fusion data to generate a second scene perception feature set.

[0153] Step B130, performing privileged domain fusion of multiple fusion dimensions on the second scene perception feature set to generate multiple fused scene perception feature sets of the second autonomous driving vehicle.

[0154] Step B140, predicting the driving state of multiple fused scene perception feature sets of the second autonomous driving vehicle through a basic machine learning network model, and generating a driving state prediction result, wherein the driving state prediction result represents the driving safety perceived by the second autonomous driving vehicle.

[0155] Step B150, determining the driving state prediction error information of the basic machine learning network model based on the driving state prediction result and the driving state annotation data of the second automatic driving vehicle.

[0156] Step B160, based on the direction of minimizing the driving state prediction error information, optimize the network parameter information of the basic machine learning network model to generate the machine learning network model.

[0157] In this embodiment, first, the server obtains the second multi-sensor fusion data. The second autonomous driving vehicle travels in a specific set driving section, which may be a section from a commercial area in the snow to a residential area, including different road types (such as main roads, branch roads), traffic conditions (such as congested intersections, unobstructed sections) and environmental scenes (such as commercial areas with tall buildings, residential areas with pedestrian activities). During the driving process, multiple sensors equipped with the vehicle, such as cameras, laser radars, millimeter-wave radars, etc., continuously collect data according to the automatic driving perception sequence. The camera continuously captures the visual images around the vehicle and may generate 30 frames of image data per second. These images contain information such as road signs, surrounding vehicles, pedestrians and buildings; the laser radar emits laser beams at a high frequency (for example, once every 0.1 seconds) and receives reflected signals to construct an accurate point cloud map of the vehicle's surrounding environment, thereby obtaining information such as the distance and shape of surrounding objects; the millimeter-wave radar focuses on monitoring the speed, distance and other dynamic information of moving objects in front, behind and to the side of the vehicle, and updates the data every certain period of time. The server arranges the data collected by these different sensors according to the autonomous driving perception time sequence to generate the second multi-sensor fusion data. This fusion data is like a collection of all-round information about the vehicle's surrounding environment at every moment during driving, providing rich materials for subsequent processing.

[0158] Next, based on the perception knowledge space of each sensor perception data in the second multi-sensor fusion data, the server extracts features from the second multi-sensor fusion data to generate a second scene perception feature set. The server will first determine the relationship between multiple perception domains similar to the previous ones. The perception domains here still include spatial positioning, object recognition, dynamic capture, environmental understanding, road condition analysis, etc. For the spatial positioning perception domain, the coordinate information of objects around the vehicle relative to the vehicle's own coordinate system is obtained from the laser radar data. For example, there is a trash can 3 meters to the left of the vehicle, and there is a car 10 meters in front. These coordinate information and the relative positioning relationship between objects constitute the perception knowledge vector of sensor perception data under the spatial positioning perception domain. In the object recognition perception domain, the image data collected by the camera can be processed by the image recognition algorithm to identify various objects, such as the brand, color, and model of the car in front. These features are organized in a hierarchical structure. For example, the basic geometric shape is described as a cuboid, and the deep semantic features include brand logos, specific exterior decorations, etc., forming a perception knowledge vector under the object recognition perception domain. In the dynamic capture perception domain, the millimeter-wave radar provides information such as vehicle speed and acceleration. For example, the speed of the vehicle in front is 50 kilometers per hour and is decelerating at an acceleration of 1 meter per second squared. These dynamic features constitute the perception knowledge vector in the form of the vector representation structure of the speed and the time series structure of the acceleration. For the environmental understanding perception domain, the weather conditions (such as sunny days), light intensity (the light intensity is strong through the light sensor data) in the comprehensive camera image, and the relationship between these environmental factors (strong light and good visibility on sunny days) form the perception knowledge vector. The road condition analysis perception domain determines the road type (such as snowy branches) and road conditions (such as flat roads and no obstacles) as the perception knowledge vector based on the map data and chassis sensor information.

[0159] Based on the set relationship between these perception domains, the server integrates the perception knowledge vectors of each perception domain to determine all the perception knowledge spaces. For example, the information about a car 10 meters in front of the vehicle in spatial positioning is integrated with the brand, color and other information of the car in object recognition, and then combined with the speed and acceleration of the car in dynamic capture, the impact of weather and lighting on vehicle driving in environmental understanding, and the road type and condition in road condition analysis, thus forming a perception knowledge space. Based on the number of all perception knowledge spaces, feature identifiers are generated for each perception knowledge space, and then the perception knowledge space mapping of each sensor perception data in the second multi-sensor fusion data is encoded into corresponding feature identifiers, and finally a second scene perception feature set is generated. This set is a refinement of the original fusion data, which contains feature information related to the driving scene extracted from different perception angles.

[0160] Afterwards, the server performs privileged domain fusion of multiple fusion dimensions on the second scene perception feature set to generate multiple fusion scene perception feature sets of the second autonomous driving vehicle. Assume that the number of fusion dimensions is 3. First, according to specific fusion dimension rules, starting from the first fusion dimension, the features in the second scene perception feature set are recombined and fused. For example, under the first fusion dimension, the features related to the area in front of the vehicle (such as the distance, speed, type of the vehicle in front, and the curvature of the road in front) are fused together according to a certain set mode to form a partially fused scene perception feature under the first fusion dimension. Then, for the second fusion dimension, according to the set fusion method, the features related to the side environment of the vehicle (such as the distance of the side vehicle, the distribution of surrounding pedestrians, and the width of the side road) are fused to generate fused scene perception features under the second fusion dimension. Finally, in the third fusion dimension, the features related to the overall driving environment of the vehicle (such as the overall traffic flow, the impact of weather conditions on the entire driving area, and the overall type of the road) are fused to obtain the fused scene perception features under the third fusion dimension. In this way, three fusion scene perception feature sets of the second autonomous driving vehicle are generated, each of which reflects the characteristic information of the vehicle driving scene from a different fusion dimension.

[0161] Subsequently, the server predicts the driving state of multiple fused scene perception feature sets of the second autonomous driving vehicle through the basic machine learning network model to generate a driving state prediction result. This basic machine learning network model contains multiple neuron layers, each of which performs different degrees of feature transformation and information extraction on the input fused scene perception feature set. For example, the first neuron layer may perform a preliminary linear transformation on the input features to extract some basic feature patterns, such as the relative relationship between the vehicle speed and the speed of surrounding vehicles. As the data is transmitted in the network model, the subsequent neuron layers will further explore deeper feature relationships, such as comprehensively judging whether the vehicle is in a safe driving state based on factors such as the vehicle speed, the distance of the obstacle ahead, and the complexity of the surrounding environment. If the model predicts that the vehicle's driving state is safe, it may be because the vehicle speed is moderate, the road ahead is clear, and there are no dangerous obstacles around; if it is predicted to be dangerous, it may be because the vehicle speed is too fast and there are pedestrians or obstacles that suddenly appear in front. This driving state prediction result represents the driving safety of the second autonomous driving vehicle's perceived driving safety.

[0162] Next, the server determines the driving state prediction error information of the basic machine learning network model based on the driving state prediction result and the driving state annotation data of the second autonomous driving vehicle. The driving state annotation data of the second autonomous driving vehicle is the real driving state information obtained in advance by manual annotation or other high-precision detection means. For example, at a certain moment, the driving state annotation data shows that the vehicle is in a safe driving state because the vehicle is driving at the prescribed speed and the surrounding environment is normal, but the basic machine learning network model predicts that the vehicle is in a dangerous driving state at this moment, which results in a prediction error. The server calculates the overall prediction error information by comparing a large number of such prediction results and annotation data. This error information can be measured in a variety of ways, such as the mean square error (MSE), which is the average of the squares of the difference between the prediction results and the annotation data.

[0163] Finally, based on the direction of minimizing the driving state prediction error information, the server optimizes the network parameter information of the basic machine learning network model and generates a machine learning network model. The server uses an optimization algorithm, such as a gradient descent algorithm, to adjust various parameters in the network model, such as the connection weights and bias terms between neurons. In the gradient descent algorithm, according to the gradient of the prediction error for each network parameter (i.e., the error change rate), the parameter value is adjusted in the opposite direction of the gradient so that the prediction error gradually decreases. For example, if an increase in a connection weight causes an increase in the prediction error, the value of the connection weight is reduced. After multiple such iterative adjustments, the network parameters are continuously optimized until a satisfactory effect is achieved, that is, the prediction error is within an acceptable range or cannot be further reduced. In this way, an optimized machine learning network model is generated, which has higher accuracy and reliability in predicting the driving state of autonomous vehicles and can be better applied to subsequent monitoring and management tasks of autonomous vehicles.

[0164] Figure 2 The electronic device 100 shown includes: a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, such as through a bus 1002. Optionally, the electronic device 100 may also include a transceiver 1004, which may be used for data interaction between the server and other servers, such as data transmission and / or data reception. It should be noted that the transceiver 1004 is not limited to one in actual scheduling, and the structure of the electronic device 100 does not constitute a limitation on the embodiments of the present application.

[0165] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0166] The bus 1002 may include a path to transmit information between the above components. The bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 1002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0167] The memory 1003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store program code and can be read by a computer, without limitation herein.

[0168] The memory 1003 is used to store program codes for executing the embodiments of the present application, and the execution is controlled by the processor 1001. The processor 1001 is used to execute the program codes stored in the memory 1003 to implement the steps shown in the above method embodiments.

[0169] In addition, an embodiment of the present application also provides a readable storage medium, in which computer executable instructions are pre-installed. When the processor executes the computer executable instructions, the above-mentioned multi-sensor fusion autonomous driving perception method is implemented.

[0170] Similarly, it should be noted that in order to simplify the description disclosed in this application, thereby helping to understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present application, multiple features are sometimes combined into one embodiment, drawings, or descriptions thereof. Similarly, it should be noted that in order to simplify the description disclosed in this application, thereby helping to understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present application, multiple features are sometimes combined into one embodiment, drawings, or descriptions thereof.

Claims

1. A multi-sensor fusion autonomous driving perception method, characterized in that: The method comprises: Acquire first multi-sensor fusion data, where the first multi-sensor fusion data is generated by arranging sensor perception data collected by a plurality of sensors of the first autonomous driving vehicle in a set driving segment based on an autonomous driving perception timing sequence; Extracting features from the first multi-sensor fusion data according to the perception knowledge space of each sensor perception data in the first multi-sensor fusion data to generate a first scene perception feature set; Perform privileged domain fusion of multiple fusion dimensions on the first scene perception feature set to generate multiple fusion scene perception feature sets of the first autonomous driving vehicle; wherein one fusion dimension corresponds to one fusion scene perception feature set, and each fusion scene perception feature set represents the spatiotemporal pattern characteristics of the first multi-sensor fusion data in the corresponding fusion dimension; Predicting a driving state of a plurality of fused scene perception feature sets of the first autonomous driving vehicle according to a machine learning network model to generate a driving state prediction result, wherein the driving state prediction result represents the driving safety of the first autonomous driving vehicle in terms of perceived driving safety; Based on the driving state prediction result, a warning reminder is given to the first automatic driving vehicle.

2. The multi-sensor fusion autonomous driving perception method according to claim 1, characterized in that: The fusion dimensions include X, and any one of the X fusion dimensions is represented as the i-th fusion dimension, where i is a positive integer and i is not greater than X; The performing privileged domain fusion of multiple fusion dimensions on the first scene perception feature set to generate multiple fusion scene perception feature sets of the first autonomous driving vehicle includes: Set i=1, and mark the first scene-aware feature set as the first fused scene-aware feature set corresponding to the first fusion dimension; If i<X, then set i=i+1, perform fusion processing of the i+1th fusion dimension on the first scene perception feature set to generate an i+1th fusion set, where i+1 is an integer greater than 1 and not greater than X; Perform feature extraction on the i+1th fusion set to generate the i+1th fusion scene perception feature set corresponding to the i+1th fusion dimension; If i=X, then X fusion scene perception feature sets corresponding to the X fusion dimensions are obtained.

3. The multi-sensor fusion autonomous driving perception method according to claim 2, characterized in that: The fusion dimension is configured to set the scale of members fused into the same feature cluster in the fusion process of the corresponding fusion dimension, and the i-th fusion dimension is configured to set the scale of members fused into the same feature cluster in the fusion process of the i-th fusion dimension to include i; The performing fusion processing of the first scene perception feature set in the (i+1)th fusion dimension to generate the (i+1)th fusion set includes: According to the setting of the i+1 fusion dimension, the first scene-aware feature set is grouped based on the order of each member in the first scene-aware feature set to generate an i+1 fusion set; wherein the i+1 fusion set includes one or more feature clusters, each of which is fused with the i+1 members in the sorted order; The step of extracting features from the (i+1)th fusion set to generate the (i+1)th fusion scene perception feature set corresponding to the (i+1)th fusion dimension includes: Connecting members in each feature cluster in the i+1th fused set to generate an i+1th reference scene-aware feature set; the i+1th reference scene-aware feature set includes one or more aggregated features, each of which is generated by aggregating members in a feature cluster in the i+1th fused set; Feature extraction is performed on the aggregated features in the (i+1)th reference scene-aware feature set to generate an (i+1)th fused scene-aware feature set corresponding to the (i+1)th fusion dimension.

4. The multi-sensor fusion autonomous driving perception method according to any one of claims 1 to 3, characterized in that: The step of extracting features from the first multi-sensor fusion data based on the perception knowledge space of each sensor perception data in the first multi-sensor fusion data to generate a first scene perception feature set includes: Acquire set arrangement nodes between Y perception domains and multiple perception knowledge vectors of each of the perception domains; the Y perception domains include: one or more of: spatial positioning, object recognition, dynamic capture, environmental understanding, and road condition analysis; Based on the set arrangement nodes between the Y perception domains and the multiple perception knowledge vectors of each of the perception domains, all perception knowledge spaces are determined; any one of the perception knowledge spaces is generated by integrating any one of the perception knowledge vectors included in each of the perception domains based on the set arrangement nodes between the Y perception domains; Generating a feature identification bit of each perceptual knowledge space in all the perceptual knowledge spaces according to the number of all the perceptual knowledge spaces; The perceptual knowledge space of each sensor perception data in the first multi-sensor fusion data is mapped to the feature identification bits of the corresponding perceptual knowledge space to generate a first scene perception feature set.

5. The multi-sensor fusion autonomous driving perception method according to any one of claims 1 to 3, characterized in that: Any sensor perception data in the first multi-sensor fusion data is represented as first sensor perception data; The method further comprises: Acquire set arrangement nodes between Y perception domains, where the Y perception domains include: one or more of: spatial positioning, object recognition, dynamic capture, environmental understanding, and road condition analysis; Acquire a perception knowledge vector of the first sensor perception data in each of the perception domains; Based on the set arrangement nodes between the Y perception domains, the perception knowledge vectors of the first sensor perception data under each of the perception domains are integrated to generate a perception knowledge space of the first sensor perception data; The step of integrating the perception knowledge vectors of the first sensor perception data in each perception domain based on the set arrangement nodes between the Y perception domains to generate the perception knowledge space of the first sensor perception data includes: For each perception knowledge vector, determine the structural analysis results of each perception knowledge vector; wherein, for the perception knowledge vector under the spatial positioning perception domain, determine the first substructure, the first substructure includes structural elements with coordinate representation and relative positioning relationship in different coordinate systems; for the perception knowledge vector under the object recognition perception domain, determine the second substructure in describing the object features, the second substructure includes a hierarchical structure of the object features, the hierarchical structure includes basic geometric shape description and deep semantic features; for the perception knowledge vector under the dynamic capture perception domain, determine the third substructure of the dynamic features, the third substructure includes the vector representation structure of the speed, the time series structure of the acceleration, etc.; for the perception knowledge vector under the environmental understanding perception domain, determine the fourth substructure about the environmental factors, the fourth substructure includes the classification structure of the weather conditions, the light intensity is represented by the numerical range structure and the association structure with other environmental factors; for the perception knowledge vector under the road condition analysis perception domain, determine the fifth substructure, the fifth substructure includes the structural relationship between the road type and the road condition; Determine the perceptual domain relationship of each perceptual domain and generate a perceptual domain relationship model, wherein the perceptual domain relationship includes the order priority of each perceptual domain in the overall perceptual domain relationship, the dependency relationship and the parallel relationship between each perceptual domain; Based on the structural analysis result and the perceptual domain relationship model, a pre-alignment operation is performed on the perceptual knowledge vector to generate a perceptual knowledge vector after the pre-alignment operation, wherein the perceptual knowledge vector after the pre-alignment operation is adapted to be integrated according to the perceptual domain relationship in terms of structure and content; Constructing a preliminary integration framework of the perception knowledge vector, determining the integrated framework structure according to the sequential relationship, dependency relationship and parallel relationship in the perception domain relationship model, wherein, for the perception domain of sequential relationship, constructing the integration chain in sequence, for the perception domain of dependency relationship, marking the flow direction and conversion relationship of the data in the integration framework, and indicating the conversion operation required on the flow direction, for the perception domain of parallel relationship, constructing a parallel integration module, and constructing an interactive mechanism for sharing information inside the parallel integration module; Perform hierarchical integration according to the preliminary integration framework to generate a perceptual knowledge space of the first sensor perception data, wherein, specifically for a perceptual domain chain with a sequential relationship, integrate layer by layer starting from a starting perceptual domain, and integrate the integration result with a perceptual knowledge vector of a next sequential perceptual domain; for perceptual domains with a dependency relationship, integrate according to the data flow and conversion relationship in the preliminary integration framework; for perceptual domains with a parallel relationship, integrate in a parallel integration module according to an interactive mechanism for sharing information.

6. The multi-sensor fusion autonomous driving perception method according to claim 1, characterized in that: The fusion dimensions include X, the machine learning network model includes a prediction unit and X focus weight embedding units, one fusion scene perception feature set corresponds to one focus weight embedding unit; X is a positive integer; The step of predicting the driving state of the first autonomous driving vehicle based on the plurality of fused scene perception feature sets according to the machine learning network model to generate a driving state prediction result includes: Loading the X fused scene perception feature sets of the first autonomous driving vehicle into corresponding focus weight embedding units for spatiotemporal pattern feature learning, generating X focus weight encoding representation data, each focus weight encoding representation data coming from a fused scene perception feature set; The X focusing weight encoding representation data are integrated and loaded into the prediction unit to perform driving state prediction and generate a driving state prediction result.

7. The multi-sensor fusion autonomous driving perception method according to claim 6, characterized in that: Any one of the X fused scene-aware feature sets is represented as an i-th fused scene-aware feature set, and the X focus weight encoding representation data include an i-th focus weight encoding representation data derived from the i-th fused scene-aware feature set; the i-th fused scene-aware feature set corresponds to an i-th focus weight embedding unit among the X focus weight embedding units, and the i-th focus weight embedding unit includes an encoding subunit and a plurality of cascaded focus weight allocation subunits; i is a positive integer not greater than X; The X fused scene perception feature sets of the first autonomous driving vehicle are respectively loaded into corresponding focus weight embedding units for spatiotemporal pattern feature learning to generate X focus weight encoding representation data, including: Loading the i-th fused scene perception feature set into the encoding subunit for encoding representation, and generating encoding representation data of the i-th fused scene perception feature set; Loading the encoded representation data of the i-th fused scene perception feature set into the multiple cascaded focus weight allocation subunits to perform focus weight fusion learning to generate the i-th focus weight encoded representation data; Wherein, the step of loading the encoded representation data of the i-th fusion scene perception feature set into the multiple cascaded focus weight allocation subunits for focus weight fusion learning to generate the i-th focus weight encoded representation data includes; Initialize the multiple cascaded focusing weight allocation subunits in the i-th focusing weight embedding unit, and determine the initial state parameters of each focusing weight allocation subunit, wherein the initial state parameters include the initial value of the internal parameter and determine the initial mode of data processing; Analyze the structural information of each focusing weight allocation subunit to determine the internal data flow, storage structure and connection relationship with other focusing weight allocation subunits, and specifically analyze whether there is a hierarchical structure inside each focusing weight allocation subunit, the data processing method and interaction relationship between different levels; determine the cascade parameters between each focusing weight allocation subunit, and the cascade parameters include the transmission format and transmission order in the cascade process; According to the structural information of the focus weight allocation subunit, the coded representation data is adaptively adjusted to generate the adaptively adjusted coded representation data and the initialized focus weight allocation subunit; The coded representation data is loaded into a first focusing weight allocation subunit. During the loading process, an initial weight is allocated to each feature in the coded representation data based on an initial state parameter of the first focusing weight allocation subunit. The initial weight is allocated based on a predefined rule of the focusing weight allocation subunit. The coded representation data after the initial weight is allocated forms a feature representation with the initial weight in the first focusing weight allocation subunit, and the coded representation data after the initial weight allocation in the first focusing weight allocation subunit is obtained. The coded representation data in the first focusing weight allocation subunit is transferred to the next focusing weight allocation subunit. During the transfer process, the coded representation data is adjusted according to the connection relationship and data processing rules between the two focusing weight allocation subunits. Specifically, the feature weights in the coded representation are adjusted. The adjustment basis is the weight adjustment strategy of the next focusing weight allocation subunit. The weight adjustment strategy is used to make adjustments based on the output features of the previous focusing weight allocation subunit and the internal state of the current focusing weight allocation subunit. This is performed sequentially between the cascaded focusing weight allocation subunits to generate coded representation data after the weights and data attributes are adjusted sequentially by multiple cascaded focusing weight allocation subunits. After the last focusing weight allocation subunit is processed, a fusion feature is extracted from the weight-adjusted coded representation data, wherein the fusion feature is obtained by weight-adjusting and fusing each feature in the coded representation data through the entire cascaded focusing weight allocation subunit, and is used to reflect the feature importance distribution of the coded representation data in the focusing weight allocation process; The coded representation data is updated based on the fusion feature to generate iterative coded representation data, where the iterative coded representation data is the i-th focused weighted coded representation data.

8. The multi-sensor fusion autonomous driving perception method according to claim 6, characterized in that: The method further comprises: Acquire second multi-sensor fusion data, where the second multi-sensor fusion data is generated by arranging sensor perception data collected by a second autonomous driving vehicle through multiple sensors within a set driving segment based on an autonomous driving perception timing sequence; Extracting features from the second multi-sensor fusion data according to the perception knowledge space of each sensor perception data in the second multi-sensor fusion data to generate a second scene perception feature set; performing privileged domain fusion of multiple fusion dimensions on the second scene perception feature set to generate multiple fused scene perception feature sets of the second autonomous driving vehicle; Predicting a driving state of a plurality of fused scene perception feature sets of the second autonomous driving vehicle by using a basic machine learning network model to generate a driving state prediction result, wherein the driving state prediction result represents the driving safety of the second autonomous driving vehicle in terms of perceived driving safety; Determining driving state prediction error information of the basic machine learning network model based on the driving state prediction result and the driving state annotation data of the second automatic driving vehicle; Based on the direction of minimizing the driving state prediction error information, the network parameter information of the basic machine learning network model is optimized to generate the machine learning network model.

9. A multi-sensor fusion autonomous driving perception device, characterized in that: The multi-sensor fusion autonomous driving perception device stores machine executable instructions, and the machine executable instructions are loaded and executed by a processor to implement the multi-sensor fusion autonomous driving perception method described in any one of claims 1-8.

10. An electronic device, characterized in that: The electronic device includes a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and the machine-executable instructions are loaded and executed by the processor to implement the multi-sensor fusion autonomous driving perception method described in any one of claims 1-8.

Images