Processing method and system for vehicle perception data and vehicle

Abstract

The embodiment of the application provides a kind of vehicle perception data processing method, system and vehicle, the method comprises: obtaining the structured perception data of vehicle, wherein structured perception data is the fusion of the original data perceived by multiple sensors on vehicle;Based on structured perception data, a preset knowledge base is searched, and target knowledge text matching structured perception data is obtained, wherein the knowledge stored in the preset knowledge base is used to represent driving behavior specification or operating mode in dangerous scene;Based on target knowledge text, risk prediction is carried out on structured perception data, and risk prediction result of the environment where the vehicle is located is obtained;Risk prediction result and structured perception data are fused, and target perception data is obtained, wherein target perception data at least includes target object existing risk in scene and risk information of target object.The application solves the technical problem that the accuracy of vehicle perception data in related technologies is low.

Description

Technical Field

[0001] This application relates to the field of vehicle intelligent perception technology, and more specifically, to a method, system, and vehicle for processing vehicle perception data. Background Technology

[0002] Autonomous driving systems can effectively grasp the basic physical state of traffic participants, such as position and speed, through multi-sensor fusion. However, autonomous driving systems in related technologies struggle to effectively understand different scenarios, making it difficult to accurately acquire perception data in specific situations, thus resulting in low accuracy of vehicle perception data in these technologies.

[0003] There is currently no good solution to the above problems. Summary of the Invention

[0004] This application provides a method, system, and vehicle for processing vehicle perception data, in order to at least solve the technical problem of low accuracy of vehicle perception data in related technologies.

[0005] According to one aspect of the embodiments of this application, a method for processing vehicle perception data is provided, comprising: acquiring structured perception data of a vehicle, wherein the structured perception data is obtained by fusing raw data perceived by multiple sensors on the vehicle; retrieving a preset knowledge base based on the structured perception data to obtain target knowledge text matching the structured perception data, wherein the knowledge stored in the preset knowledge base is used to characterize driving behavior norms or operating modes in dangerous scenarios; performing risk prediction on the structured perception data based on the target knowledge text to obtain a risk prediction result of the vehicle's environment; and fusing the risk prediction result and the structured perception data to obtain target perception data, wherein the target perception data includes at least a target object in the scenario that poses a risk and risk information of the target object.

[0006] Furthermore, based on the structured perception data, a search is performed on a preset knowledge base to obtain target knowledge text that matches the structured perception data, including one of the following: extracting at least one keyword from the structured perception data, and searching the preset knowledge base based on the at least one keyword to obtain target knowledge text that matches the at least one keyword; encoding the structured perception data to obtain a query vector, and searching the preset knowledge base based on the query vector to obtain target knowledge text with a similarity greater than a preset similarity to the query vector; searching the preset knowledge base based on at least one keyword and a query vector to obtain target knowledge text that matches at least one keyword and a query vector; constructing a knowledge graph based on the preset knowledge base, and traversing the knowledge graph based on the structured perception data to obtain the target knowledge text;

[0007] The system retrieves a first knowledge text that matches the structured perception data by searching a pre-defined knowledge base based on the structured perception data. Then, it performs a web search based on the structured perception data to obtain a second knowledge text. Finally, it combines the first and second knowledge texts to obtain the target knowledge text.

[0008] Furthermore, the method also includes: acquiring legal information related to vehicle driving and historical driving behavior data; performing natural language processing on the legal information to construct a first knowledge base, wherein the knowledge in the first knowledge base includes driving behavior norms and the scenarios corresponding to the driving behavior norms; performing in-depth mining on the historical driving behavior data to construct a second knowledge base, wherein the knowledge text in the second knowledge base includes objects in the environment and operating modes when objects are at risk; obtaining a preset knowledge base based on the first and second knowledge bases; preferably, performing natural language processing on the legal information to construct the first knowledge base includes: decomposing the legal information to obtain multiple rule atoms, wherein different rule atoms correspond to different driving behavior norms; annotating the multiple rule atoms with scenarios to obtain the scenarios corresponding to the multiple rule atoms; constructing a knowledge graph based on the dependencies between the multiple rule atoms and the scenarios corresponding to the multiple rule atoms to obtain the first knowledge base.

[0009] Furthermore, risk prediction is performed on structured perception data based on target knowledge text to obtain environmental risk prediction results, including: filling the target knowledge text and structured perception data into the prompt word template to generate target prompt words; calling the risk prediction model and inputting the target prompt words into the risk prediction model, using the risk prediction model to perform risk prediction, and obtaining risk prediction results.

[0010] Furthermore, the target knowledge text and structured perception data are filled into the prompt word template to generate target prompt words, including: obtaining historical perception data and the historical risk prediction results corresponding to the historical perception data; filling the target knowledge text, structured perception data, historical perception data and historical risk prediction results into the prompt word template to generate target prompt words; preferably, the risk prediction model is a large model deployed locally, or a model composed of multiple models, or a question-and-answer model.

[0011] Furthermore, the risk prediction results and structured perception data are fused to obtain target perception data, including: associating the risk prediction results with at least one object in the structured perception data to obtain the target object; quantifying the risk level in the risk prediction results to obtain the risk score corresponding to the target object; and spatializing the risk score in a preset space of the vehicle to obtain the risk information of the target object.

[0012] Furthermore, the risk score is spatialized within a pre-defined space of the vehicle to obtain risk information of the target object, including: generating a risk potential field in the pre-defined space based on the risk score and the category of the target object, wherein the risk potential field is centered on the target object; adding semantic tags to the target object, wherein the semantic tags are used to characterize the risk attributes of the target object; and obtaining risk information based on the risk potential field and the semantic tags.

[0013] Furthermore, the method also includes: determining the risk priority of multiple sensing objects based on risk indicators between multiple sensing objects and the vehicle in the structured sensing data; extracting information from the structured sensing data based on the risk priority to obtain the vehicle status, environmental information, and at least one target object; generating sensing text based on the vehicle status, environmental information, and at least one target object; and retrieving a preset knowledge base based on the structured sensing data to obtain target knowledge text that matches the structured sensing data, including: retrieving a preset knowledge base based on the sensing text to obtain target knowledge text that matches the sensing text; and performing risk assessment on the structured sensing data based on the target knowledge text. Risk prediction, obtaining environmental risk prediction results, includes: generating risk prediction text based on perceived text and target knowledge text; performing risk prediction on the risk prediction text to obtain risk prediction results; preferably, after fusing the risk prediction results and structured perceived data to obtain target perceived data, the method further includes: adjusting the vehicle's driving behavior or driving trajectory based on the target perceived data; preferably, after performing risk prediction on structured perceived data based on target knowledge text to obtain environmental risk prediction results, the method further includes: adjusting the vehicle's driving behavior or driving trajectory based on the risk prediction results; and / or, generating prompt information based on the risk prediction results.

[0014] According to another aspect of the embodiments of this application, a vehicle perception data processing system is also provided, comprising: a data acquisition module for acquiring structured perception data of a vehicle, wherein the structured perception data is obtained by fusing raw data perceived by multiple sensors on the vehicle; a knowledge retrieval module for retrieving a preset knowledge base based on the structured perception data to obtain target knowledge text matching the structured perception data, wherein the knowledge stored in the preset knowledge base is used to characterize driving behavior norms or operating modes in dangerous scenarios; a risk prediction module for performing risk prediction on the structured perception data based on the target knowledge text to obtain a risk prediction result for the vehicle's environment; and a result fusion module for fusing the risk prediction result and the structured perception data to obtain target perception data, wherein the target perception data includes at least a target object in the scene that poses a risk and risk information of the target object.

[0015] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0016] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0018] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0019] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0020] In this application embodiment, a method for processing vehicle perception data is proposed. First, structured perception data of the vehicle is acquired. Then, a preset knowledge base is retrieved based on the structured perception data to obtain target knowledge text that matches the structured perception data. Next, risk prediction is performed on the structured perception data based on the target knowledge text to obtain the risk prediction result of the vehicle's environment. Finally, the risk prediction result and the structured perception data are fused to obtain the target perception data.

[0021] This application first collects structured perception data from the vehicle, which is derived from the fusion of raw data perceived by multiple sensors, reflecting the state of objects and environmental conditions in the current driving environment. Next, the structured perception data is used as a query to retrieve target knowledge text highly relevant to the scene from a pre-defined knowledge base. Then, risk detection is performed on the structured perception data based on the target knowledge text, generating risk prediction results. Finally, the risk prediction results are fused with the structured perception data to generate target perception data. Target perception data is an enhanced perception data that integrates cognitive-level risk information, enabling the vehicle to identify and predict the behavior of target objects in the scene, and the risk level and nature of their behavior. This application adopts a scene risk cognition fusion approach based on risk prediction and retrieval enhancement. By combining multi-sensor fused structured data with knowledge text from a pre-defined knowledge base, it generates deep risk prediction results, achieving the goal of improving the accuracy and decision-making quality of vehicle perception data. This results in a more powerful scene understanding and risk prediction ability for autonomous driving systems in complex traffic scenarios, thus solving the technical problem of low accuracy of vehicle perception data in related technologies. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a flowchart of a method for processing vehicle perception data according to an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of a scenario risk cognition fusion system based on a large language model and retrieval enhancement generation according to an embodiment of this application;

[0025] Figure 3 This is a flowchart of a scenario risk cognition fusion method based on a large language model and retrieval enhancement generation according to an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of a vehicle perception data processing system according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] According to an embodiment of this application, an embodiment of a method for processing vehicle perception data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] This embodiment provides a method for processing vehicle perception data. Figure 1 This is a flowchart of a method for processing vehicle perception data according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0031] Step S102: Obtain the structured perception data of the vehicle, wherein the structured perception data is obtained by fusing the raw data perceived by multiple sensors on the vehicle.

[0032] The aforementioned vehicles can refer to vehicles with autonomous driving capabilities. The types of vehicles may include, but are not limited to, vehicles capable of fully automated driving and partially automated driving, with the specific vehicle type to be determined based on actual circumstances. The vehicle is the primary carrier of the vehicle perception data processing method proposed in this application, carrying various sensing devices, computing hardware, and autonomous driving algorithms. The vehicle can be used to achieve safe driving in different scenarios based on the vehicle perception data processing method proposed in this application.

[0033] The aforementioned structured perception data refers to a unified, standardized data format that can be directly read and analyzed, formed by processing and fusing raw data collected from multiple sensors on a vehicle. The types of structured perception data can include, but are not limited to, spatiotemporal coordinate information, object classification information, object state information, and environmental information. The specific type of structured perception data needs to be determined based on actual requirements. Structured perception data can be presented using formats such as JavaScript Object Notation (JSON) and Extensible Markup Language (XML). Structured perception data is fundamental for autonomous driving systems to understand the external environment. It can provide information such as the precise location, speed, acceleration, and direction of motion of objects around the vehicle, helping the vehicle make real-time driving decisions.

[0034] In one optional embodiment, a multi-sensor array, including high-resolution cameras, multi-beam LiDAR, and millimeter-wave radar, is used around the vehicle to capture raw data such as visual information, distance, speed, and orientation from the environment. Subsequently, a data fusion engine collects this real-time perception information from various sources, using algorithms to eliminate redundancy, address blind spots, correct temporal and spatial errors, and ensure the consistency and integrity of the data from each sensor. During this process, the sensor data is transformed into a standard, easily interpretable structured format, such as a JSON object, which contains the precise geometric position, dynamic state (e.g., speed, acceleration), and relationship to the environment (e.g., distance, orientation) of each traffic participant. This structured perception data not only reflects the static layout of the environment but also reveals dynamic changes, providing the autonomous driving system with a comprehensive, real-time updated driving scenario model, which is the cornerstone for the vehicle to make safe and rational driving decisions.

[0035] In one alternative embodiment, data streams from sensing devices can be aggregated and processed in real time, and then a unified, vehicle-centric bird's-eye view (BEV) data, or structured sensing data, can be generated by running an upstream multi-sensor fusion algorithm. This data is organized in a structured format such as JSON and contains the precise geometric and physical states of each traffic participant in the scene.

[0036] Step S104: Retrieve the preset knowledge base based on the structured perception data to obtain the target knowledge text that matches the structured perception data. The knowledge stored in the preset knowledge base is used to represent the driving behavior norms or operating modes in dangerous scenarios.

[0037] The aforementioned pre-built knowledge base refers to a pre-constructed knowledge base, which may include, but is not limited to, a traffic regulations database, a driving behavior pattern database, an accident case database, a road condition information database, and a general knowledge database. The specific pre-built knowledge base needs to be constructed according to actual needs. Through its extensive knowledge related to traffic regulations, safe driving practices, historical accident cases, and complex driving scenarios, the pre-built knowledge base can be used to instantly determine whether driving behavior complies with local traffic regulations, avoid violations, and predict potential risks in a timely manner, thereby enabling preventative measures to be taken in advance.

[0038] The aforementioned target knowledge text refers to a set of knowledge entries or rules most relevant to the current perceived data, found from a pre-defined knowledge base during the retrieval process of structured perception data. Target knowledge text may include, but is not limited to, regulatory matching text, accident prevention text, operational guide text, risk analysis text, etc., and the specific target knowledge text needs to be determined based on the actual retrieval results. Target knowledge text can be used to provide decision-making support, enabling a more accurate assessment of the compliance and potential risks of the current scenario.

[0039] The aforementioned driving behavior guidelines refer to a set of behavioral principles and standards that instruct drivers on how to operate a vehicle in different driving situations. These guidelines can be based on traffic regulations, industry standards, driver's manuals, and professional driver training knowledge. They are used to ensure the safety, order, and efficiency of road traffic.

[0040] The aforementioned operational modes in hazardous scenarios refer to the special driving strategies and execution modes that need to be adopted when facing potential threats or extreme driving conditions. Operational modes may include, but are not limited to, defensive driving modes, emergency avoidance modes, disaster response modes, medical emergency response modes, system failure emergency modes, and traffic control compliance modes. The specific operational mode needs to be determined based on the actual scenario. Operational modes are used to minimize risks and ensure the safety of the vehicle and passengers.

[0041] In one optional embodiment, a vector-based semantic similarity retrieval method can be used to search a preset knowledge base based on structured perception data to obtain target knowledge text that matches the structured perception data. Alternatively, a keyword-based sparse retrieval method can also be used to search a preset knowledge base based on structured perception data to obtain target knowledge text that matches the structured perception data. Employing retrieval methods such as vector-based semantic similarity retrieval or keyword-based sparse retrieval to search a preset knowledge base based on structured perception data can significantly improve the accuracy and safety of decision-making in complex traffic situations for autonomous driving systems.

[0042] Step S106: Based on the target knowledge text, perform risk prediction on the structured perception data to obtain the risk prediction result of the vehicle's environment.

[0043] The aforementioned risk prediction results refer to the assessment and prediction of potential risks in the current vehicle environment, derived from in-depth analysis of structured perception data using target knowledge text. Risk prediction results may include, but are not limited to, collision risk, path intrusion risk, dynamic obstacle risk, environmental risk prediction, regulatory risk prediction, and technical failure risk. Specific risk prediction results need to be determined based on the actual prediction situation. Risk prediction results can provide a forward-looking perspective, enabling vehicles to take preventative measures before risk events actually occur, greatly improving driving safety. Route selection can also be adjusted based on risk prediction results to avoid high-risk areas, ensuring a journey that is both efficient and safe.

[0044] In one optional embodiment, the risk prediction method for risk prediction based on target knowledge text in structured perception data may include, but is not limited to, the following methods:

[0045] The first approach is rule-based risk prediction, which uses regulations and driving behavior norms in a pre-set knowledge base to predict potential regulatory violations or driving risks based on common sense through logical judgment and rule matching.

[0046] The second approach is prediction based on statistical models, which involves using statistical analysis methods, such as machine learning models, to train on a large amount of structured perception data and historical risk events to predict dynamic risks around vehicles.

[0047] The third approach is scene understanding and prediction based on deep learning, which uses deep neural networks to perform in-depth analysis of driving scenarios, extract high-level abstract features, and then predict the risks involved.

[0048] The fourth approach is knowledge reasoning and prediction based on knowledge graphs. This involves using entity relationships and logical rules in knowledge graphs to analyze structured perception data and infer potential risk points.

[0049] The risk prediction methods described above are for illustrative purposes only. The specific risk prediction methods should be determined based on actual needs, and no restrictions are imposed here.

[0050] In one optional embodiment, target knowledge entries retrieved from a pre-defined knowledge base and matching the current driving scenario are used to perform deep analysis on the structured perception data collected in real time to predict the risks present in the vehicle's environment. This process, by integrating driving behavior norms, regulatory requirements, and past accident cases contained in knowledge texts, identifies potential threats in complex scenarios, thereby generating predictions of the risk status of the vehicle's surrounding environment and guiding the autonomous driving system to make safe decisions and plans.

[0051] Step S108: The risk prediction results and structured perception data are fused to obtain target perception data, wherein the target perception data includes at least the target objects in the scene that have risks and the risk information of the target objects.

[0052] The aforementioned target perception data refers to enhanced perception data obtained by combining structured perception data with risk prediction results. Target perception data may include, but is not limited to, a list of risk perception objects, an environmental risk map, and a situational risk analysis report. The risk perception object list includes all identified objects that pose a risk, along with their respective risk levels and prediction results. The environmental risk map overlays risk prediction information onto a map, forming a map representation that integrates real-time perception and potential risk prediction. The situational risk analysis report summarizes the risk prediction results for the current driving scenario, including detailed descriptions and analyses of each risk object, as well as a risk assessment of the overall driving environment. Specific target perception data needs to be determined based on actual needs. Target perception data can be used to provide more comprehensive environmental perception information, improve route planning and driving behavior, and avoid potential hazards.

[0053] The aforementioned target objects can refer to traffic participants or other entities within the vehicle's perception range that are identified as potentially causing or affecting driving safety. The types of target objects may include, but are not limited to, traffic participants such as vehicles, pedestrians, and bicycles, as well as non-participant objects such as traffic signs, road obstacles, and weather conditions. Specific target objects need to be determined based on the actual detection situation. Identifying target objects can help the system determine the source and location of risks, facilitating the implementation of targeted safety measures.

[0054] The aforementioned risk information can refer to a detailed description of risk predictions related to the target object. Risk information may include, but is not limited to, the nature of the risk, its severity (risk level), the probability of occurrence, the scope of its impact, and recommended actions or safety strategies. Specific risk information needs to be determined based on the actual situation. Risk information can be used to transform vague risk perceptions into concrete risk levels, providing a quantitative basis for decision-making. Simultaneously, based on risk information, vehicle behavior can be adjusted in a timely manner, such as slowing down, increasing safe distance, and changing lanes, to reduce risk.

[0055] In one optional embodiment, the fusion method for fusing risk prediction results and structured perception data may include, but is not limited to, the following methods:

[0056] The first method, spatial fusion, involves matching the risk location information in the risk prediction results with the spatial coordinate data in the structured perception data to determine the specific location and scope of the risk, thereby calculating the size, shape, and location of the risk area in order to avoid it spatially.

[0057] The second approach, time fusion, takes into account the time dimension of risk prediction results and aligns them with the timestamps of structured sensing data to assess how risk changes over time. Time windows can be used to link sensing data from different points in time with the predicted risk development, forming a time series to predict future risk states.

[0058] The third approach is dynamic weighted fusion, which assigns different weights to perceived data and risk prediction results during the fusion process. The weight values ​​are dynamically adjusted based on the real-time driving environment and risk assessment.

[0059] The fourth approach is hierarchical fusion, which involves fusion at multiple levels. First, fusion is performed at a local or micro-level (e.g., at the level of an individual traffic participant), and then secondary fusion is performed at a more macro-level (e.g., at the level of the entire traffic scenario). Techniques such as recurrent neural networks or attention mechanisms can be used to process individual-level risk and perception information first, and then integrate the overall scenario's risk prediction through a higher-level model.

[0060] The above integration methods are just examples. The specific integration method needs to be determined according to actual needs, and no limit is set here.

[0061] In one optional embodiment, risk prediction results and structured perception data are fused to generate target perception data that includes at least the target objects in the scene that pose a risk and the risk information of those target objects. This fusion process, by integrating cognitive-level risk assessment with raw perception information, ensures that the vehicle's autonomous driving system can not only identify physical objects in the environment but also understand the nature and level of risk that these objects may pose. The target perception data contains updated information on target objects in the scene that are considered to pose a risk, as well as risk descriptions and quantitative indicators associated with each object, thereby providing a comprehensive environmental model that reflects both the positional state of objects and includes risk assessment for subsequent decision-making and planning.

[0062] In this application embodiment, a method for processing vehicle perception data is proposed. First, structured perception data of the vehicle is acquired. Then, a preset knowledge base is retrieved based on the structured perception data to obtain target knowledge text that matches the structured perception data. Next, risk prediction is performed on the structured perception data based on the target knowledge text to obtain the risk prediction result of the vehicle's environment. Finally, the risk prediction result and the structured perception data are fused to obtain the target perception data.

[0063] This application first collects structured perception data from the vehicle, which is derived from the fusion of raw data perceived by multiple sensors, reflecting the state of objects and environmental conditions in the current driving environment. Next, the structured perception data is used as a query to retrieve target knowledge text highly relevant to the scene from a pre-defined knowledge base. Then, risk detection is performed on the structured perception data based on the target knowledge text, generating risk prediction results. Finally, the risk prediction results are fused with the structured perception data to generate target perception data. Target perception data is an enhanced perception data that integrates cognitive-level risk information, enabling the vehicle to identify and predict the behavior of target objects in the scene, and the risk level and nature of their behavior. This application adopts a scene risk cognition fusion approach based on risk prediction and retrieval enhancement. By combining multi-sensor fused structured data with knowledge text from a pre-defined knowledge base, it generates deep risk prediction results, achieving the goal of improving the accuracy and decision-making quality of vehicle perception data. This results in a more powerful scene understanding and risk prediction ability for autonomous driving systems in complex traffic scenarios, thus solving the technical problem of low accuracy of vehicle perception data in related technologies.

[0064] Optionally, the system retrieves target knowledge text matching the structured perception data by searching a preset knowledge base based on the structured perception data, including one of the following: extracting at least one keyword from the structured perception data and retrieving the preset knowledge base based on the at least one keyword to obtain target knowledge text matching the at least one keyword; encoding the structured perception data to obtain a query vector and retrieving the preset knowledge base based on the query vector to obtain target knowledge text with a similarity greater than a preset similarity to the query vector; retrieving the preset knowledge base based on at least one keyword and a query vector to obtain target knowledge text matching the at least one keyword and a query vector; or constructing a knowledge graph based on the preset knowledge base and traversing the knowledge graph based on the structured perception data to obtain the target knowledge text.

[0065] The system retrieves a first knowledge text that matches the structured perception data by searching a pre-defined knowledge base based on the structured perception data. Then, it performs a web search based on the structured perception data to obtain a second knowledge text. Finally, it combines the first and second knowledge texts to obtain the target knowledge text.

[0066] At least one of the aforementioned keywords can refer to words extracted from structured perception data that represent the core information or key features of the data. The types of keywords can include, but are not limited to, scenario description keywords, traffic participant keywords, and behavioral keywords; the specific keyword type needs to be determined based on actual needs. Keywords can be used to extract the main information from structured perception data, facilitating rapid retrieval and understanding. Through keywords, entries related to driving scenarios in a pre-defined knowledge base can be quickly located.

[0067] The query vector mentioned above can refer to a high-dimensional vector transformed from structured data through a text embedding model. The query vector carries the semantic features of the data and can be used to compare its similarity with other vectors in the knowledge base within the vector space.

[0068] The aforementioned similarity refers to the degree of proximity between two vectors in a multidimensional space, used to measure the semantic similarity of texts. Similarity types may include, but are not limited to, cosine similarity and Euclidean distance; the specific similarity must be determined based on actual needs. In the knowledge retrieval of this application, similarity is a key indicator for determining the relevance between the query text and knowledge base entries.

[0069] The aforementioned preset similarity can refer to a pre-defined threshold used to define which knowledge items are sufficiently relevant to be selected as target knowledge text during the retrieval process. Only items that meet or exceed the preset similarity will be considered, ensuring the quality of the retrieval results. The level of the preset similarity can be used to determine the breadth of knowledge retrieval; a lower preset similarity will recall more knowledge items.

[0070] The aforementioned knowledge graph can refer to a structured knowledge representation model. A knowledge graph stores knowledge in the form of entities and relationships between them, forming a graph structure where entities are nodes and relationships are edges. Through the entity relationships in the knowledge graph, logical reasoning and knowledge discovery can be performed to identify hidden or indirect risk connections. Knowledge graphs are used for information integration, connecting scattered related knowledge items to form a more complete and coherent knowledge system.

[0071] The aforementioned first knowledge text may refer to the knowledge entry that most closely matches the structured perception data, obtained through retrieval from a local preset knowledge base.

[0072] The aforementioned second knowledge text can refer to knowledge text derived from web searches. This second knowledge text provides additional information related to the structured perception data, which is not present in the local pre-set knowledge base. The second knowledge text can be used to add real-time or region-specific information, such as temporary traffic control or weather warnings, allowing the system to access new or highly relevant information when encountering uncommon or unconventional scenarios.

[0073] In one optional embodiment, a preset knowledge base is retrieved based on structured perception data to obtain target knowledge text that matches the structured perception data. The specific technical steps are as follows:

[0074] First, at least one keyword is extracted from the structured perception data, such as "right turn" and "inner wheel difference," and a keyword search is performed on a pre-defined knowledge base to obtain target knowledge text that matches the keyword. This method utilizes the explicit index of keywords in the knowledge base to quickly locate the legal or behavioral common sense fragments most relevant to the real-time scenario, greatly improving the immediacy and accuracy of decision-making.

[0075] Secondly, the structured perception data is encoded to obtain query vectors, and a pre-defined knowledge base is retrieved based on these query vectors to find target knowledge texts with a similarity greater than a preset value. Using a vector similarity retrieval strategy, even when faced with complex scene details, the system can accurately match rules in the knowledge base that conform to the current context's logic through high-dimensional semantic comparison, enhancing the depth and breadth of scene understanding.

[0076] Third, the preset knowledge base is searched using keywords and query vectors to obtain target knowledge text that matches at least one keyword and query vector. This dual retrieval mechanism combining keywords and query vectors ensures that even when the scene description contains ambiguous or polysemous information, highly relevant and comprehensive target knowledge text can be obtained through multi-angle cross-validation. This method not only improves the robustness of the retrieval but also guarantees the consistency and reliability of the matching results.

[0077] Fourth, a knowledge graph is constructed based on a pre-set knowledge base, and the knowledge graph is traversed based on structured perception data to obtain the target knowledge text. Through knowledge graph traversal, the system can understand the intricate relationship between laws and regulations and common sense, identify a series of related rule chains, and understand the scenario not only as a single rule, but also as a more comprehensive rule network perspective, which helps to make more reasonable risk assessments.

[0078] Fifth, the system uses structured perception data to retrieve information from a pre-defined knowledge base, obtaining the first knowledge text. Then, it performs a web search based on the structured perception data to obtain the second knowledge text. Finally, it combines the real-time information obtained from the web search (the second knowledge text) with the static knowledge in the pre-defined knowledge base (the first knowledge text) for summary analysis to generate the target knowledge text. This step is particularly suitable for handling abnormal situations caused by factors such as weather changes, road construction, and emergencies. By introducing real-time network data, the system can capture these changes immediately, adjust decision-making strategies in a timely manner, and greatly enhance the ability and safety of autonomous vehicles to respond to emergencies.

[0079] Optionally, the method further includes: acquiring legal information related to vehicle driving and historical driving behavior data; performing natural language processing on the legal information to construct a first knowledge base, wherein the knowledge in the first knowledge base includes driving behavior norms and the scenarios corresponding to the driving behavior norms; performing in-depth mining on the historical driving behavior data to construct a second knowledge base, wherein the knowledge text in the second knowledge base includes objects in the environment and operating modes when the objects are at risk; obtaining a preset knowledge base based on the first and second knowledge bases; preferably, performing natural language processing on the legal information to construct the first knowledge base includes: decomposing the legal information to obtain multiple rule atoms, wherein different rule atoms correspond to different driving behavior norms; annotating the multiple rule atoms with scenarios to obtain the scenarios corresponding to the multiple rule atoms; constructing a knowledge graph based on the dependencies between the multiple rule atoms and the scenarios corresponding to the multiple rule atoms to obtain the first knowledge base.

[0080] The aforementioned legal information may refer to official documents and regulations related to road traffic laws, local traffic rules, and industry standards and policies related to intelligent connected vehicles.

[0081] The aforementioned historical driving behavior data can refer to data collected from past driving experiences, including actual operational records of human drivers and autonomous driving systems in different road environments. Historical driving behavior data can be used to reflect real-world driving behavior patterns and potential risks.

[0082] The aforementioned first knowledge base can refer to a set of road traffic rules refined and structured using natural language processing technology, along with descriptions of the applicability of these rules in different scenarios. This first knowledge base can be used to provide clear rule guidance and scenario-based regulatory understanding for autonomous driving systems.

[0083] The aforementioned second knowledge base can refer to a set of driving behavior patterns and risk contingency plans obtained through in-depth mining of historical driving behavior data. The second knowledge base primarily focuses on the driver's tacit knowledge, particularly their experience and skills in handling complex and dangerous scenarios. It can be used to predict and understand the behavior of other road users and the potential risks these behaviors pose to autonomous vehicles.

[0084] The construction process of the aforementioned second knowledge base includes: first, named entity recognition, that is, identifying key traffic participants and road features from historical data; then, relation extraction, extracting the relationships between these entities, such as the interaction patterns between vehicles and pedestrians; and finally, scenario-plan association, that is, defining driving behavior plans for specific scenarios (such as ghost peeks or sudden stops), forming a "scenario ontology-risk plan" structure.

[0085] The aforementioned multiple rule atoms can refer to the most basic, indivisible rule units after being broken down from a legal provision. Each rule atom describes a specific driving behavior norm. Multiple rule atoms may include, but are not limited to, safe distance rule atoms, speed limit rule atoms, priority rule atoms, etc. The specific rule atoms need to be determined according to the specific legal provisions.

[0086] Multiple rule atoms can be used to break down complex legal provisions into smaller rules that are easier to understand and enforce.

[0087] The aforementioned scene annotation refers to adding applicable specific scene or context information to each rule based on the rule atom. Scene annotation can help autonomous driving systems correctly apply the corresponding rule atom when encountering similar scenes.

[0088] The scenarios corresponding to the aforementioned multiple rule atoms can refer to specific driving scenarios or situations associated with each rule atom. The scenarios corresponding to multiple rule atoms constitute the node information in the regulatory knowledge graph, providing background for the application of rule atoms.

[0089] In one optional embodiment, legal information related to vehicle operation and historical driving behavior data are first acquired. These two types of information form the basis for the autonomous vehicle's deep understanding of traffic rules and social driving habits. Next, natural language processing is performed on the legal information to construct a first knowledge base. This knowledge base not only includes driving behavior norms but also associates them with applicable specific scenarios, thereby providing the vehicle with contextualized regulatory guidance.

[0090] Secondly, we conduct in-depth mining of historical driving behavior data to build a second knowledge base. This knowledge base focuses on risk scenarios and coping strategies encountered in actual driving, including objects in the environment and human operating patterns when objects are at risk, providing vehicles with driving intelligence based on big data analysis.

[0091] Finally, the first and second knowledge bases are combined to obtain a pre-defined knowledge base, which becomes the core resource for deep scenario understanding and proactive risk assessment. The construction of this pre-defined knowledge base achieves comprehensive coverage from legal provisions to practical driving experience, ensuring that the vehicle can make reasonable judgments based on sufficient information in various scenarios. The vehicle can understand the rule context of complex scenarios and also anticipate potential unknown risks. This application of integrated knowledge significantly improves the vehicle's decision-making level, enabling it to exhibit higher safety and robustness when facing long-tail scenarios.

[0092] In one optional embodiment, firstly, the legal information is meticulously broken down into multiple rule atoms, each of which is an independent expression of driving behavior norms. Next, these rule atoms are labeled with scenarios to obtain the scenarios corresponding to each rule atom, thus clarifying the environment in which they take effect. Then, based on the logical dependencies between rule atoms, the multiple rule atoms, and the scenarios corresponding to each rule atom, a knowledge graph is constructed to obtain a first knowledge base. This first knowledge base form of the graph enables the model to understand the interactions and hierarchical structures between regulations, greatly improving the accuracy and flexibility of rule interpretation and providing a solid legal basis for vehicle decision-making in complex scenarios.

[0093] Optionally, risk prediction is performed on the structured perception data based on the target knowledge text to obtain the risk prediction result of the environment, including: filling the target knowledge text and structured perception data into the prompt word template to generate target prompt words; calling the risk prediction model and inputting the target prompt words into the risk prediction model, using the risk prediction model to perform risk prediction, and obtaining the risk prediction result.

[0094] The aforementioned prompt word template can refer to a structured framework or guidance text. Prompt word templates can be used to guide and construct prompts input to the model. Prompt word templates can include, but are not limited to, scenario description templates, rule reference templates, risk prediction instruction templates, knowledge query templates, etc., and the specific prompt word template needs to be determined based on actual needs. Prompt word templates can provide a systematic way to organize and format input data, ensuring that the model can understand the structure and content of the input information.

[0095] The aforementioned target cue words can refer to customized input text generated from cue word templates and specific scenario data, used to directly input into risk prediction models or Large Language Models (LLMs). Target cue words can be used to present detailed information about driving scenarios and related driving behavior norms, risk cases, and other knowledge to the model in an easily understandable natural language form, helping it to perform risk analysis and prediction.

[0096] The aforementioned risk prediction model can refer to an artificial intelligence model used to assess risks in driving scenarios. Types of risk prediction models may include, but are not limited to, rule-based prediction models, data-driven learning models, and inference models based on large language models (LLMs). The specific risk prediction model needs to be determined based on actual needs. Risk prediction models can be used to quantify uncertainties and potential hazards in driving scenarios, providing a numerical indicator or level classification for risk assessment, and can be used for behavioral prediction.

[0097] In one optional embodiment, the target knowledge text and real-time perceived structured data are first injected into a preset prompt word template to automatically generate target prompt words. These target prompt words are then passed as input to a risk prediction model, triggering the model's cognitive reasoning process and yielding a risk prediction result. This mechanism ensures that the system can perform compliance analysis and predictive risk assessment of the current scenario based on in-depth information from a driving safety knowledge base, thereby providing decision support beyond mere physical perception when autonomous vehicles face complex traffic conditions.

[0098] Optionally, the target knowledge text and structured perception data are filled into the prompt word template to generate target prompt words, including: obtaining historical perception data and the historical risk prediction results corresponding to the historical perception data; filling the target knowledge text, structured perception data, historical perception data and historical risk prediction results into the prompt word template to generate target prompt words; preferably, the risk prediction model is a large model deployed locally, or a model composed of multiple models, or a question-and-answer model.

[0099] The aforementioned historical perception data refers to the information about the vehicle's surrounding environment collected by its sensors (such as cameras, radar, and lidar) at a specific point in the past. Historical perception data may include, but is not limited to, records of road conditions, the location and status of traffic participants, traffic signs, and traffic lights; the specific historical perception data needs to be determined based on actual needs. Historical perception data can be used for trend analysis and can also help models understand long-term, dynamic driving scenarios, such as changes in traffic flow and the impact of weather conditions.

[0100] The aforementioned question-answering model refers to a natural language processing model specifically designed to answer questions. This model can understand and parse the semantics of questions, search for answers from existing textual data, or generate answers based on its own knowledge. The question-answering model can understand complex driving scenario descriptions based on input prompts, including current perception data, historical data, and target knowledge text.

[0101] In one optional embodiment, historical perception data and its corresponding historical risk prediction results are first acquired. Then, the target knowledge text, structured perception data, historical perception data, and historical risk prediction results are filled into the prompt word template to generate the target prompt word. This measure essentially introduces a memory mechanism that allows the system to better understand the continuity and dynamic changes of the current driving scenario based on past perception data and risk prediction results, thereby improving the accuracy and comprehensiveness of cognitive reasoning.

[0102] In one optional embodiment, the risk prediction model can select a large locally deployed model or a set of multiple models, as well as a specially designed question-answering model, as the risk prediction model. This further enhances the system's autonomy and response speed. Local deployment avoids frequent communication with cloud servers, eliminating the uncertainty caused by network latency, and also ensures the privacy and security of driving data. The hybrid architecture of multiple models allows for flexible allocation of computing resources and inference strategies according to different types of driving scenarios, thereby improving the accuracy of risk prediction in specific scenarios. The question-answering model focuses on answering inquiries specific to driving scenarios, providing risk assessments and response suggestions quickly and accurately through concise dialogue, which greatly improves the efficiency and relevance of decision-making.

[0103] Optionally, the risk prediction results and structured perception data are fused to obtain target perception data, including: associating the risk prediction results with at least one object in the structured perception data to obtain the target object; quantifying the risk level in the risk prediction results to obtain the risk score corresponding to the target object; and spatializing the risk score in a preset space of the vehicle to obtain the risk information of the target object.

[0104] The aforementioned at least one object can refer to a specific traffic participant or object present in the current driving environment, as perceived by the vehicle through multimodal sensors. At least one object may include, but is not limited to, vehicles, pedestrians, bicycles, buses, obstacles, traffic signs, etc., and the specific at least one object needs to be determined based on the actual structured perception data. By detecting and determining at least one object, the vehicle can understand the surrounding traffic conditions in real time, thereby improving driving safety.

[0105] The risk score mentioned above can be a quantitative indicator of the probability and severity of a risk posed by a certain object or situation in the predicted driving environment. The risk score can be a numerical value within a specific range, such as [0, 1], where 1 represents a higher risk. The risk score helps the system understand and handle the severity of the current risk. The values ​​above are for illustrative purposes only; specific values ​​need to be determined based on actual needs.

[0106] The aforementioned spatialization refers to the process of transforming abstract risk scoring information into physical spatial coordinates or regions of the environment in which the autonomous vehicle operates. In this process, risk is no longer merely a numerical assessment, but is mapped to the actual driving space. Spatialization can clarify the impact of current risks in the real space, facilitating intuitive understanding.

[0107] In one optional embodiment, a one-to-one correspondence is first established between the risk prediction results and at least one object in the structured perception data to obtain the target object, ensuring that each piece of risk information accurately points to a specific traffic participant or environmental element in the scene. Subsequently, the risk level in the risk prediction results is quantified to obtain a risk score for the target object, transforming the subjective description into an objective risk score. This score encompasses the severity and immediacy of the risk, providing a quantitative reference for subsequent decision-making. Finally, the obtained risk score is spatialized within a pre-defined space in the vehicle to generate risk information matching the location of the risk object. This series of operations not only enriches the target perception data but also constructs an intuitive driving environment model that includes risk scenarios, thereby improving the autonomous driving system's ability to handle long-tail scenarios and enhancing the system's safety and robustness.

[0108] Optionally, the risk score is spatialized in a preset space of the vehicle to obtain the risk information of the target object, including: generating a risk potential field in the preset space based on the risk score and the category of the target object, wherein the risk potential field is centered on the target object; adding semantic tags to the target object, wherein the semantic tags are used to characterize the risk attributes of the target object; and obtaining the risk information based on the risk potential field and the semantic tags.

[0109] The category of the target object can be a type identifier. Categories may include, but are not limited to, motor vehicles, non-motor vehicles (bicycles), pedestrians, animals, traffic signs, road markings, etc., with the specific category to be determined based on the actual target object. Determining the category of the target object helps to target relevant knowledge bases and models for more accurate risk assessment, and also facilitates downstream decision-making and planning, enabling the adoption of appropriate behavioral strategies based on different traffic participants.

[0110] The aforementioned risk potential field refers to a directional and intense virtual field created around a target object within a pre-defined space. This risk potential field reflects the degree and scope of risk that the target object may pose within a specific timeframe. The strength and distribution of the risk potential field can guide vehicles in planning their driving paths to avoid high-risk areas. Furthermore, the risk potential field can be dynamically updated over time, reflecting changes in the target object's behavior and fluctuations in risk levels, making response strategies more real-time and flexible.

[0111] The aforementioned semantic labels refer to a set of descriptive information attached to a target object to characterize its potential risk attributes, behavioral intentions, and state. Semantic labels may include, but are not limited to, risk attribute labels, behavioral intention labels, and state description labels; the specific semantic labels are driven by actual needs. Semantic labels can be used to enhance the representation of the target object, enabling autonomous driving systems to make decisions based on richer information, such as identifying pedestrians' intention to cross the road or potential violations by motor vehicles.

[0112] In one optional embodiment, a risk potential field is first dynamically generated in a preset space based on the risk score and the category of the target object. This potential field, centered on the target object, intuitively reflects the degree and distribution of potential risks around the target. Simultaneously, semantic tags are added to the target object. These tags not only identify the object's basic attributes but also incorporate risk attributes, such as "high risk—inner wheel difference collision" or "medium risk—ghost pedestrian path intrusion." Finally, based on a comprehensive consideration of the risk potential field and semantic tags, detailed, spatially located risk information is generated. This information not only reveals the nature of the risk but also clarifies its specific location and the entities involved. This helps the autonomous driving system make more informed decisions, such as adjusting the driving route to avoid risk areas or taking predictive deceleration measures against specific objects. This improves the autonomous vehicle's ability to cope with complex traffic scenarios, enhancing driving safety and robustness.

[0113] Optionally, the method further includes: determining the risk priority of multiple sensing objects based on risk indicators between multiple sensing objects and the vehicle in the structured sensing data; extracting information from the structured sensing data based on the risk priority to obtain the vehicle status, environmental information, and at least one target object; generating sensing text based on the vehicle status, environmental information, and at least one target object; and retrieving a preset knowledge base based on the structured sensing data to obtain target knowledge text matching the structured sensing data, including: retrieving the preset knowledge base based on the sensing text to obtain target knowledge text matching the sensing text; and performing risk assessment on the structured sensing data based on the target knowledge text. The method for predicting environmental risks includes: generating risk prediction text based on perceived text and target knowledge text; performing risk prediction on the risk prediction text to obtain a risk prediction result; preferably, after fusing the risk prediction result and structured perceived data to obtain target perceived data, the method further includes: adjusting the vehicle's driving behavior or driving trajectory based on the target perceived data; preferably, after performing risk prediction on structured perceived data based on target knowledge text to obtain an environmental risk prediction result, the method further includes: adjusting the vehicle's driving behavior or driving trajectory based on the risk prediction result; and / or, generating prompt information based on the risk prediction result.

[0114] The aforementioned perceived objects can refer to any entity in the vehicle's surrounding environment that is identified and tracked by the perception system. Perceived objects can include, but are not limited to, dynamic perceived objects (such as pedestrians, vehicles, and animals) and static perceived objects (such as road conditions, traffic infrastructure, and environmental elements). The specific perceived objects need to be determined based on the actual driving environment. Perceived objects serve as the foundation for the autonomous driving system to acquire environmental information and are also a direct data source for subsequent advanced functions such as risk assessment, path planning, and decision-making.

[0115] The aforementioned risk metrics can refer to quantitative standards for assessing the degree and urgency of potential risks posed by perceived objects to autonomous vehicles. Risk metrics may include, but are not limited to, relative speed, Time-to-Collision (TTC) prediction, minimum safe distance, field-of-view occlusion, and behavioral anomaly index. Relative speed represents the speed difference between the perceived object and the vehicle; TTC predicts the time required for a collision based on the current speed and distance; minimum safe distance indicates the minimum safe contact distance between the perceived object and the vehicle based on its type and behavior; field-of-view occlusion indicates whether the perceived object obstructs the vehicle's frontal or lateral view, and the proportion of obstructed area; and the behavioral anomaly index represents the probability of uncertainty or deviation from the norm in the perceived object's behavior. Risk metrics can be used to quickly identify which objects in a scene require priority attention, helping the system prioritize high-risk events to improve overall safety and response efficiency.

[0116] The aforementioned risk prioritization refers to quantifying each perceived object using risk indicators and then ranking them according to the magnitude and urgency of the risk they pose to the vehicle. Based on the calculation results of the risk indicators, risk priorities can be divided into four levels: low, medium, high, and very high. Different levels trigger different system response strategies. The above risk prioritization is only an example; the specific risk priority needs to be determined based on the actual situation. Risk prioritization enables the vehicle's autonomous driving system to selectively focus on high-risk objects, improve resource allocation, and ensure timely handling of the most important safety threats.

[0117] The vehicle status mentioned above refers to the vehicle's own operating condition. Vehicle status may include, but is not limited to, speed, acceleration, steering angle, braking status, and battery level, etc., and the specific vehicle status needs to be determined based on the actual situation. Vehicle status can serve as one of the bases for assessing the overall driving safety of the vehicle and formulating risk avoidance strategies.

[0118] The aforementioned perceptual text can refer to a form of converting structured perceptual data into natural language descriptions. Perceptual text can serve as a bridge linking physical perception and cognitive reasoning, and it can be used to ensure that autonomous driving systems can perform high-level contextual understanding and decision-making based on natural language processing technology.

[0119] The aforementioned risk prediction text can refer to text that describes scenario risks, compliance, and future trends based on perceived text and target knowledge text, and after reasoning through a large language model. Risk prediction text can provide a structured risk assessment perspective, guiding autonomous driving systems on how to interpret current traffic scenarios and predict potential safety threats.

[0120] The aforementioned prompts can refer to suggestions or warnings provided to the driver or autonomous driving system regarding how to operate the vehicle safely, based on risk prediction results. These prompts may include, but are not limited to, safe driving advice, risk warnings, regulatory reminders, and emergency avoidance instructions; the specific prompts will be determined based on driver needs and risk prediction results. These prompts can be used to enhance human-machine interaction, raise the driver's awareness of potential risks, or guide the autonomous driving system to adjust its driving strategy to avoid risks.

[0121] In one optional embodiment, the system first determines the risk priority of multiple sensed objects based on risk indicators (such as TTC) between multiple sensed objects and the vehicle in the structured sensing data. This step ensures priority attention to high-risk entities, improving the efficiency and relevance of real-time scene analysis. Subsequently, information is extracted from the structured sensing data based on the risk priority to obtain the vehicle's status, environmental information, and at least one target object. This process effectively filters out the core information needed for decision-making, avoiding interference from irrelevant data, thereby improving the accuracy of subsequent processing. Finally, based on the extracted vehicle status, environmental information, and target object, sensing text is generated. By converting structured data into natural language descriptions, LLM can better understand and process scene information, enhancing the effectiveness of cognitive reasoning.

[0122] In one optional embodiment, a preset knowledge base is retrieved based on the perceived text to obtain target knowledge text that matches the perceived text. This retrieval mechanism ensures the semantic relevance of the matched knowledge, providing accurate support for subsequent in-depth analysis.

[0123] In one optional embodiment, risk prediction text is generated by combining perceived text and target knowledge text; and after performing risk prediction on the risk prediction text, a risk prediction result is obtained. This step, by introducing prior knowledge, significantly improves the accuracy and comprehensiveness of risk assessment, enabling the autonomous driving system to make more reasonable decisions based on social rules and common sense about driving.

[0124] In one optional embodiment, after fusing risk prediction results and structured perception data to obtain target perception data, the vehicle's driving behavior or trajectory is adjusted based on this data. This adjustment strategy ensures that the vehicle can take appropriate avoidance measures in a timely manner after recognizing a risk, such as changing lanes, slowing down, or stopping, thereby significantly improving driving safety. Similarly, the prompts generated based on risk prediction results can inform the driver or passengers of potential dangers, promoting safe travel through human-machine collaboration. Furthermore, the system can also directly convert target knowledge text into direct constraints on the vehicle's trajectory, or improve risk detection capabilities by adjusting the resource allocation of the perception system. All of these solutions improve the robustness and safety of the system to varying degrees, ensuring that autonomous vehicles can cope with complex traffic situations and reducing the probability of accidents.

[0125] In one alternative embodiment, Figure 2 This is a schematic diagram of a scenario risk cognition fusion system based on a large language model and retrieval enhancement generation, according to an embodiment of this application. Figure 2 As shown, the scenario risk cognition fusion system mainly includes a scenario semantic conversion module, a retrieval augmented generation (RAG) module, an LLM cognitive reasoning module, and a cognitive result fusion and application module.

[0126] When the upstream multi-sensor fusion system sends structured perception data (BEV) to the scene semantic conversion module in the scene risk cognition fusion system, the scene semantic conversion module converts the structured perception data (BEV) into a natural language scene description, which is then sent to the RAG module. The RAG module processes the natural language scene description through an external driving safety knowledge base to obtain enhanced contextual information, which is then sent to the LLM cognitive reasoning module. The LLM cognitive reasoning module interacts with an external large-scale language model to process the enhanced contextual information and obtain a structured cognitive result, which is then sent to the cognitive result fusion and application module. Finally, the cognitive result fusion and application module processes the result to obtain an enhanced BEV or quantified risk, which is then sent to the downstream planning and decision-making module.

[0127] The scenario risk cognition fusion system, through natural language description and knowledge retrieval, can deeply understand complex traffic scenarios, especially those requiring social experience and rule awareness. The inclusion of an LLM reasoning module enables decision-making not only based on real-time perception but also considering prior knowledge and potential rule conflicts, resulting in more informed and safer driving decisions. Furthermore, the cognitive result fusion and application module spatializes risk information and provides it to the planning and decision-making module in an enhanced BEV format, making risk assessment and avoidance a visualized and intuitive process. Combining cognitive reasoning and spatialized risk information, the system can effectively predict and respond to long-tail scenarios and other potential risks, significantly improving the safety and adaptability of autonomous vehicles in real-world driving. Through these designs, the scenario risk cognition fusion system of this application can effectively enhance the understanding and response capabilities of autonomous vehicles to complex traffic scenarios.

[0128] Figure 3 This is a flowchart of a scenario risk perception fusion method based on a large language model and retrieval enhancement generation according to an embodiment of this application, such as... Figure 3 As shown, after the method begins, it first acquires structured perceived data (BEV); then performs scene semantic transformation to generate natural language descriptions; next, it uses RAG to retrieve relevant laws and common sense from the knowledge base; then it constructs a prompt and calls LLM for cognitive reasoning; it parses the structured cognitive results returned by LLM; then it spatializes the structured cognitive results to generate a semantic risk field and dynamic labels; finally, it sends the output values ​​of the enhanced environment model downstream to the planning and decision-making module, and the method ends.

[0129] Specifically, acquiring structured perception data: The system's primary task begins with upstream multi-sensor fusion devices collecting and processing structured information about the vehicle's surrounding environment in real time, forming structured perception data in the form of a bird's-eye view (BEV). This data precisely depicts the physical states of traffic participants (such as pedestrians and vehicles), including their positions, speeds, and accelerations, as well as environmental elements such as weather conditions, road attributes, and traffic signs.

[0130] The scene semantic transformation module receives structured perception data from upstream sources and uses pre-defined sentence templates and rules to convert BEV information in plain data format into human-readable natural language descriptions. This process involves not only syntactic restructuring of the data but also prioritizing traffic participants to ensure that the description highlights key points and conveys information without loss. The generated natural language description provides easily understood and processed input for subsequent high-level analysis.

[0131] Utilizing RAG to retrieve relevant knowledge: Upon receiving a natural language description, the RAG module immediately initiates a process of retrieving the most relevant regulations and behavioral common sense from the driving safety knowledge base. Using a locally deployed text embedding model, the RAG module transforms the descriptive text into a vector representation, and then performs a vector similarity search within a pre-built knowledge base index to determine the Top-K rules and cases that best fit the current scenario. Through this knowledge retrieval, the system obtains background information and potential rules about the scenario, laying the foundation for further deep reasoning.

[0132] Constructing Prompts and Invoking LLM for Cognitive Reasoning: With enhanced contextual information, the LLM cognitive reasoning module begins its work. First, the module integrates the original natural language description with retrieved relevant knowledge to construct a multi-dimensional, structured prompt, ensuring the completeness and relevance of information delivery. Then, the module invokes the locally deployed large language model or accesses the cloud-based LLM via an Application Programming Interface (API) to perform deep cognitive reasoning. Based on parsing and understanding the prompts, the model analyzes the compliance of the scenario, predicts risk events, and outputs structured results rich in deep cognitive information. This process fully leverages the zero-shot learning capability, logical reasoning ability, and prior information from the knowledge base of the large model, improving the accuracy and breadth of scenario understanding.

[0133] Parsing Structured Cognitive Results: The structured cognitive results returned by LLM are received and parsed by the Cognitive Results Fusion and Application module. This module parses these results to ensure they conform to predefined structured data for Cognitive Risk Analysis, facilitating further processing and application.

[0134] Spatializing Cognitive Outcomes: The cognitive outcome fusion and application module matches risk assessment information from structured cognitive outcomes with specific entities in the environmental model. Through spatialized risk representation, a semantic risk field and dynamically enhanced labels are generated, concretizing abstract compliance and risk forecasts into risk heatmaps and labels on the BEV map, providing an intuitive reference for planning decisions.

[0135] Output to downstream planning and decision-making module: The enhanced BEV map or risk quantification data, which incorporates cognitive risk information, is output to the downstream planning and decision-making module. This information directly affects the vehicle's trajectory planning and driving strategy, ensuring that the decision-making process is not only based on real-time perception but also considers social rules and potential risks, thereby significantly improving the safety and robustness of autonomous vehicles in complex traffic environments.

[0136] The entire method starts from raw sensor data, goes through a series of steps such as semanticization, knowledge retrieval, deep reasoning, and risk spatialization, and finally outputs to the downstream decision-making module, forming a closed-loop processing link from perception to cognition to action, aiming to provide more comprehensive and intelligent contextual understanding and decision support for autonomous vehicles.

[0137] In an optional embodiment, the scenario risk cognition fusion method based on large language model and retrieval enhancement generation proposed in this application mainly includes the following modules: scenario semantic transformation module, retrieval enhancement generation (RAG) module, LLM cognitive reasoning module, and cognitive result fusion and application module.

[0138] The specific descriptions of the scene semantic transformation module, the retrieval enhancement generation (RAG) module, the LLM cognitive reasoning module, and the cognitive result fusion and application module are as follows:

[0139] First, for the scene semantic transformation module: the module input is a structured data object in a unified format, which describes the vehicle's own state, environmental information, and the state of surrounding traffic participants in real time. In a preferred embodiment, the data object can be in JSON format, and its specific fields may include: timestamp: timestamp, used for data alignment; ego_vehicle: vehicle state, including information such as speed, acceleration, and lane markings; map_info: high-precision map information, including the current road name, road type, speed limit, traffic signs, and point of interest (POI) information, such as "school area" and "accident-prone area"; objects: a list of objects, where each object represents a perceived traffic participant, including ID, category (pedestrian / car / truck / bicycle), 3D position, velocity vector, orientation, and preliminary state (such as "stationary", "turning", "brake light on") or relationship (such as "occluded", "located in blind spot") determined by traditional fusion algorithms.

[0140] The processing flow includes: (1) Priority sorting, which sorts all objects in the object list according to the distance between the object and the vehicle, relative speed, and potential collision time (TTC). (2) Core information extraction, which extracts the vehicle's status, key environmental information (such as "at the entrance of the primary school"), and information of the top N objects with higher priority. (3) Template filling, which designs one or more dynamic statement templates. The template contains placeholders to fill in the extracted core information. For example, a template could be: "At [location], the vehicle is traveling at [speed]. Ahead at [distance], a [category] is in [state]." (4) Natural language synthesis, which combines and adjusts the filled statement in terms of syntax and logic to form a coherent, fluent, and information-free natural language paragraph. In particular, for some key states, such as "hazard lights" and "lane occupation", emphatic words will be used to describe them.

[0141] The module outputs a text string, which is a comprehensive and accurate natural language snapshot of the current driving scenario. This output will serve as the query input for the subsequent retrieval enhancement generation module.

[0142] Second, regarding the Retrieval Enhancement Generation (RAG) module: The core function of the RAG module is to respond to real-time scene queries generated by the Scene Semanticization Transformation module, and perform high-dimensional semantic retrieval from a deeply processed and improved multi-source heterogeneous driving safety knowledge base, providing accurate and rich prior knowledge for subsequent cognitive reasoning. The input to this module is the natural language text string (String) output by the Scene Semanticization Transformation module. This string is a real-time snapshot of the current driving scene.

[0143] The knowledge base in the construction and indexing of the driving safety knowledge base can include, but is not limited to, structured regulatory bases and multi-source behavioral common sense bases. After determining the knowledge base, it is vectorized and indexed.

[0144] The internal processing flow of the driving safety knowledge base mainly includes query vectorization, vector similarity retrieval, context aggregation, and reordering.

[0145] The output of the driving safety knowledge base module is a structured data object that packages all the information required for subsequent LLM reasoning. It contains at least two key fields: original_query and retrieved_context. Original_query represents the original scenario description text, while retrieved_context represents the integrated reference context text retrieved from the knowledge base.

[0146] Third, for the LLM cognitive reasoning module: The input to the LLM cognitive reasoning module is the structured data object output by the Retrieval Enhancement Generation (RAG) module.

[0147] The LLM cognitive reasoning module employs a structured cognitive reasoning framework and dynamic prompt word generation. To ensure that the reasoning process of the Large Language Model (LLM) is controllable, reproducible, and highly structured, the core of this module is not simply prompt word concatenation, but rather the construction of a structured cognitive reasoning framework. This framework is responsible for dynamically and programmatically organizing input data into an execution object containing rich context and a rigorous logical chain, and submitting it to the LLM for parsing and reasoning. In a preferred embodiment, the prompt words generated by the structured cognitive reasoning framework, as an execution object, may include the following components: system role and output mode definition, which is responsible for top-level constraints on the behavior of the LLM, defining its professional roles and output data structures; context management and injection, which is responsible for providing the LLM with all the information needed for decision-making and introducing historical information to achieve a deeper level of situational awareness; and reasoning chain construction, which connects the above elements into an executable logical chain.

[0148] The LLM cognitive reasoning module's model invocation and reasoning includes: constructing complete prompt words, and implementing model invocation and reasoning. The output of the LLM cognitive reasoning module is a structured JSON object containing deep cognitive information. This object precisely describes the compliance and potential risks of the scenario.

[0149] Fourth, the cognitive outcome fusion and application module is the final execution end of the cognitive ability of this application. Its core task is to transform the highly structured risk judgment output by the LLM cognitive reasoning module, which contains deep semantics, into quantitative indicators and spatial representations that can be directly understood and utilized by the downstream planning and decision-making module, thereby achieving a direct and effective impact on vehicle behavior.

[0150] The input to the cognitive results fusion and application module is the JSON object output by the LLM cognitive reasoning module.

[0151] The processing flow of the cognitive outcome fusion and application module includes: risk entity location and association, risk level quantification, and risk spatial representation.

[0152] The output of the cognitive results fusion and application module, and its downstream applications, constitute a "cognitive enhancement" of the original environmental model of the autonomous driving system. It is no longer a purely geometric world, but a driving scenario filled with semantic information and risk predictions. Specific outputs may include: an enhanced BEV map: a bird's-eye view overlaid with a semantic risk field and dynamic semantic labels; and a risk object list: a sorted list containing all objects identified as having cognitive risk in the current scene, along with their detailed risk information (score, type, location, etc.).

[0153] This output directly serves the downstream planning and decision-making module, fundamentally enhancing its capabilities: For trajectory planning, the planner treats the "semantic risk field" as a repulsive field or high-cost area, proactively avoiding it during trajectory generation. For example, when facing a truck with "inner wheel difference" risk, it automatically plans a trajectory with a greater lateral distance and a more conservative speed. For behavioral decision-making, the decision logic can formulate more granular driving behaviors based on "dynamic semantic tags." For example, when a "pedestrian (high risk: probability of chasing a ball)" is detected, the system's decision will escalate from "decelerate" to "prepare to brake or even sound the horn as a warning," a decision-making pattern more aligned with that of expert human drivers.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0155] According to an embodiment of this application, a system for processing vehicle perception data is provided. It should be noted that this device can be used to execute the aforementioned method for processing vehicle perception data. Specific embodiments are the same as the method embodiments described above, and will not be repeated here.

[0156] Figure 4 This is a schematic diagram of a vehicle perception data processing system according to an embodiment of this application, such as... Figure 4 As shown, the system includes: a data acquisition module 402, a knowledge retrieval module 404, a risk prediction module 406, and a result fusion module 408.

[0157] The data acquisition module 402 is used to acquire structured perception data of the vehicle, wherein the structured perception data is obtained by fusing raw data perceived by multiple sensors on the vehicle; the knowledge retrieval module 404 is used to retrieve target knowledge text matching the structured perception data from a preset knowledge base, wherein the knowledge stored in the preset knowledge base is used to characterize driving behavior norms or operating modes in dangerous scenarios; the risk prediction module 406 is used to perform risk prediction on the structured perception data based on the target knowledge text, thereby obtaining the risk prediction result of the vehicle's environment; the result fusion module 408 is used to fuse the risk prediction result and the structured perception data to obtain target perception data, wherein the target perception data includes at least the target object in the scene that poses a risk and the risk information of the target object.

[0158] Optionally, the knowledge retrieval module is used to extract at least one keyword from the structured perception data, and retrieve a preset knowledge base based on the at least one keyword to obtain target knowledge text that matches the at least one keyword; it is used to encode the structured perception data to obtain a query vector, and retrieve a preset knowledge base based on the query vector to obtain target knowledge text that has a similarity greater than a preset similarity with the query vector; it is used to retrieve a preset knowledge base based on at least one keyword and a query vector to obtain target knowledge text that matches the at least one keyword and the query vector; and it is used to construct a knowledge graph based on the preset knowledge base, and traverse the knowledge graph based on the structured perception data to obtain target knowledge text.

[0159] This tool is used to retrieve a preset knowledge base based on structured perception data, obtain a first knowledge text that matches the structured perception data, perform a web search based on the structured perception data to obtain a second knowledge text, and summarize the first and second knowledge texts to obtain the target knowledge text.

[0160] Optionally, the system is also used to: construct a first knowledge base by performing natural language processing on the legal information based on relevant legal information and historical driving behavior data, wherein the knowledge in the first knowledge base includes driving behavior norms and the scenarios corresponding to the driving behavior norms; construct a second knowledge base by performing in-depth mining on the historical driving behavior data, wherein the knowledge text in the second knowledge base includes objects in the environment and operating modes when the objects are at risk; and obtain a preset knowledge base based on the first and second knowledge bases; preferably, constructing the first knowledge base by performing natural language processing on the legal information includes: decomposing the legal information to obtain multiple rule atoms, wherein different rule atoms correspond to different driving behavior norms; annotating the multiple rule atoms with scenarios to obtain the scenarios corresponding to the multiple rule atoms; and constructing a knowledge graph based on the dependencies between the multiple rule atoms and the scenarios corresponding to the multiple rule atoms to obtain the first knowledge base.

[0161] Optionally, the risk prediction module is used to fill the prompt word template with the target knowledge text and structured perception data to generate the target prompt word; call the risk prediction model and input the target prompt word into the risk prediction model, use the risk prediction model to perform risk prediction, and obtain the risk prediction result.

[0162] Optionally, the risk prediction module is also used to obtain historical perception data and the historical risk prediction results corresponding to the historical perception data; to fill the target knowledge text, structured perception data, historical perception data and historical risk prediction results into the prompt word template to generate target prompt words; preferably, the risk prediction model is a large model deployed locally, or a model composed of multiple models, or a question-and-answer model.

[0163] Optionally, the result fusion module is used to associate the risk prediction result with at least one object in the structured perception data to obtain the target object; quantify the risk level in the risk prediction result to obtain the risk score corresponding to the target object; and spatialize the risk score in the preset space of the vehicle to obtain the risk information of the target object.

[0164] Optionally, the result fusion module is also used to generate a risk potential field in a preset space based on the risk score and the category of the target object, wherein the risk potential field is centered on the target object; add semantic tags to the target object, wherein the semantic tags are used to characterize the risk attributes of the target object; and obtain risk information based on the risk potential field and the semantic tags.

[0165] Optionally, the system is also used to determine the risk priority of multiple sensing objects based on risk indicators between multiple sensing objects and the vehicle in structured sensing data; extract information from the structured sensing data based on the risk priority to obtain the vehicle status, environmental information, and at least one target object; generate sensing text based on the vehicle status, environmental information, and at least one target object; and retrieve target knowledge text matching the structured sensing data by searching a preset knowledge base based on the structured sensing data, including: retrieving target knowledge text matching the sensing text by searching the preset knowledge base based on the sensing text; and performing risk assessment on the structured sensing data based on the target knowledge text. The method for predicting environmental risks includes: generating risk prediction text based on perceived text and target knowledge text; performing risk prediction on the risk prediction text to obtain a risk prediction result; preferably, after fusing the risk prediction result and structured perceived data to obtain target perceived data, the method further includes: adjusting the vehicle's driving behavior or driving trajectory based on the target perceived data; preferably, after performing risk prediction on structured perceived data based on target knowledge text to obtain an environmental risk prediction result, the method further includes: adjusting the vehicle's driving behavior or driving trajectory based on the risk prediction result; and / or, generating prompt information based on the risk prediction result.

[0166] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application when it runs.

[0167] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0168] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0169] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0170] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of this application.

[0171] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0173] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0175] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0176] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing vehicle perception data, characterized in that, include: The structured perception data of the vehicle is obtained by fusing raw data perceived by multiple sensors on the vehicle. The structured perception data is used to retrieve the preset knowledge base to obtain the target knowledge text that matches the structured perception data. The knowledge stored in the preset knowledge base is used to characterize driving behavior norms or operating modes in dangerous scenarios. Risk prediction is performed on the structured perception data based on the target knowledge text to obtain the risk prediction result of the vehicle's environment; The risk prediction results and the structured perception data are fused to obtain target perception data, wherein the target perception data includes at least the target object in the scenario that has a risk and the risk information of the target object.

2. The method according to claim 1, characterized in that, The step of retrieving a preset knowledge base based on the structured perception data to obtain target knowledge text that matches the structured perception data includes one of the following: Extract at least one keyword from the structured perception data, and search the preset knowledge base based on the at least one keyword to obtain the target knowledge text that matches the at least one keyword; The structured perception data is encoded to obtain a query vector, and the preset knowledge base is retrieved based on the query vector to obtain the target knowledge text with a similarity greater than a preset similarity to the query vector; Based on the at least one keyword and the query vector, the preset knowledge base is searched to obtain the target knowledge text that matches the at least one keyword and the query vector; A knowledge graph is constructed based on the preset knowledge base, and the knowledge graph is traversed based on the structured perception data to obtain the target knowledge text; The preset knowledge base is retrieved based on the structured perception data to obtain a first knowledge text that matches the structured perception data. A network search is then performed based on the structured perception data to obtain a second knowledge text. The first knowledge text and the second knowledge text are then combined to obtain the target knowledge text.

3. The method according to claim 2, characterized in that, The method further includes: Obtain legal information related to vehicle operation, as well as historical driving behavior data; The legal information is subjected to natural language processing to construct a first knowledge base, wherein the knowledge in the first knowledge base includes driving behavior norms and the scenarios corresponding to the driving behavior norms; The historical driving behavior data is deeply mined to construct a second knowledge base, wherein the knowledge text in the second knowledge base includes objects in the environment and operating modes when the objects are at risk; Based on the first knowledge base and the second knowledge base, the preset knowledge base is obtained; Preferably, the step of performing natural language processing on the legal information to construct a first knowledge base includes: The legal information is broken down into multiple rule atoms, where different rule atoms correspond to different driving behavior norms; Scene annotation is performed on the multiple rule atoms to obtain the scenes corresponding to the multiple rule atoms; Based on the dependencies between the multiple rule atoms, a knowledge graph is constructed from the multiple rule atoms and the scenarios corresponding to the multiple rule atoms to obtain the first knowledge base.

4. The method according to claim 1, characterized in that, The step of performing risk prediction on the structured perception data based on the target knowledge text to obtain the risk prediction result of the environment includes: The target knowledge text and the structured perception data are filled into the prompt word template to generate target prompt words; The risk prediction model is invoked, and the target prompt word is input into the risk prediction model. The risk prediction model is then used to perform risk prediction, and the risk prediction result is obtained.

5. The method according to claim 4, characterized in that, The step of filling the target knowledge text and the structured perception data into the prompt word template to generate target prompt words includes: Obtain historical sensing data and the corresponding historical risk prediction results; The target knowledge text, the structured perception data, the historical perception data, and the historical risk prediction results are filled into the prompt word template to generate the target prompt word; Preferably, the risk prediction model is a large model deployed locally, or a model composed of multiple models, or a question-and-answer model.

6. The method according to claim 1, characterized in that, The process of fusing the risk prediction results and the structured perception data to obtain target perception data includes: The risk prediction result is associated with at least one object in the structured perception data to obtain the target object; Based on the risk level in the risk prediction results, a risk score corresponding to the target object is obtained by quantification. The risk score is spatialized within a preset space of the vehicle to obtain the risk information of the target object.

7. The method according to claim 6, characterized in that, The step of spatializing the risk score within a preset space of the vehicle to obtain the risk information of the target object includes: Based on the risk score and the category of the target object, a risk potential field is generated in the preset space, wherein the risk potential field is centered on the target object; Add semantic tags to the target object, wherein the semantic tags are used to characterize the risk attributes of the target object; The risk information is obtained based on the risk potential field and the semantic label.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Based on the risk indicators between multiple sensing objects and the vehicle in the structured sensing data, the risk priority of the multiple sensing objects is determined; Based on the risk priority, information is extracted from the structured perception data to obtain the vehicle status, environmental information, and at least one target object. Based on the vehicle status, the environmental information, and the at least one target object, generate perceived text; The step of retrieving a preset knowledge base based on the structured perception data to obtain target knowledge text that matches the structured perception data includes: The preset knowledge base is searched based on the perceived text to obtain the target knowledge text that matches the perceived text; The step of performing risk prediction on the structured perception data based on the target knowledge text to obtain the risk prediction result of the environment includes: Based on the perceived text and the target knowledge text, a risk prediction text is generated; Perform risk prediction on the risk prediction text to obtain the risk prediction result; Preferably, after fusing the risk prediction results and the structured perception data to obtain target perception data, the method further includes: The driving behavior or driving trajectory of the vehicle is adjusted based on the target perception data; Preferably, after performing risk prediction on the structured sensing data based on the target knowledge text to obtain the risk prediction result of the environment, the method further includes: Adjust the driving behavior or trajectory of the vehicle based on the risk prediction results; and / or, A prompt message is generated based on the risk prediction results.

9. A system for processing vehicle perception data, characterized in that, include: The data acquisition module is used to acquire structured perception data of the vehicle, wherein the structured perception data is obtained by fusing raw data perceived by multiple sensors on the vehicle. The knowledge retrieval module is used to retrieve a preset knowledge base based on the structured perception data to obtain target knowledge text that matches the structured perception data. The knowledge stored in the preset knowledge base is used to characterize driving behavior norms or operating modes in dangerous scenarios. The risk prediction module is used to perform risk prediction on the structured perception data based on the target knowledge text to obtain the risk prediction result of the environment in which the vehicle is located. The result fusion module is used to fuse the risk prediction results and the structured perception data to obtain target perception data, wherein the target perception data includes at least the target object with risk in the scenario and the risk information of the target object.

10. A vehicle, characterized in that, include: The vehicle perception data processing system as described in claim 9.

Images