Data processing method and device, electronic equipment and automatic driving vehicle

Abstract

The invention provides a data processing method and device, electronic equipment and an automatic driving vehicle, and relates to the technical field of computers, in particular to the technical field of automatic driving and artificial intelligence. According to the implementation scheme, the vehicle state and environment information of the automatic driving vehicle are obtained; natural language features including the vehicle state and the environment information are generated; and inputting the natural language features into a large language model, the controller is coupled to the large language model to control the large language model to output a current autonomous driving scene description, a future prediction for the described autonomous driving scene, and a driving evaluation for the autonomous vehicle based on the scene description and the future prediction of the scene.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of autonomous driving and artificial intelligence technology, and specifically to a data processing method, device, electronic device, computer-readable storage medium, computer program product, and autonomous driving vehicle. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] Evaluating autonomous driving during the process can provide feedback for autonomous driving behavior and further improve the autonomous driving model.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a data processing method, an apparatus, an electronic device, a computer-readable storage medium, a computer program product, and an autonomous driving vehicle.

[0006] According to one aspect of the present disclosure, there is provided a data processing method, comprising: acquiring vehicle status and environmental information of an autonomous driving vehicle; generating natural language features including the vehicle status and the environmental information; and inputting the natural language features into a large language model to control the large language model to output the following items: a current autonomous driving scene description, a future prediction for the described autonomous driving scene, and generating a driving evaluation of the autonomous driving vehicle based on the contents of the scene description and the future prediction of the scene.

[0007] According to another aspect of the present disclosure, a data processing device is provided, including: an acquisition unit, configured to acquire vehicle status and environmental information of an autonomous driving vehicle; a feature generation unit, configured to generate natural language features including the vehicle status and the environmental information; and an evaluation unit, configured to input the natural language features into a large language model to control the large language model to output the following items: a current autonomous driving scene description, a future prediction for the described autonomous driving scene, and a driving evaluation of the autonomous driving vehicle generated based on the contents of the scene description and the future prediction of the scene.

[0008] According to another aspect of the present disclosure, an electronic device is also provided, including: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method according to an embodiment of the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, where the computer instructions are used to enable the computer to execute the data processing method according to the embodiment of the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is further provided, including a computer program, wherein the computer program implements the data processing method according to the embodiment of the present disclosure when executed by a processor.

[0011] According to another aspect of the present disclosure, an autonomous driving vehicle is also provided, comprising a data processing device according to an embodiment of the present disclosure and one of the above-mentioned electronic devices.

[0012] According to one or more embodiments of the present disclosure, the general problem-solving capabilities of a large language model can be utilized to effectively analyze autonomous driving scenarios and output more accurate evaluation results.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0016] Figure 2 An exemplary process of a data processing method according to an embodiment of the present disclosure is shown;

[0017] Figure 3 An exemplary principle of an autonomous driving evaluation model according to an embodiment of the present disclosure is shown;

[0018] Figure 4 A structural block diagram of a data processing device according to an embodiment of the present disclosure is shown;

[0019] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0021] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0022] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0023] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0024] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1, the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0025] In an embodiment of the present disclosure, the motor vehicle 110 may include a computing device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0026] The server 120 may run one or more services or software applications that enable methods for data processing related to autonomous driving. In some embodiments, the server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0027] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0028] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0029] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0030] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, one or more networks 130 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, WiFi), and / or any combination of these and other networks.

[0031] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The data repository 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The data repository 150 may be of different types. In some embodiments, the data repository used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to a command.

[0032] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0033] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0034] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth of the IEEE802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0035] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of a computing device on the motor vehicle side (client) according to the present disclosure.

[0036] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0037] Autonomous driving systems need to evaluate their driving behaviors in real time in complex traffic environments to ensure driving safety, comfort, and efficiency. However, most automatic evaluation technologies are based on rules pre-defined by experts or end-to-end machine learning scoring models, and lack the ability to understand and respond to human language descriptions. This results in the system being unable to flexibly and accurately evaluate autonomous driving behaviors and adjust driving strategies accordingly when dealing with complex and diverse driving scenarios. In addition, related technologies lack interpretability and personalization in their evaluations. Therefore, there is an urgent need for an automatic evaluation technology for autonomous driving that can integrate multimodal information and dynamically adjust according to the language descriptions and preferences of human users.

[0038] The automatic evaluation technology of autonomous driving in related technologies mainly relies on rule-based or machine learning models. These models usually use camera, radar and other sensor data input to evaluate whether the vehicle's behavior meets pre-set standards. Common methods include:

[0039] 1) Rule-based evaluation method: The quality of driving behavior is judged by preset rules (such as lane keeping and obstacle avoidance).

[0040] 2) Statistical analysis method: Use statistical models to analyze driving behavior, such as probability models, distribution analysis, etc.

[0041] 3) Machine learning model: Evaluate driving behavior through supervised learning or unsupervised learning models to identify abnormal or bad driving behavior.

[0042] The above evaluation method has the following shortcomings.

[0043] (1) Single modality: It mainly relies on single modal data (such as visual or sensor data), lacks the ability to comprehensively process multimodal information, and cannot fully understand the multi-dimensional information in complex driving scenarios.

[0044] (2) Lack of personalization and flexibility: Most existing technologies use fixed evaluation criteria, which are difficult to dynamically adjust according to the preferences of different users or changes in scenarios.

[0045] (3) Lack of explainability: Existing evaluation models usually output simple scoring or classification results, lack detailed explanations of the evaluation results, and are unable to provide meaningful feedback to developers or users.

[0046] In order to improve the autonomous driving evaluation technology, the present invention proposes an autonomous driving automatic evaluation technology based on a multimodal large model. By comprehensively processing multimodal data such as vision, 3D point cloud, map information, etc., it realizes a detailed and dynamic evaluation of the driving behavior of the autonomous driving system, and provides detailed reward explanations and personalized reward mechanisms.

[0047] Figure 2 An exemplary process of a data processing method according to an embodiment of the present disclosure is shown.

[0048] In step S202, the vehicle state and environment information of the autonomous driving vehicle are obtained.

[0049] In step S204, natural language features including vehicle status and environment information are generated.

[0050] In step S206, natural language features are input into the large language model to control the large language model to output the following items: a description of the current autonomous driving scenario, a future prediction of the described autonomous driving scenario, and a driving evaluation of the autonomous driving vehicle generated based on the content of the scenario description and the future prediction of the scenario.

[0051] By utilizing the data processing method provided by the embodiments of the present disclosure, it is possible to utilize the general problem-solving capabilities of a large language model to effectively analyze autonomous driving scenarios and output more accurate evaluation results.

[0052] Hereinafter, embodiments of the present disclosure will be described in detail.

[0053] In step S202, the vehicle status and environmental information of the autonomous driving vehicle may be obtained.

[0054] The vehicle state may include at least one of the speed, direction, acceleration, and steering angle of the autonomous driving vehicle. Other vehicle data may also be collected as the vehicle state according to actual conditions. The vehicle state information may be collected by sensors in the autonomous driving system, and the above vehicle state information may be converted into a description in natural language. For example, a natural language description template describing the vehicle state may be pre-set, and the vehicle state information described in natural language may be generated in combination with the vehicle state information collected by the sensor. In the example, the natural language description template may be expressed as "the current speed of the vehicle is ___, the current direction is ___, the current acceleration is ___, and the current steering angle is ___". The vehicle state data collected by the sensor may be filled into the above template, and the vehicle state information described in natural language may be generated.

[0055] Environmental information may include at least one of the following: images collected by a camera, information collected by a radar, and map information. Environmental information may represent the driving environment of an autonomous driving vehicle, thereby providing a basis for decision-making and evaluation of autonomous driving. Without departing from the principles of the present disclosure, environmental information may also include any other information related to the driving environment.

[0056] In step S204 , natural language features including vehicle status and environment information may be generated.

[0057] For information about the vehicle state described in natural language, the vehicle state can be tokenized to obtain a first natural language feature corresponding to the vehicle state. In some examples, the vehicle state described in natural language can be input into a large language model and encoded by a tokenizer module of the large language model.

[0058] For other forms of vehicle status information that are not described in natural language, any suitable encoding method may be used to convert the vehicle status information into a form that can be processed by the large language model. For example, the vehicle status information may be mapped to the vocabulary space of the large language model, or the vehicle status information may be processed using a machine learning model to generate features of the vehicle status information that can be processed by the large language model.

[0059] For the environmental information, the environmental information can be mapped to the vocabulary space of the large language model to obtain a second natural language feature corresponding to the environmental information. In the example, different types of environmental information can be encoded, and then the encoded environmental information is mapped to the vocabulary space.

[0060] For the visual information collected by cameras, radars and other devices included in the environmental information, the visual information can be encoded using a visual encoder. For example, the visual information can be processed by a convolutional neural network or a visual transformer to extract environmental features. The environmental features mentioned here may include roads, vehicles, pedestrians, traffic signs, etc. The encoded visual information can be effectively represented and passed to subsequent processing modules. For the 3D point cloud information collected by radar or lidar included in the environmental information, the point cloud information can be encoded using a 3D point cloud encoder. For example, the point cloud information can be encoded using a point cloud processing network (such as PointNet). 3D point cloud data can provide depth and spatial structure information of the environment, such as the distance, size and shape of obstacles, which is crucial for the autonomous driving system to understand complex three-dimensional environments. For the map information from the navigation system included in the environmental information, the map encoder can be used to vector encode the map information. The map information mentioned here may include road structure, driving path, traffic rules, etc. The map information provides the location information of the vehicle in the macro environment, which helps to evaluate the path planning and decision direction.

[0061] Through encoding, multimodal environmental information can be encoded into features that can be uniformly processed by subsequent modules. Then, the encoding results of the environmental information can be mapped to the factor space of the vocabulary of the large language model. In the example, the encoding results of the environmental information can be mapped to the latent space through a multi-layer neural network. The above multi-layer neural network can be implemented using a multi-layer MLP. This mapping process can effectively fuse the encoding results of environmental information of different modalities, and ensure that different types of data from vision, 3D point clouds and maps can be comprehensively analyzed and processed in the same latent space, laying the foundation for the unified evaluation of multimodal large language models.

[0062] In step S206, the natural language features generated in step S204 may be input into a large language model to control the large language model to output the following items: a description of the current autonomous driving scenario, a future prediction of the described autonomous driving scenario, and a driving evaluation of the autonomous driving vehicle generated based on the content of the scenario description and the future prediction of the scenario.

[0063] In some embodiments, the large language model may output in the following order: (1) a description of the current autonomous driving scenario; (2) a future prediction of the described autonomous driving scenario; and (3) generating a driving evaluation of the autonomous driving vehicle based on the content of the scenario description and the future prediction of the scenario.

[0064] The preset prompt information (prompt) and the natural language features generated in step S204 can be input into the large language model together. The prompt information is used to guide the large language model to generate an evaluation result for the current driving behavior according to a predetermined reasoning logic. In the method provided by the embodiment of the present disclosure, the large language model not only understands the input vehicle state and environmental information based on its own general problem-solving capabilities, but also needs to first generate a current autonomous driving scene description according to a predetermined reasoning path, and then make a future prediction of the current driving scene based on the information in the scene description, and finally evaluate the driving behavior based on a combination of the current vehicle state and environmental information and the predicted information.

[0065] The large language model can generate a description of the current scene based on multimodal input information. The scene description can include structured scene information, such as at least one of the road conditions, the behavior of traffic participants, ground signs, and weather conditions. The purpose of the scene description is to express all the key information of the current driving environment in a structured manner, providing a basis for subsequent scene reasoning and evaluation generation.

[0066] Based on the detailed scene description, the multimodal large model performs evaluation-related scene reasoning. The main task of scene reasoning is to make future predictions for the current traffic scene. The content of future predictions includes events that will affect driving behavior in the described scene at a future moment, such as risks that will arise, such as "the vehicle ahead may suddenly stop" or "pedestrians may cross the intersection". The information predicted in the future will directly affect the final autonomous driving evaluation results, ensuring that the autonomous driving system can foresee possible risks in advance and respond appropriately.

[0067] Using the above method, the autonomous driving model can be evaluated more comprehensively and accurately. In some scenarios, the autonomous driving vehicle may perform some driving behaviors that do not comply with the predetermined rules to avoid more serious accidents. For example, if there are pedestrians crossing the road in the current driving scenario, the vehicle may choose a driving path that presses the solid line of the lane, otherwise it will hit the pedestrian. If the rule-based evaluation method in the relevant technology is adopted, it may be simply considered that the vehicle's driving path presses the solid line and violates the traffic rules, so a more negative evaluation will be made. However, if the overall driving scenario is considered, it will be found that the vehicle avoids more serious accidents through lighter illegal driving behaviors, so a more positive evaluation may be required for this driving behavior. Therefore, using the data processing method of the embodiment of the present disclosure, the large model can infer and predict the driving scenario based on the input vehicle status and environmental information, thereby improving the comprehensiveness and accuracy of the evaluation of driving behavior.

[0068] In some embodiments, the input of the large language model may also include a first evaluation criterion in the form of natural language, so that the large language model generates a first driving evaluation of the autonomous driving vehicle according to the first evaluation criterion. Among them, the evaluation criterion may be an evaluation criterion for the autonomous driving behavior input by the user through natural language. Exemplary evaluation criteria may include specific instructions such as "giving priority to safe driving" and "reducing emergency braking on urban roads". The input evaluation criterion may be encoded by the tokenizer module of the large language model and passed as input to the large language model. In this way, the driving behavior can be evaluated according to the preferences provided by the user and a corresponding reward can be generated. By inputting the evaluation criterion and information such as vehicle status and environmental information into the large language model together, the driving behavior can be evaluated in a personalized manner. In some examples, method 200 may also include replacing the first evaluation criterion with a second evaluation criterion different from the first evaluation criterion to control the large language model to generate a second driving evaluation of the autonomous driving vehicle according to the second evaluation criterion. Using this method, the evaluation criterion of the autonomous driving behavior can be adjusted at any time according to the actual situation, so that the evaluation criteria can be easily changed to achieve personalized evaluation generation.

[0069] The driving evaluation output in step S206 may include at least one of a score for the driving behavior of the autonomous driving vehicle, a warning for potential risks, suggestions for improving the current driving strategy, and a natural language explanation of the driving evaluation. Based on the scene description and scene reasoning, the large language model generates the final autonomous driving evaluation results and related explanations. The evaluation results may include a score for driving behavior, a warning for potential risks, suggestions for improving the current driving strategy, etc. At the same time, the large language model can also generate a natural language explanation of the evaluation results, detailing why such an evaluation was made and which factors had a significant impact on the evaluation results. This explanatory information helps developers and users better understand the decision-making process and enhance trust in the autonomous driving system.

[0070] The data processing method provided by the embodiment of the present disclosure has multimodal processing capabilities, can comprehensively utilize multiple data sources such as vision, language, and sensors, can more accurately understand complex driving scenarios, and improve the evaluation accuracy of autonomous driving behavior. In addition, the reward mechanism can be dynamically adjusted according to the preferences of different users and real-time driving scenarios to meet personalized needs. Furthermore, a detailed natural language explanation can be generated for the evaluation results, which enhances the interpretability of the evaluation and user trust. By evaluating driving behavior using the method of the embodiment of the present disclosure, the reward strategy can be adjusted in real time to ensure that the best driving decisions are made in different driving environments.

[0071] Figure 3The exemplary principle of the autonomous driving evaluation model according to the embodiment of the present disclosure is shown. Figure 3 The example implementation shown in Figure 2 A data processing method 200 is described.

[0072] like Figure 3 As shown, the evaluation criteria 301, the vehicle state 302 and the environmental information 303 can be input into the multimodal large language model 330. As shown in the figure, the environmental information 303 can include information collected by a camera, a radar, a lidar or information from a map system. The multimodal environmental information can be encoded using a multimodal encoder 310. In the example, the multimodal encoder 310 may include at least one of a visual encoder, a 3D point cloud encoder and a map encoder. The encoding result of the multimodal encoder can be mapped to the vocabulary latent space of the large language model using an adaptation layer 320, so that the large language model can understand the meaning of the input environmental information.

[0073] The large language model 330 can generate a description of the current scene based on the input information, and then perform scene reasoning for future predictions of the evaluation-related scenes based on the detailed scene description, and finally generate the final autonomous driving evaluation results based on the scene description and scene reasoning. The large language model 330 here not only needs to generate evaluation results based on the above-mentioned reasoning path, but also needs to generate corresponding natural language description results for the scene description and scene reasoning tasks performed in the middle of the reasoning process, rather than just generating intermediate vectors as feature transformations corresponding to the reasoning process. Using this method, the large language model can obtain more detailed and complete scene descriptions and reasoning information, and therefore can obtain more accurate driving behavior evaluation results.

[0074] Figure 4 FIG. 4 is a block diagram showing a structure of a data processing device 400 according to an embodiment of the present disclosure. Figure 4 The data processing device 400 shown in FIG. Figure 2 A data processing method 200 is described.

[0075] like Figure 4 As shown, the device 400 includes an acquisition unit 410 , a feature generation unit 420 and an evaluation unit 430 .

[0076] The acquisition unit 410 may be configured to acquire vehicle status and environment information of the autonomous driving vehicle;

[0077] The feature generation unit 420 may be configured to generate natural language features including vehicle status and environment information;

[0078] The evaluation unit 430 can be configured to input natural language features into the large language model to control the large language model to output the following: a current autonomous driving scene description, a future prediction of the described autonomous driving scene, and a driving evaluation of the autonomous driving vehicle generated based on the content of the scene description and the future prediction of the scene.

[0079] In some embodiments, the large language model may output in the following order: (1) a description of the current autonomous driving scenario; (2) a future prediction of the described autonomous driving scenario; and (3) generating a driving evaluation of the autonomous driving vehicle based on the content of the scenario description and the future prediction of the scenario.

[0080] In some embodiments, the vehicle state includes at least one of the speed, direction, acceleration, and steering angle of the autonomous vehicle.

[0081] In some embodiments, the environmental information includes at least one of the following: images collected by a camera, information collected by a radar, and map information.

[0082] In some embodiments, the feature generation unit is configured to: tokenize the vehicle state to obtain a first natural language feature corresponding to the vehicle state; map the environmental information to the vocabulary space of the large language model to obtain a second natural language feature corresponding to the environmental information.

[0083] In some embodiments, the input of the large language model also includes a first evaluation criterion in natural language form, so that the large language model generates a first driving evaluation of the autonomous driving vehicle according to the first evaluation criterion.

[0084] In some embodiments, the device 400 also includes an updating unit configured to replace the first evaluation criterion with a second evaluation criterion different from the first evaluation criterion, so as to control the large language model to generate a second driving evaluation of the autonomous driving vehicle according to the second evaluation criterion.

[0085] In some embodiments, the scene description includes structured scene information, and the scene information includes at least one of road conditions, behaviors of traffic participants, ground signs, and weather conditions.

[0086] In some embodiments, the content of the future prediction includes events that affect driving behavior and occur in the future in the described scenario.

[0087] In some embodiments, the driving evaluation includes at least one of a score of the driving behavior of the autonomous vehicle, a warning of potential risks, a suggestion for improvement of the current driving strategy, and a natural language interpretation of the driving evaluation.

[0088] It should be understood that Figure 4 The various modules or units of the apparatus 400 shown in FIG. 4 can be used in conjunction with the reference Figure 2 The above-described method 200 corresponds to each step in the method 200. Therefore, the operations, features and advantages described above for the method 200 are also applicable to the device 400 and the modules and units included therein. Therefore, the operations, features and advantages described above for the method 200 are also applicable to the device 400 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0089] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various units discussed herein may be separated into multiple units, and / or at least some functionality of multiple units may be combined into a single unit.

[0090] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 4 The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of units 410 to 430 can be implemented together in a system on chip (System on Chip, SoC). SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0091] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0092] According to another aspect of the present disclosure, an electronic device is also provided, including: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method according to an embodiment of the present disclosure.

[0093] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, where the computer instructions are used to enable the computer to execute the data processing method according to the embodiment of the present disclosure.

[0094] According to another aspect of the present disclosure, a computer program product is further provided, including a computer program, wherein the computer program implements the data processing method according to the embodiment of the present disclosure when executed by a processor.

[0095] According to another aspect of the present disclosure, an autonomous driving vehicle is also provided, comprising a data processing device according to an embodiment of the present disclosure and one of the above-mentioned electronic devices.

[0096] refer to Figure 5 , a block diagram of an electronic device 500 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0097] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0098] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 can be any type of device that can input information to the electronic device 500. The input unit 506 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 507 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 can include but is not limited to a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0099] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as method 200. For example, in some embodiments, the method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the method 200 in any other appropriate manner (e.g., by means of firmware).

[0100] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0102] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0104] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0105] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0106] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be performed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document is not limited here.

[0107] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A data processing method, comprising: Obtain vehicle status and environment information of the autonomous vehicle; generating a natural language feature including the vehicle state and the environment information; as well as The natural language features are input into a large language model to control the large language model to output the following: a current autonomous driving scene description, a future prediction of the described autonomous driving scene, and a driving evaluation of the autonomous driving vehicle generated based on the scene description and the future prediction of the scene.

2. The data processing method according to claim 1, wherein: The large language model outputs in the following order: (1) a description of the current autonomous driving scenario; (2) a future prediction of the described autonomous driving scenario; and (3) generating a driving evaluation of the autonomous driving vehicle based on the content of the scenario description and the future prediction of the scenario.

3. The data processing method according to claim 1, wherein: The vehicle state includes at least one of the speed, direction, acceleration, and steering angle of the autonomous driving vehicle.

4. The data processing method according to claim 1, wherein: The environmental information includes at least one of the following: Images collected by cameras, information collected by radars, and map information.

5. The data processing method according to claim 1, wherein: Generating natural language features including the vehicle state and the environment information includes: Tokenizing the vehicle state to obtain a first natural language feature corresponding to the vehicle state; The environmental information is mapped to the vocabulary space of the large language model to obtain a second natural language feature corresponding to the environmental information.

6. The data processing method according to any one of claims 1 to 5, wherein: The input of the large language model also includes a first evaluation criterion in natural language form, so that the large language model generates a first driving evaluation of the autonomous driving vehicle according to the first evaluation criterion.

7. The data processing method as described in claim 6 also includes replacing the first evaluation criterion with a second evaluation criterion different from the first evaluation criterion to control the large language model to generate a second driving evaluation of the autonomous driving vehicle according to the second evaluation criterion.

8. The data processing method according to any one of claims 1 to 5, wherein: The scene description includes structured scene information, and the scene information includes at least one of road conditions, behaviors of traffic participants, ground signs, and weather conditions.

9. The data processing method according to any one of claims 1 to 5, wherein: The content of the future prediction includes events that affect driving behavior and occur in the future in the described scenario.

10. The data processing method according to any one of claims 1 to 5, wherein: The driving evaluation includes at least one of a score for the driving behavior of the autonomous driving vehicle, a warning for potential risks, suggestions for improving the current driving strategy, and a natural language interpretation of the driving evaluation.

11. A data processing device, comprising: an acquisition unit configured to acquire vehicle status and environment information of the autonomous driving vehicle; a feature generation unit configured to generate a natural language feature including the vehicle state and the environment information; An evaluation unit is configured to input the natural language features into a large language model to control the large language model to output the following: a current autonomous driving scene description, a future prediction of the described autonomous driving scene, and generate a driving evaluation of the autonomous driving vehicle based on the scene description and the future prediction of the scene.

12. The data processing apparatus according to claim 11, wherein: The large language model outputs in the following order: (1) a description of the current autonomous driving scenario; (2) a future prediction of the described autonomous driving scenario; and (3) generating a driving evaluation of the autonomous driving vehicle based on the content of the scenario description and the future prediction of the scenario.

13. The data processing apparatus according to claim 11, wherein: The vehicle state includes at least one of the speed, direction, acceleration, and steering angle of the autonomous driving vehicle.

14. The data processing apparatus according to claim 11, wherein: The environmental information includes at least one of the following: Images collected by cameras, information collected by radars, and map information.

15. The data processing apparatus according to claim 11, wherein: The feature generation unit is configured to: Tokenizing the vehicle state to obtain a first natural language feature corresponding to the vehicle state; The environmental information is mapped to the vocabulary space of the large language model to obtain a second natural language feature corresponding to the environmental information.

16. The data processing device according to any one of claims 11 to 15, wherein: The input of the large language model also includes a first evaluation criterion in natural language form, so that the large language model generates a first driving evaluation of the autonomous driving vehicle according to the first evaluation criterion.

17. The data processing device as described in claim 16 further includes an updating unit configured to replace the first evaluation criterion with a second evaluation criterion different from the first evaluation criterion, so as to control the large language model to generate a second driving evaluation of the autonomous driving vehicle according to the second evaluation criterion.

18. The data processing device according to any one of claims 11 to 15, wherein: The scene description includes structured scene information, and the scene information includes at least one of road conditions, behaviors of traffic participants, ground signs, and weather conditions.

19. The data processing device according to any one of claims 11 to 15, wherein: The content of the future prediction includes events that affect driving behavior and occur in the future in the described scenario.

20. The data processing device according to any one of claims 11 to 15, wherein: The driving evaluation includes at least one of a score for the driving behavior of the autonomous driving vehicle, a warning for potential risks, suggestions for improving the current driving strategy, and a natural language interpretation of the driving evaluation.

21. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

24. An autonomous driving vehicle comprising: A data processing device as claimed in any one of claims 11 to 20 or an electronic device as claimed in claim 21.

Images