Vehicle-mounted question and answer method, device, equipment, medium and product

Abstract

The invention provides a vehicle-mounted question-answering method, device and equipment, a medium and a product, and the method comprises the steps: obtaining scene description data of a vehicle in a current driving scene in response to a vehicle-mounted question-answering request; performing scene understanding on the scene description data through a pre-trained vehicle-mounted question and answer model, and generating reply content for the vehicle-mounted question and answer request according to a scene understanding result; wherein the vehicle-mounted question and answer model is obtained by performing scene understanding task training on a large language model according to scene description sample data. According to the invention, the understanding ability of the driving scene of the vehicle can be improved, and the accuracy of the generated answer in the vehicle-mounted question and answer scene is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent question answering, and in particular to an in-vehicle question answering method, device, equipment, medium and product. Background Art

[0002] In the field of transportation, whether it is traditional vehicles or autonomous vehicles, user interactive question-answering systems are increasingly playing a key role, aiming to enable vehicles to understand and respond to user needs.

[0003] At present, in-vehicle question-answering technology mainly relies on rule-based methods, template matching-based methods or deep learning-based methods. However, these methods have limitations in understanding and parsing questions raised by users during driving, resulting in insufficient accuracy of generated answers and difficulty in meeting user needs. Summary of the invention

[0004] Based on the above-mentioned technical status, the present application provides an in-vehicle question-and-answer method, device, equipment, medium and product, which can improve the accuracy of understanding user questions during driving, and thereby improve the accuracy and relevance of answers.

[0005] In order to achieve the above technical objectives, this application specifically proposes the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a vehicle-based question-and-answer method is provided, comprising: in response to an vehicle-based question-and-answer request, obtaining scene description data of a vehicle in a current driving scenario; performing scene understanding on the scene description data through a pre-trained vehicle-based question-and-answer model, and generating reply content for the vehicle-based question-and-answer request based on the scene understanding result; wherein the vehicle-based question-and-answer model is obtained by training a large language model for a scene understanding task based on scene description sample data.

[0007] In some implementations, the scene description data is determined using the following steps: determining environmental understanding data based on sensor data of the vehicle, the sensor data of the vehicle including data characterizing a vehicle state and an environment surrounding the vehicle; and converting the environmental understanding data into scene description data in a preset format.

[0008] In some implementations, determining environmental understanding data based on sensor data of the vehicle includes: constructing an environmental model of the vehicle's current environment based on the sensor data to obtain the environmental understanding data, wherein the environmental model includes various static elements and various dynamic elements, and each of the static elements and each of the dynamic elements has corresponding status information.

[0009] In some implementations, the scene description data in the preset format includes: vehicle positioning data, interaction relationships between the vehicle and other vehicles or pedestrians, behavior prediction results of other vehicles or pedestrians, path planning information, navigation assistance information and at least one of vehicle status parameters.

[0010] In some implementations, the in-vehicle question and answer model is trained using the following steps: obtaining scene description sample data, the scene description sample data including scene description training data and scene description test data; inputting the scene description training data into a large language model so that the large language model maps the scene description training data into a knowledge representation within the large language model to obtain an initial in-vehicle question and answer model; testing the initial in-vehicle question and answer model using the scene description test data, and when the test result does not meet a preset condition, continuing the training until the test result meets the preset condition to obtain the in-vehicle question and answer model.

[0011] In some implementations, the scene description sample data is obtained by performing environmental modeling based on data representing the vehicle state and the surrounding environment and converting the data into a preset format.

[0012] In some implementations, the reply content includes the answer corresponding to the in-vehicle question and answer request, as well as the comprehensive score of the answer and the score explanation content; wherein, after generating the reply content for the in-vehicle question and answer request based on the scene understanding result, the method further includes: optimizing the in-vehicle question and answer model based on the comprehensive score of the answer corresponding to the in-vehicle question and answer request.

[0013] In some implementations, the comprehensive score of the answer is determined using the following steps: determining the score of the answer corresponding to the in-vehicle question and answer request from each scoring dimension, wherein each scoring dimension includes at least one of an accuracy dimension, a relevance dimension, and a timeliness dimension; and determining the comprehensive score of the answer based on the scores corresponding to each scoring dimension.

[0014] According to a second aspect of an embodiment of the present application, an in-vehicle question and answer device is provided, comprising: an acquisition unit, for obtaining scene description data of a vehicle in a current driving scenario in response to an in-vehicle question and answer request; an in-vehicle question and answer unit, for performing scene understanding on the scene description data through a pre-trained in-vehicle question and answer model, and generating reply content for the in-vehicle question and answer request based on the scene understanding result; wherein the in-vehicle question and answer model is obtained by training a large language model for a scene understanding task based on scene description sample data.

[0015] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory and a processor; the memory is connected to the processor and is used to store programs; the processor is used to implement the in-vehicle question and answer method as described in the first aspect by running the programs in the memory.

[0016] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the in-vehicle question-and-answer method as described in the first aspect is implemented.

[0017] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, cause the processor to execute: the in-vehicle question-and-answer method as described in the first aspect.

[0018] The embodiments of the present application provide an in-vehicle question-and-answer method, device, equipment, medium, and product, which, upon receiving an in-vehicle question-and-answer request, obtains scene description data of the vehicle in the current driving scenario, and performs scene understanding on the scene description data through a pre-trained in-vehicle question-and-answer model, and generates a reply content for the in-vehicle question-and-answer request based on the scene understanding result; wherein the in-vehicle question-and-answer model is obtained by training a large language model for a scene understanding task based on scene description sample data. Since the model has pre-learned and trained the scene understanding capability, it can more accurately understand and interpret the current driving scenario of the vehicle based on the scene description data. Afterwards, the model can generate diverse and accurate answers to user questions based on the scene understanding results, and provide more accurate question-and-answer services. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0020] Figure 1 A flowchart of an in-vehicle question-and-answer method provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of the principle of the in-vehicle question-answering method provided in an embodiment of the present application;

[0022] Figure 3 A flowchart of a training process of an in-vehicle question-answering model provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the model training process provided in the embodiment of the present application;

[0024] Figure 5 A schematic diagram of the architecture of fine-tuning training of a large language model provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of an in-vehicle question-and-answer scenario provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the functions implemented by the in-vehicle question-and-answer method provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the vehicle hardware architecture provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of the process flow of front-end data processing provided in an embodiment of the present application;

[0029] Fig.10 A schematic diagram of the structure of a vehicle-mounted question-and-answer device provided in an embodiment of the present application;

[0030] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solution provided in the embodiments of the present application can be exemplarily applied to hardware devices such as processors, electronic devices, servers (including cloud servers), or packaged into software programs to be run. When the hardware device executes the processing of the technical solution of the embodiments of the present application, or the above-mentioned software program is run, the automatic splitting of the target task and the automatic calling of the application program interface required for the task can be achieved to complete the purpose of the target task. The embodiments of the present application only exemplarily introduce the specific processing of the technical solution of the present application, and do not limit the specific implementation form of the technical solution of the present application. Any technical implementation form that can execute the processing of the technical solution of the present application can be adopted by the embodiments of the present application.

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] Before introducing this application solution, the relevant technologies are first introduced:

[0034] With the rapid development of science and technology, the application of intelligent systems in many fields is becoming more and more extensive, especially in improving user experience and automation level. The environmental perception, decision-making and execution control capabilities of intelligent systems have been significantly enhanced, promoting the development of application scenarios such as robot navigation and smart home control. In the field of transportation, whether it is traditional vehicles or autonomous vehicles, user interactive question-and-answer technology is playing an increasingly important role. By integrating advanced perception technology and natural language processing (NLP) capabilities, this technology enables vehicles to understand and respond to users' complex needs and realize drive question answering (DQA).

[0035] The current user interactive question-answering technology is mainly manifested in the following aspects:

[0036] Rule-based approach: Rely on predefined logical rules to parse user questions and retrieve answers from a structured knowledge base. These rules define how to extract key information from user questions and generate answers based on this information. However, the design and maintenance of rules are complex, and it is difficult to cope with the diversity and complexity in natural language, making it difficult to fully cover all potential questioning scenarios, and there is a lack of effective response mechanisms for the ambiguity and diversity in natural language. For example, some autonomous driving systems receive user voice commands through speech recognition technology, and when performing corresponding operations or providing preset answers based on the recognized commands, they usually need to rely on a fixed set of instructions, requiring users to follow preset grammar and vocabulary to ask questions. Its limitation is that users cannot ask complex questions outside the instruction set.

[0037] Template matching-based methods: By comparing user questions with pre-set patterns, key information is identified and corresponding responses are generated. For example, some systems integrate question-answering modules to answer questions by matching with pre-built knowledge bases. These knowledge bases contain a large number of preset questions and answers. When a user asks a question, the system searches the knowledge base for the best matching question and returns the corresponding answer. Although this method can simplify the question parsing process to a certain extent, it is incapable of handling non-standardized or complex queries. Due to the limited coverage of the knowledge base, it cannot effectively handle novel or undefined questions.

[0038] Deep learning-based methods: Using the powerful representation learning capabilities of neural networks, automatically learn patterns from a large number of question-answer pairs to achieve understanding and answering of questions. However, existing deep learning methods still have limitations in processing long texts, cross-domain knowledge transfer, and providing explanatory answers.

[0039] In summary, existing user interactive question-and-answer technologies still have many deficiencies in understanding user questions and generating high-quality answers. For example, in a driving environment, there is a lack of in-depth understanding of complex driving scenarios, and answer generation lacks flexibility and accuracy. Often, only preset answers can be provided, and answers cannot be dynamically generated according to the real-time environment, resulting in an inability to accurately answer complex questions involving environmental understanding. And it cannot help users understand the reliability and basis of the answers. In order to overcome these limitations, a technical solution for in-vehicle question-and-answering is urgently needed to improve the accuracy of the system's understanding of user questions, optimize the quality of generated answers, and thus improve the overall user experience.

[0040] In view of this, the embodiments of the present application are committed to providing a vehicle-mounted question-answering method, device, equipment, medium and product, which can more accurately understand the user's questions during driving by perceiving and understanding the vehicle's surrounding environment information, and then use the generation capability of the large language model to generate diverse and accurate answers to the user's questions. In addition, it is also possible to provide users with scores and score explanations based on the complexity of the user's questions and the accuracy of the answers, to help users better understand the potential risks during driving. Each of them is described in detail in the following embodiments.

[0041] Exemplary Methods

[0042] Figure 1 This is a flow chart of a vehicle-mounted question-answering method provided in an embodiment of the present application. Figure 1 As shown, the vehicle-mounted question-answering method provided in this embodiment includes step S101 and step S102:

[0043] S101. In response to an in-vehicle question-and-answer request, obtain scene description data of the vehicle in a current driving scene.

[0044] When a user initiates an in-vehicle question-and-answer request through an in-vehicle terminal or personal device by voice, touch screen interaction, gesture control or other input methods, such as "What is the traffic condition ahead?" or "Do I need to take a detour?" or "Where is the nearest gas station?", it is necessary to obtain detailed description data of the vehicle in the current driving scenario in order to provide sufficient information support for the in-vehicle question-and-answer model, ensuring that it can accurately understand the specific environment in which the vehicle is currently located, thereby generating accurate responses.

[0045] The scene description data is the description data of the current driving scene of the vehicle, including the description data of the vehicle itself and the environment around the vehicle. Specifically, the description data of the vehicle itself may include but is not limited to information such as speed, direction, acceleration, and vehicle status; while the description data of the vehicle's surrounding environment is more extensive, involving road types (such as highways, city streets), traffic flow, weather conditions, surrounding infrastructure (such as traffic lights, signs), the behavior of other road users (such as pedestrians, bicycles, other vehicles), and potential obstacles or emergencies.

[0046] In some embodiments, the scene description data may be determined using the following steps a1 and a2:

[0047] Step a1: determining environmental understanding data based on sensor data of the vehicle, wherein the sensor data of the vehicle includes data representing the vehicle state and the environment surrounding the vehicle.

[0048] Figure 2 The schematic diagram of the principle of the vehicle-mounted question-answering method provided in the embodiment of the present application is shown in FIG. Figure 2 As shown, determining environmental understanding data based on vehicle sensor data includes: acquiring vehicle sensor data; generating path planning data and control strategy data based on the vehicle sensor data through a regulatory control agent; and generating environmental understanding data based on the vehicle sensor data, path planning data and control strategy data.

[0049] In autonomous driving technology, the core task of step a1 is to collect and process raw sensor data through the vehicle's sensor network to build a deep understanding of the vehicle's surrounding environment. This process involves the coordinated work of multiple high-precision sensors, such as cameras, radars (Radio Detection and Ranging), and laser radars (Light Detection and Ranging, LiDAR), each of which is responsible for capturing different types of environmental information.

[0050] Specifically, cameras are used to capture visual information and identify traffic signs, traffic lights, lane lines, pedestrians and obstacles; radars are used to detect the distance, speed and angle of moving target objects (such as other vehicles or pedestrians) in the surrounding environment; and lidar is used to provide precise three-dimensional spatial information to help create detailed environmental maps.

[0051] The data collected by these sensors in real time undergoes preliminary preprocessing, including noise filtering, outlier detection, and data correction, to ensure data quality and consistency. Afterwards, the preprocessed data is converted into a unified standard format for easy storage and efficient management. In order to support fast retrieval and update, an indexing mechanism can also be established to facilitate quick query of required data based on key fields and features.

[0052] After completing the standardization operation, the sensor data can be transmitted to the background processing unit through the vehicle network (such as CAN bus or dedicated data line), so that it can build a comprehensive environmental model based on the sensor data.

[0053] Specifically, when determining the environment understanding data according to the sensor data, step a1 specifically includes: constructing an environment model of the vehicle's current environment according to the sensor data to obtain the environment understanding data. The environment model includes various static elements and various dynamic elements, and each static element and each dynamic element has corresponding state information.

[0054] The sensor data of the vehicle includes not only the sensor data collected about the vehicle's surrounding environment, but also the sensor data collected about the vehicle's own state.

[0055] The environment model built based on sensor data contains not only static elements (such as roads, traffic signs and lights), but also dynamic elements (such as other vehicles and pedestrians), each of which is configured with status information such as position and speed. The environment model is continuously updated based on real-time sensor data to ensure the accuracy and timeliness of its scene description.

[0056] Step a2: converting the environmental understanding data into scene description data in a preset format.

[0057] Continue reading Figure 2 ,In step a2, the environmental understanding data is further converted into scene description data in a preset format.,Step a2 aims to simplify complex environmental information into structured data that is easy to understand and use, so as to facilitate subsequent decision making and interactive response.

[0058] Among them, the scene description data in the preset format includes: vehicle positioning data, the interaction relationship between the vehicle and other vehicles or pedestrians, behavior prediction results of other vehicles or pedestrians, path planning information, navigation auxiliary information and at least one of the vehicle status parameters.

[0059] Vehicle positioning data refers to the determination of the vehicle's position in the environmental model, including the relative position relationship between the vehicle and the road, other vehicles and pedestrians.

[0060] Interaction relationship refers to the description of the interaction relationship between vehicles and other vehicles, and vehicles and pedestrians, such as observed static roads and dynamic pedestrians and vehicles; the angle of the vehicle from the vehicle, the distance from the vehicle, the speed of the vehicle, and the direction of the vehicle and the vehicle; the angle of the pedestrian from the vehicle, the distance from the vehicle, the speed of the pedestrian, and the direction of the pedestrian and the vehicle, in order to evaluate potential collision risks and driving strategies.

[0061] Behavior prediction results refer to predicting the dynamic behavior of other vehicles and pedestrians through machine learning algorithms so that more accurate decisions and responses can be made.

[0062] Path planning information refers to path planning based on the current driving scenario and vehicle status to determine the vehicle's driving route and destination.

[0063] The navigation assistance information includes the distance from the vehicle to the nearest traffic light intersection, the current road type (such as a roundabout, an intersection or other types), and whether there are traffic lights in the current environment.

[0064] The vehicle status parameters include the vehicle's speed, steering angle, acceleration pedal percentage, braking percentage, and next action.

[0065] In addition, scene description data can also include dynamic navigation updates, which adjusts navigation suggestions based on real-time environmental changes to ensure the accuracy and timeliness of navigation information.

[0066] It is worth noting that step a1 and step a2 are continuous and real-time processes. The sensors on the vehicle continuously collect data about the vehicle itself and its surroundings, and continuously perform a series of processing on them before sending them to the background processing unit. The background processing unit synchronously performs data analysis, environmental modeling, and scene description data generation. When an in-vehicle question-and-answer request is received, the latest scene description data can be immediately called to give instant and accurate answers or suggestions.

[0067] In the above-mentioned preprocessing operation on the sensor, in order to reduce errors and improve the accuracy of the data, a Kalman filter and a data fusion network may be used to fuse the data from different sensors.

[0068] The above embodiments introduce the implementation method of directly retrieving the converted scene description data. In some implementation methods, the scene description data can also be generated in real time. That is, when the in-vehicle question and answer request is received, the scene description data is generated in real time according to the environmental understanding data.

[0069] Continue reading Figure 1 After step S101, the in-vehicle question-and-answer method of this embodiment may further include the following step S102.

[0070] S102: Perform scene understanding on the scene description data through a pre-trained in-vehicle question-and-answer model, and generate a reply content for the in-vehicle question-and-answer request based on the scene understanding result.

[0071] Continue reading Figure 2 , by inputting the scene description data into the in-vehicle question and answer model, it can be applied to the in-vehicle question and answer task.

[0072] In step S102, the in-vehicle question-answering model is obtained by training the large language model for scene understanding tasks based on the scene description sample data. Therefore, the in-vehicle question-answering model has scene understanding capabilities. When the scene description data is input into the in-vehicle question-answering model, the model can perform in-depth scene understanding of the scene description data based on the scene understanding capabilities obtained in advance, and accurately respond to the in-vehicle question-answering request based on the scene understanding results. In order to better understand the training mechanism of the in-vehicle question-answering model, the following will be described in detail with reference to the accompanying drawings:

[0073] Figure 3 A flowchart of the training process of an in-vehicle question-answering model provided in an embodiment of the present application.

[0074] like Figure 3 As shown, the training process of the in-vehicle question-answering model includes the following steps S301 to S303:

[0075] Step S301: Acquire scene description sample data, where the scene description sample data includes scene description training data and scene description test data.

[0076] The scene description sample data is obtained by modeling the environment based on the data representing the vehicle state and the surrounding environment and converting it into a preset format. Specifically, a large amount of scene description sample data from different driving conditions can be collected, and the sample data contains a variety of driving scenarios. The original data is cleaned, labeled and formatted to ensure data quality and consistency for subsequent training.

[0077] The method for acquiring the scene description sample data in this embodiment is similar to the method for acquiring the scene description data described above. For details, reference may be made to the method for acquiring the scene description data described above, which will not be described in detail here.

[0078] After the scene description sample data is obtained, it can be divided into two parts, namely, scene description training data and scene description test data. The scene description training data is used for model training, and the scene description test data is used for model performance testing after training.

[0079] Figure 4 Schematic diagram of the model training process provided in the embodiment of the present application. Figure 4 As shown, by manually labeling or automatically labeling the data generated by the real vehicle, each element in the sensor data and the meaning represented by each element are annotated, thereby obtaining representation data of environmental perception, that is, scene description sample data.

[0080] Step S302: input the scene description training data into the large language model, so that the large language model maps the scene description training data into the knowledge representation inside the large language model, and obtains an initial in-vehicle question-answering model.

[0081] In step S302, a large language model (LLM) is trained to achieve in-depth understanding and accurate analysis of the vehicle's current driving scene. This process is the stage of model learning and training scene understanding capabilities.

[0082] Specifically, by inputting the scene description training data into the large language model, the scene description training data is learned through the machine learning algorithm, and the deep meaning behind the data is gradually understood. For example, in this process, the model needs to understand the details of the current driving scene of the vehicle and map the scene description data into the knowledge representation inside the model. Specifically, the model must not only identify static elements, such as road types, traffic signs and other infrastructure, but also dynamically analyze real-time changing factors, such as the behavior patterns of other vehicles, pedestrian movements, and unforeseen obstacles. Through the comprehensive processing of these multi-dimensional information, the model can gradually establish a comprehensive understanding of the driving scene and convert the information of the external world into internal knowledge representation. This knowledge representation is associated with the original scene description data through a series of complex mapping relationships. The scene description data is mapped to the knowledge representation of the large language model, and a mapping relationship between the current driving scene of the vehicle and the knowledge representation of the large language model is generated.

[0083] Continue reading Figure 4 After obtaining the scene description sample data, a large language model can be trained based on it to obtain an in-vehicle question-answering model. The in-vehicle question-answering model is then used to implement driving question-answering task processing. The specific training process is described below in conjunction with the accompanying drawings.

[0084] Figure 5 This is a schematic diagram of the architecture of the large language model fine-tuning training provided in the embodiment of the present application. Figure 5 As shown, the architecture of the large language model fine-tuning training includes: a vector encoder (Vector Encoder) 31, a vector former (VectorFormer) 32 and an adaptive large language model (LLM with Adaptor) 33; wherein the vector encoder 31 receives input data, including other vehicle information (Vehicle tokens), autonomous driving vehicle information (Ego Vehicle tokens), route information (Route tokens) and pedestrian information (Pedestrian tokens); and processes these different tokens through a multi-layer perceptron (MLP) to generate vector representations, and then integrates these vector representations through a cross attention mechanism (Cross Att) to form a comprehensive understanding of the entire scene.

[0085] The vectorizer 32 performs scene enhancement understanding according to the scene understanding result output by the vector encoder 31 through the self-attention mechanism and the cross-attention mechanism. Among them, the self-attention mechanism helps the model understand the relationship within each element, while the cross-attention mechanism helps the model understand the relationship between different elements.

[0086] The adaptive large language model 33 receives the processed vector representation and further processes it through multiple layers of neural networks. Each of the multiple layers includes components such as layer normalization, feedforward, and attention mechanism. The attention mechanism processes information through query (Q), key (K), and value (V), ensuring that the model can effectively capture the key information in the scene.

[0087] Finally, the model maps the scene description data to its internal knowledge representation and establishes the association between the scene and the knowledge.

[0088] In this embodiment, the large language model receives scene description data from the autonomous driving system and, after multi-layer processing and attention mechanisms, forms a deep understanding of the scene; it then maps it to internal knowledge representations that can help the model understand and predict various situations in the current driving environment, thereby making reasonable decisions.

[0089] Step S303: Test the initial in-vehicle question-answering model using scenario description test data. When the test result does not meet the preset conditions, continue training until the test result meets the preset conditions, thereby obtaining the in-vehicle question-answering model.

[0090] After the large language model completes multiple rounds of training, the performance of the initial in-vehicle question-and-answer model can be tested based on the scenario description test data, and whether to continue training can be determined based on the test results.

[0091] For example, if the prediction accuracy of the scenario description test data is greater than the preset accuracy, the test result is considered to meet the preset conditions, and the initial in-vehicle question-answering model is used as the final in-vehicle question-answering model and applied to the in-vehicle question-answering task processing. If the prediction accuracy of the scenario description test data is less than or equal to the preset accuracy, the test result is considered to not meet the preset conditions, and the initial in-vehicle question-answering model is continued to be trained until the test result meets the preset conditions to obtain the in-vehicle question-answering model.

[0092] After the model training is completed, the model can be applied to perform in-vehicle question and answering to generate answers to user questions. The answers can be direct answers or suggestive answers, which are not limited in this embodiment.

[0093] Since the sample data used in this embodiment covers detailed information in various driving scenarios, including but not limited to road conditions, traffic rules, vehicle behaviors and behavioral patterns of other road users. By learning from these sample data, the model can gradually build up the ability to understand complex driving environments, thereby identifying and interpreting the key elements in the scene and their interrelationships. Therefore, when the scene description data collected in real time is input into the in-vehicle question and answer model, it can quickly mobilize its inherent knowledge structure and accurately interpret the current driving scene. Moreover, this scene understanding is not just a simple recognition of static information, but also involves the prediction of dynamic situations and suggestions for response strategies. Therefore, the model can generate question and answer responses that are both in line with the actual situation and instructive, providing timely and effective assistance to the driver. The specific implementation process of in-vehicle question and answer is introduced below in conjunction with the accompanying drawings:

[0094] Figure 6 This is a schematic diagram of an in-vehicle question-and-answer scenario provided in an embodiment of the present application. Figure 6 As shown, the scene description data corresponding to the driving scene currently in which the vehicle is located include: I observed 4 vehicles and 3 pedestrians. The names of the 4 vehicles are A1, A2, A3 and A4; the vehicle angles of A1, A2, A3 and A4 are 91.7, 88.6, 88.2 and 91.4 respectively; the distances are 11 meters, 21 meters, 21 meters and 50 meters respectively; the speeds are 7km / h, 0Km / h, 0Km / h and 17Km / h respectively; the driving directions are: opposite to my direction, the same as my direction, the same as my direction and opposite to my direction respectively.

[0095] The names of the three pedestrians are B1, B2 and B3; the angles of B1, B2 and B3 are all 178.6; the distances are 7m, 12m and 19m respectively; the speeds are 15Km / h, 15Km / h and 0Km / h respectively; the movement directions are all from right to left.

[0096] There are currently no traffic lights.

[0097] My current speed is 0 km / h; the steering wheel is not straight.

[0098] Here are my actions: Gas pedal percentage: 0%; Brake pedal percentage: 100%; Steering percentage: 0%.

[0099] Through pre-learning and training, the large language model can quickly understand the above scenario description content and provide real-time answers based on user questions.

[0100] For example, if the user's question is: "How many cars do you see on the road?" the answer is: "I see 4 cars."

[0101] For example, if the user's question is "How many meters away is the nearest car from me?" the answer is "The nearest car is 11 meters away."

[0102] In order to further improve the question understanding and answer generation capabilities of the in-vehicle question-answering model, this embodiment can also comprehensively score the answers in the reply content and optimize the in-vehicle question-answering model based on the comprehensive score. Specifically, comprehensively scoring the answers in the reply content includes the following steps b1 and b2:

[0103] Step b1: Determine the score of the answer corresponding to the in-vehicle question and answer request from each scoring dimension.

[0104] In step b1, the answer corresponding to the in-vehicle question and answer request needs to be evaluated from various scoring dimensions, and each scoring dimension includes at least one of the accuracy dimension, the relevance dimension, and the timeliness dimension.

[0105] Specifically, based on direct user feedback (such as likes, dislikes, and ratings) or indirect behavioral signals (such as clicking on a link, continuing to ask questions or not), it can be determined whether the answer accurately solves the user's problem, and the answer can be scored based on the user's feedback data on the accuracy of the answer.

[0106] By evaluating whether the answer is closely related to the user's question, a score under the relevance dimension can be obtained.

[0107] Timeliness can be determined by the time interval between the time when the user asks the question and the time when the answer is generated. If the time interval is greater than the preset time, it is considered negative feedback data; conversely, if the time interval is less than the preset time, it is considered positive feedback data.

[0108] For each dimension, the score under that dimension can be determined according to the proportion of negative feedback data.

[0109] Step b2: Determine the comprehensive score of the answer based on the scores corresponding to each scoring dimension.

[0110] In step b2, the comprehensive score of the answer can be obtained by weighted summing up the scores of each dimension. This scoring mechanism can not only help quantify the quality of the answer, but also provide answer transparency, allowing users to understand the basis behind the score.

[0111] As more user interaction data accumulates, the model continuously learns and adapts to new patterns and trends, thereby continuously improving the accuracy and reliability of answer selection. In addition, in order to enhance the user experience, the scoring results can also be integrated into the reply content.

[0112] In some embodiments, in order to help users better understand the reliability and basis of answers, scores and explanations of scores may also be incorporated into the reply content to provide users with explanations about the reliability of answers.

[0113] For example, the user question is: "How many cars do you see on the road?" The answer is: "I see 4 cars." The score for this answer is 10 points, because the number of vehicles is correctly identified. Also, the user question is "How many meters is the nearest car to me?" The answer is "The nearest car is 11 meters." The score for this answer is 10 points, because the number of vehicles is correctly identified.

[0114] In order to more intuitively demonstrate how the in-vehicle question-and-answer method of this embodiment works in actual driving, the following is an explanation in combination with specific application scenarios:

[0115] For example, when a user experiences the end-to-end self-driving tour intelligent driving function, the user can say through the AI ​​voice assistant after getting in the car: "Take me to scenic spot A". The in-vehicle question-and-answer method of this embodiment quickly understands the user's intention through the in-vehicle question-and-answer model, and combines the pre-trained in-vehicle question-and-answer model and scene description data to generate the following response: "Okay, I have planned a route to scenic spot A for you. Now I will start the autonomous driving mode."

[0116] Next, during the autonomous driving process, the user can query the real-time traffic conditions, such as asking "How many cars do you see?" The in-vehicle question-answering method of this embodiment quickly understands the user's intention through the in-vehicle question-answering model, and combines the pre-trained in-vehicle question-answering model and the scene description data to generate the following response: "There are 3 cars on the current road."

[0117] When the user wants to confirm the driving strategy, he can ask: "Will you turn or go straight next?" The in-vehicle question-answering method of this embodiment quickly understands the user's intention through the in-vehicle question-answering model, and combines the pre-trained in-vehicle question-answering model and the scene description data to generate the following response: "I will go straight at the next intersection."

[0118] When a user wants to check the traffic signal at an intersection, he can ask: "Is there a traffic light at the intersection ahead?" The in-vehicle question-answering method of this embodiment quickly understands the user's intention through the in-vehicle question-answering model, and combines the pre-trained in-vehicle question-answering model and the scene description data to generate the following reply: "There is a traffic light at the intersection ahead."

[0119] When the user wants to change the navigation route, he can ask: "How can I get home quickly?" The in-vehicle question-answering method of this embodiment quickly understands the user's intention through the in-vehicle question-answering model, and combines the pre-trained in-vehicle question-answering model and the scene description data to generate the following reply: "I suggest you continue along the current road, then turn right at the next intersection, and go straight for about 2 kilometers to reach your home."

[0120] This embodiment processes the sensor data of the vehicle itself and its surrounding environment, standardizes it into scene description data, and uses pre-trained scene understanding capabilities to achieve accurate analysis of complex driving environments. Based on this, the model can generate diverse and accurate answers to user questions and provide more accurate question-and-answer services. The following is an introduction to the in-vehicle question-and-answer method of this embodiment from the functional side:

[0121] Figure 7 This is a schematic diagram of the functions implemented by the vehicle question-and-answer method provided in the embodiment of the present application. Figure 7 As shown, the function implemented by the in-vehicle question-answering method can be implemented by an in-vehicle question-answering system, and the in-vehicle question-answering system includes:

[0122] (1) Environmental understanding ability

[0123] Utilizing the advanced sensors and data processing capabilities of the autonomous driving system, real-time, all-round perception of the environment around the vehicle is achieved. It can not only identify static elements (such as roads and traffic signs), but also understand dynamic traffic conditions and provide detailed background information for questions and answers.

[0124] (2) Large language model understanding

[0125] Through the advanced semantic understanding capabilities of the large language model, it is possible to parse questions raised by drivers in natural language through question parsing and semantic analysis, and can effectively handle colloquial, elliptical sentences, or expressions with local characteristics.

[0126] (3) Answer Generation

[0127] Generate customized answers based on the driver's specific questions, context and vehicle conditions to ensure the effectiveness and comprehensibility of information transmission.

[0128] (4) Score calculation and score interpretation

[0129] By introducing a scoring mechanism, the relevance, accuracy and timeliness of each answer are quantitatively evaluated, and a scoring explanation is provided to help users better understand the scoring basis of the answer and increase the transparency of the system. At the same time, the model can be continuously optimized based on user feedback on the scoring explanation, thereby enhancing users' understanding of the scoring explanation.

[0130] In this embodiment, combined with the environmental understanding ability and large language model of the autonomous driving system, a deep understanding and personalized answers to the driver's questions can be achieved. And the system can feedback the answers in real time to ensure that the driver gets the required information at the critical moment. In addition, the knowledge base is dynamically updated as the environment changes and data accumulates to maintain the accuracy and timeliness of the question-and-answer service. Because it is based on powerful environmental understanding capabilities and language models, it can adapt to various driving scenarios (including urban roads, highways, country roads, etc.) and user needs, and has a wider range of applicability. In addition, user satisfaction can be improved by learning the driver's interaction habits and continuously optimizing the interaction experience.

[0131] The in-vehicle intelligent question-answering method of this embodiment can be applied to various driving task question-answering scenarios in autonomous driving vehicles or vehicles equipped with a driving assistance system.

[0132] For example, during daily driving, when the driver may have questions about the surrounding environment, he may ask: "What is the traffic condition ahead?" or "Do I need to take a detour?" This embodiment can analyze the road conditions ahead in real time, provide a detailed description of the traffic conditions, and suggest whether the route needs to be changed, thereby realizing daily driving assistance.

[0133] When the driver wants to know the next planning and decision of the autonomous driving system, he can ask questions such as "Are you turning left or going straight ahead?" or "Why do you choose the middle lane?" This embodiment can explain the reasoning process and provide a detailed description of the internal planning and decision-making of the system.

[0134] When the driver is unsure how to interpret certain traffic signs, he can ask: "What does this stop sign mean?" This embodiment can explain the meaning of traffic signs and provide corresponding driving suggestions.

[0135] In driving teaching and training scenarios, the autonomous driving system can act as a driving coach to answer students' questions, such as "Why do I need to use high beams in this situation?" The system not only provides answers, but also gives scores and explanations to help students understand driving rules.

[0136] When the system detects a potential hazard and controls the vehicle to take action, the user can ask: "Why does the system recommend that I slow down?" The system can explain the detected dangerous situation, such as pedestrians crossing the road ahead or slippery roads, and provide score explanations, emphasizing the importance of safe driving and providing driving safety reminders.

[0137] When a driver asks, “What’s the fastest route?” or “Are there any recommended restaurants along the way?” the system can provide optimal route suggestions and information about points of interest along the way based on real-time traffic information and user preferences.

[0138] In an emergency situation, such as a vehicle breakdown or accident, the driver may ask, “What should I do?” The system can provide guidance on what to do in an emergency and explain the reasons for each step.

[0139] In rainy and snowy weather, the driver may ask, "What is the current road friction coefficient?" The system can provide road conditions based on sensor data and give corresponding driving suggestions.

[0140] When the driver may have questions about certain traffic regulations, he can ask: "Can I park temporarily here?" The system will give an answer as to whether parking is allowed based on the current geographical location and traffic regulations.

[0141] The intelligent in-vehicle question-answering method of this embodiment can also be applied to various fields. For example, in an urban traffic environment, the system collects traffic information in real time through on-vehicle sensors, and the driver can ask questions through voice or touch screen, such as "Is there a traffic accident ahead?" The system will analyze the collected data, generate answers through a large language model, and provide an estimated time to arrive at the destination and possible traffic conditions. The scoring system will score based on the accuracy of the real-time traffic conditions and the relevance of the answers.

[0142] In the field of public transportation, such as buses or taxis, this embodiment can receive inquiries from drivers or passengers through the vehicle terminal. For example, the driver asks "What transfer lines are there at the next stop?" The system will combine real-time bus information and language models to provide accurate transfer information. In operation, the driver inputs the question through voice, and the system responds quickly and displays it on the vehicle screen.

[0143] In the field of long-distance driving, drivers may ask questions about route planning, rest stop locations, or weather forecasts. The system uses GPS and weather data, combined with the understanding ability of a large language model, to provide detailed route guidance and weather conditions. The scoring system will score based on the comprehensiveness and usefulness of the information provided.

[0144] During the testing phase of an autonomous vehicle, the test engineer may ask questions about the vehicle's behavior, such as "Why did the vehicle slow down at this point?" The system will use the large language model to explain the vehicle's behavior logic based on the vehicle's sensor data and the preset test scenario. The scoring system will score based on the accuracy and logic of the explanation.

[0145] In the field of driving training, students can ask the system questions about driving skills and rules. The system will provide customized guidance and explanations based on the student's learning progress and driving behavior through a large language model. The scoring system will score based on the effectiveness of the guidance and the student's acceptance.

[0146] For special vehicles such as ambulances and fire trucks, drivers may need to quickly understand the best rescue route or operation guide. The system will prioritize these urgent issues and provide instant route planning and operation suggestions through a large language model. The scoring system will score based on the response speed and accuracy of the answer.

[0147] In a car-sharing service, users may ask questions about the vehicle status, remaining battery, or the location of charging stations. The system uses the vehicle's diagnostic data and the understanding capabilities of a large language model to provide vehicle status information and charging suggestions. The scoring system will score based on the accuracy of the information and user satisfaction.

[0148] Through the above-mentioned rich and diverse application scenarios, the vehicle-mounted intelligent question-answering method of this embodiment not only improves driving safety and efficiency, but also improves user experience, and is applicable to various driving task question-answering scenarios. The realization of the above functions requires the support of the vehicle's hardware architecture. Based on this, this embodiment also proposes an integrated vehicle hardware architecture, such as Figure 8 As shown, the architecture includes a vehicle platform, a storage device, a communication device, and a human-computer interaction device.

[0149] Among them, the vehicle platform includes an autonomous driving vehicle, a sensor component, an autonomous driving computing unit and a large language model computing unit.

[0150] Vehicles with autonomous driving functions include but are not limited to electric vehicles, hybrid vehicles or traditional fuel vehicles.

[0151] Sensor components include cameras, radars (such as millimeter-wave radars), laser radars (LIDAR), ultrasonic sensors, and infrared sensors, which are used to perceive the surrounding environment in real time.

[0152] The autonomous driving computing unit is a high-performance computing platform that runs deep learning models and executes decision-making algorithms to process sensor data and build environmental models.

[0153] The large language model computing unit installed on the vehicle has powerful computing resources for language model training and real-time question-answering services. In the calculation process of the large language model, you can try to use more advanced computing units, such as neural network processors (NPUs) and graphics processing units (GPUs), to improve computing efficiency.

[0154] The storage device may be a high-speed solid-state drive (SSD) for storing a large amount of training data and model parameters, and storing a large language model and related knowledge base.

[0155] The communication equipment includes the vehicle communication module and the satellite navigation system; the vehicle communication module can use a high-speed data transmission network to support the communication between the ECU and the sensor in the vehicle, such as CAN, Ethernet, WiFi, etc., for exchanging data with the outside world and receiving updates. The satellite navigation system can use the Global Positioning System (GPS) and the Global Navigation Satellite System (GLONASS) for vehicle positioning and navigation.

[0156] The human-computer interaction device includes a voice recognition module, a display screen, an audio output, and a personal device; the voice recognition module can use a high-performance microphone array and a voice recognition chip to capture and process the driver's voice commands. The display screen can use a high-definition touch screen or a dashboard to display text, images, and navigation information to the driver. The audio output can use an in-vehicle speaker system to play voice prompts and information generated by the system. The personal device can be a smartphone or other personal device that is connected to the system through an application.

[0157] The control execution device may be an electronic control unit (ECU), which is used to control various electronic systems of the vehicle, such as acceleration, braking, steering, etc.

[0158] The power management system includes a battery management system and a power converter. The battery management system is used to manage the charging and discharging process of the battery in electric vehicles to ensure the stability and efficiency of the power supply. The power converter is used to convert the power provided by the battery or generator into a voltage and current suitable for the use of on-board electronic equipment.

[0159] The in-vehicle intelligent question-answering method of this embodiment not only relies on the background processing unit to parse and generate answers, but also requires an efficient and intuitive user interface (UI) as front-end support. The interface is responsible for receiving the user's in-vehicle question-answering requests and ensuring that these requests can be accurately transmitted to the background processing unit for processing. The front-end data processing flow is as follows: Fig. 9 As shown, it includes the following steps c1 to c5:

[0160] Step c1: User input

[0161] Users can initiate in-car question-and-answer requests in a variety of ways, including voice input and text input. For voice input, the system converts the user's voice questions into text form through high-precision voice recognition technology. In addition, users can also directly enter questions in text form through the touch screen, keyboard or other input devices.

[0162] Step c2: Problem analysis

[0163] The front-end parser uses natural language processing (NLP) technology to parse the questions input by users, including word segmentation, part-of-speech tagging, named entity recognition, etc., to understand the content and structure of the questions. After that, the intent recognition algorithm is used to determine the user's intention of asking questions, such as obtaining information, querying routes, seeking advice, etc.

[0164] Step c3, interactive interface

[0165] In order to enhance the user experience, an intuitive and versatile user interface (UI) can be provided to allow users to interact with the system in a variety of ways, including touch, voice commands, etc. At the same time, the interface design needs to focus on real-time feedback mechanisms, such as problem confirmation, operation progress prompts, etc., so that users can understand the response status of the system at any time and increase the transparency and smoothness of the interaction process.

[0166] Step c4: Information transmission

[0167] The parsed user questions and related context information are packaged and passed to the back-end processing unit through the defined data interface, thus ensuring the accuracy, completeness and real-time nature of question transmission.

[0168] Step c5: Error handling

[0169] The system also has a built-in exception handling mechanism that can identify and handle various input errors or system failures. When an abnormal situation is detected, the system will display an error message to the user and provide possible solutions to ensure that the user experience is not affected.

[0170] Through efficient speech recognition and natural language processing technology, as well as intuitive user interface design, the front-end architecture in this embodiment ensures that users can interact with the system easily and conveniently. More importantly, the close collaboration between the front-end and the back-end not only ensures the accurate transmission of questions, but also achieves efficient response of the system, providing users with an intelligent and personalized driving assistance experience.

[0171] To sum up, this embodiment significantly improves the performance and user experience of the in-vehicle intelligent question-and-answer system by optimizing the collaborative process between the front-end and the back-end and combining advanced technology with a user-friendly interface design, and is suitable for various complex driving scenarios.

[0172] The above embodiment introduces the technical solution for question-answering task processing in the autonomous driving scenario. In addition, the vehicle-mounted question-answering method of this embodiment can also be applied to the following fields:

[0173] (1) Smart home field, such as being applied to smart home systems (smart speakers and smart lights, etc.) and helping users control home devices through voice interaction.

[0174] (2) Intelligent assistants, such as those used on smartphones, tablet computers and other devices, help users complete daily tasks such as scheduling and information query.

[0175] (3) Industrial automation: For example, it is applied to industrial robots to improve production efficiency through voice interaction to guide operations.

[0176] Exemplary Devices

[0177] Corresponding to the above-mentioned in-vehicle question and answer method, an embodiment of the present application also provides an in-vehicle question and answer device. Fig.10 Schematic diagram of the structure of a vehicle-mounted question-answering device provided in an embodiment of the present application. Fig.10 As shown, the in-vehicle question and answer device provided in the embodiment of the present application includes: an acquisition unit 901 and an in-vehicle question and answer unit 902; wherein the acquisition unit 901 is used to respond to an in-vehicle question and answer request and obtain scene description data of the vehicle in the current driving scene; the in-vehicle question and answer unit 902 is used to perform scene understanding on the scene description data through a pre-trained in-vehicle question and answer model, and generate reply content for the in-vehicle question and answer request according to the scene understanding result; wherein the in-vehicle question and answer model is obtained by training a large language model for a scene understanding task based on scene description sample data.

[0178] In some embodiments, the scene description data is determined using the following steps: determining environmental understanding data based on sensor data of the vehicle, wherein the sensor data of the vehicle includes data characterizing the vehicle state and the environment surrounding the vehicle; and converting the environmental understanding data into scene description data in a preset format.

[0179] In some embodiments, determining the environmental understanding data based on the sensor data of the vehicle includes: constructing an environmental model of the vehicle's current environment based on the sensor data to obtain the environmental understanding data, wherein the environmental model includes various static elements and various dynamic elements, and each of the static elements and each of the dynamic elements has corresponding status information.

[0180] In some embodiments, the scene description data in the preset format includes: vehicle positioning data, interaction relationships between the vehicle and other vehicles or pedestrians, behavior prediction results of other vehicles or pedestrians, path planning information, navigation assistance information and at least one of vehicle status parameters.

[0181] In some embodiments, the in-vehicle question and answer model is trained using the following steps: obtaining scene description sample data, the scene description sample data including scene description training data and scene description test data; inputting the scene description training data into a large language model so that the large language model maps the scene description training data into a knowledge representation within the large language model to obtain an initial in-vehicle question and answer model; testing the initial in-vehicle question and answer model using the scene description test data, and when the test result does not meet the preset conditions, continuing the training until the test result meets the preset conditions to obtain the in-vehicle question and answer model.

[0182] In some embodiments, the scene description sample data is obtained by performing environmental modeling based on data representing the vehicle state and the surrounding environment and converting the data into a preset format.

[0183] In some embodiments, the reply content includes the answer corresponding to the in-vehicle question and answer request, as well as the comprehensive score of the answer and the score explanation content; the device also includes: an optimization unit 903, which is used to optimize the in-vehicle question and answer model according to the comprehensive score of the answer corresponding to the in-vehicle question and answer request.

[0184] In some embodiments, the comprehensive score of the answer is determined by the following steps: determining the score of the answer corresponding to the in-vehicle question and answer request from each scoring dimension, wherein each scoring dimension includes at least one of the accuracy dimension, the relevance dimension and the timeliness dimension; determining the comprehensive score of the answer based on the scores corresponding to each scoring dimension.

[0185] The vehicle-mounted question-and-answer device provided in this embodiment belongs to the same application concept as the vehicle-mounted question-and-answer method provided in the above embodiments of this application, and can execute the vehicle-mounted question-and-answer method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the vehicle-mounted question-and-answer method. For technical details not fully described in this embodiment, please refer to the specific processing content of the vehicle-mounted question-and-answer method provided in the above embodiments of this application, and will not be repeated here.

[0186] The functions implemented by the above acquisition unit 901, the in-vehicle question and answer unit 902 and the optimization unit 903 can be implemented by the same or different processors respectively, and the embodiment of the present application is not limited thereto.

[0187] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0188] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0189] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0190] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0191] Exemplary Electronic Devices

[0192] The present application embodiment provides an electronic device, see Fig.11 As shown, the electronic device includes:

[0193] Memory 200 and processor 210;

[0194] The memory 200 is connected to the processor 210 and is used to store programs;

[0195] The processor 210 is used to implement the in-vehicle question-and-answer method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0196] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0197] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0198] A bus may include a pathway that transfers information between components of a computer system.

[0199] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0200] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0201] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0202] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0203] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0204] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0205] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any in-vehicle question-and-answer method provided in the above embodiments of the present application.

[0206] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the in-vehicle question and answer method introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the introduction to the embodiment of the in-vehicle question and answer method.

[0207] Exemplary computer program products and storage media

[0208] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the in-vehicle question-and-answer method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0209] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0210] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored. The computer program is executed by a processor to execute the steps of the in-vehicle question and answer method according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically the steps of the above in-vehicle question and answer method can be implemented.

[0211] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0212] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0213] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0214] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0215] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0216] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0217] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0218] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0219] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0220] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0221] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vehicle-mounted question-answering method, characterized in that: include: Responding to an in-vehicle question-and-answer request, obtaining scene description data of the vehicle in the current driving scene; Performing scene understanding on the scene description data through a pre-trained in-vehicle question-and-answer model, and generating a reply content for the in-vehicle question-and-answer request according to the scene understanding result; Among them, the in-vehicle question and answer model is obtained by training a large language model for scene understanding tasks based on scene description sample data.

2. The method according to claim 1, characterized in that The scene description data is determined by the following steps: determining environmental understanding data based on sensor data of the vehicle, the sensor data of the vehicle including data representing a state of the vehicle and an environment surrounding the vehicle; The environmental understanding data is converted into scene description data in a preset format.

3. The method according to claim 2, characterized in that The determining of environmental understanding data based on sensor data of the vehicle comprises: An environmental model of the vehicle's current environment is constructed based on the sensor data to obtain the environmental understanding data, wherein the environmental model includes various static elements and various dynamic elements, and each of the static elements and each of the dynamic elements has corresponding status information.

4. The method according to claim 2, characterized in that: The scene description data in the preset format includes: vehicle positioning data, interaction relationships between the vehicle and other vehicles or pedestrians, behavior prediction results of other vehicles or pedestrians, path planning information, navigation assistance information and at least one of vehicle status parameters.

5. The method according to any one of claims 1 to 4, characterized in that The in-vehicle question-answering model is trained using the following steps: Acquire scene description sample data, wherein the scene description sample data includes scene description training data and scene description test data; Inputting the scene description training data into a large language model so that the large language model maps the scene description training data into a knowledge representation within the large language model to obtain an initial in-vehicle question-answering model; The initial in-vehicle question-answering model is tested using the scenario description test data. When the test result does not meet the preset conditions, the training is continued until the test result meets the preset conditions, thereby obtaining the in-vehicle question-answering model.

6. The method according to claim 5, characterized in that The scene description sample data is obtained by performing environment modeling based on data representing the vehicle state and the surrounding environment and converting the data into a preset format.

7. The method according to any one of claims 1 to 4, characterized in that The reply content includes the answer corresponding to the in-vehicle question and answer request, as well as the comprehensive score of the answer and the score explanation content; Wherein, after generating a reply content for the in-vehicle question and answer request according to the scene understanding result, the method further includes: The in-vehicle question and answer model is optimized according to the comprehensive score of the answer corresponding to the in-vehicle question and answer request.

8. The method according to claim 7, characterized in that The overall score of the answers is determined using the following steps: Determine the score of the answer corresponding to the in-vehicle question and answer request from each scoring dimension, wherein each scoring dimension includes at least one of an accuracy dimension, a relevance dimension, and a timeliness dimension; The comprehensive score of the answer is determined according to the scores corresponding to the respective scoring dimensions.

9. A vehicle-mounted question-and-answer device, characterized in that: include: An acquisition unit, configured to respond to an in-vehicle question-answering request and acquire scene description data of the vehicle in a current driving scene; An in-vehicle question-and-answer unit, configured to perform scene understanding on the scene description data through a pre-trained in-vehicle question-and-answer model, and generate a reply content for the in-vehicle question-and-answer request according to the scene understanding result; Among them, the in-vehicle question and answer model is obtained by training a large language model for scene understanding tasks based on scene description sample data.

10. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 8 by running the program in the memory.

11. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, cause the processor to implement the method as claimed in any one of claims 1 to 8.

Images