Intelligent cabin counterfactual sample generation method, intelligent cabin interaction method, device, system and vehicle

Abstract

Embodiments of the present application provide a kind of intelligent cockpit counterfactual sample generation method, intelligent cockpit interaction method, equipment, system and vehicle, comprising: obtaining at least one original cockpit interaction instruction;Determine at least one influence condition that the execution result of original cockpit interaction instruction is influenced;Under the condition that original cockpit interaction instruction remains unchanged, at least one condition in influence condition is adjusted, and at least one counterfactual state is generated;For the execution scene of original cockpit interaction instruction under counterfactual state, the corresponding processing result of intelligent cockpit system is determined;Original cockpit interaction instruction, counterfactual state and corresponding processing result are associated to form a counterfactual sample data.Can generate the sample data of different processing results corresponding to the same cockpit interaction instruction under different vehicle states, to improve the learning ability of intelligent cockpit model to vehicle state constraint and safety strategy;And using automatic generation mode, to reduce the cost of manual annotation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for generating counterfactual samples for an intelligent cockpit, an intelligent cockpit interaction method, equipment, system, and vehicle. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs), vision-language models (IVLMs), and multimodal large models (Omni) have been widely applied in the fields of intelligent driving and smart cockpits. To improve the generalization ability of large models in specific vertical domains, supervised fine-tuning (SFT) technology has become a key method, and the effectiveness of this technology highly depends on large-scale, high-quality data support. In existing smart cockpit training data construction schemes, training samples are typically only labeled with "what the user said" and "what the corresponding intent is." For example, if the user says "open the trunk," the action of opening the trunk is invoked. The model only needs to learn the mapping to the operation of "performing the action of opening the trunk."

[0003] However, intelligent cockpit scenarios are highly dynamic and safety-sensitive. The processing logic for the same voice command should differ significantly depending on vehicle speed, weather conditions, occupant status, and vehicle configuration. Current technologies lack training samples for this: the same sentence should be processed differently under different vehicle conditions. For example: User: Open the trunk; Vehicle stationary: Can open; Vehicle traveling at high speed: Cannot open; Vehicle model does not support power tailgate: Prompt "Not supported". The lack of corresponding training samples can lead to unexpected responses from the cockpit agent, even affecting driving safety. Currently, the few solutions that recognize this problem mainly rely on manual construction by human experts. Manually constructing such data is time-consuming, labor-intensive, and prone to oversights. Summary of the Invention

[0004] This invention provides a method for generating counterfactual samples for intelligent cockpits, an intelligent cockpit interaction method, a device, a system, and a vehicle. It enables the generation of sample data that corresponds to different processing results for the same cockpit interaction command under different vehicle states, thereby improving the intelligent cockpit model's ability to learn vehicle state constraints and safety strategies, ensuring that the processing results output by the model meet expectations, and improving driving safety. Furthermore, it adopts an automated generation method, thereby reducing the cost of manual annotation.

[0005] Firstly, this embodiment provides a method for generating counterfactual samples for intelligent cockpits, the method comprising: Obtain at least one original cockpit interaction command; Determine at least one influencing condition that affects the execution result of the original cockpit interaction command; While keeping the original cockpit interaction commands unchanged, at least one of the influencing conditions is adjusted to generate at least one counterfactual state; For the execution scenario of the original cockpit interaction command in the counterfactual state, determine the corresponding processing result of the intelligent cockpit system; The original cockpit interaction command, the counterfactual state, and the corresponding processing result are associated to form a counterfactual sample data.

[0006] Secondly, this embodiment provides an intelligent cockpit interaction method, which includes: Obtain the cockpit interaction commands input by the user and the current counterfactual state of the vehicle corresponding to the cockpit interaction commands; The cockpit interaction commands and the current counterfactual state are input into a pre-trained intelligent cockpit model to obtain the processing result output by the intelligent cockpit model. The intelligent cockpit model is trained using counterfactual sample data generated by the intelligent cockpit counterfactual sample generation method as described in the first aspect embodiment. Based on the processing result, the vehicle is controlled to perform corresponding cockpit operations, or corresponding prompts are output to the user.

[0007] Thirdly, this embodiment provides an electronic device, which includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the smart cockpit counterfactual sample generation method according to any embodiment of the present invention.

[0008] Fourthly, this embodiment provides an intelligent cockpit system, which includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the intelligent cockpit interaction method according to any embodiment of the present invention.

[0009] Fifthly, this embodiment provides a vehicle including the intelligent cockpit system as described in the fourth aspect embodiment.

[0010] This invention provides a method for generating counterfactual samples for an intelligent cockpit, an intelligent cockpit interaction method, a device, a system, and a vehicle. The method includes: acquiring at least one original cockpit interaction command; determining at least one influencing condition that affects the execution result of the original cockpit interaction command; adjusting at least one of the influencing conditions while keeping the original cockpit interaction command unchanged to generate at least one counterfactual state; determining the corresponding processing result of the intelligent cockpit system for the execution scenario of the original cockpit interaction command in the counterfactual state; and associating the original cockpit interaction command, the counterfactual state, and the corresponding processing result to form a counterfactual sample data. This technical solution can generate sample data showing different processing results for the same cockpit interaction command under different vehicle states. Based on the generated sample data, the intelligent cockpit model can be supervised and fine-tuned, thereby improving the intelligent cockpit model's learning ability for vehicle state constraints and safety strategies, ensuring that the model's output processing results meet expectations, and improving driving safety. Furthermore, the method uses automated generation to generate samples, thereby reducing manual annotation costs.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating a method for generating counterfactual samples in an intelligent cockpit according to Embodiment 1 of the present invention. Figure 2 This is a flowchart illustrating another method for generating counterfactual samples in a smart cockpit, as provided in Embodiment 2 of the present invention. Figure 3 This is a flowchart illustrating an intelligent cockpit interaction method provided in Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention; Figure 5 This is a structural schematic diagram of an intelligent cockpit system provided in Embodiment 5 of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] It's important to note that existing smart cockpit models typically only label the training data with "what the user said" and "what the corresponding intent was," lacking data that addresses the following: the same instruction should be processed differently under different vehicle conditions—that is, counterfactual data: the same command + different cockpit states + different processing results = counterfactual data. For example, a user command might be: "Open the trunk." If the vehicle is stationary: it can be opened; if the vehicle is traveling at high speed: it cannot be opened; if the vehicle model does not support a power tailgate: a message indicating it is not supported. Without corresponding training data, the model could easily learn to invoke the "open trunk" action whenever the user says "open the trunk." This is the risk factor of smart cockpit agents. Existing solutions either construct only the necessary data for ordinary cockpit agents (e.g., turning on the air conditioning) using a large model, lacking this type of counterfactual training data, or they employ a method of manually constructing counterfactual data. The lack of corresponding training data can lead to unexpected responses from the cockpit agent, even affecting driving safety. Manually constructing such data is time-consuming, labor-intensive, and prone to oversight. Therefore, a method is needed to solve these problems.

[0017] Example 1 Figure 1This is a flowchart illustrating a method for generating counterfactual samples for a smart cockpit according to Embodiment 1 of the present invention. This method is applicable to the generation of counterfactual samples for a smart cockpit. The method can be executed by a smart cockpit counterfactual sample generation device, which can be implemented in hardware and / or software and is generally integrated into an electronic device.

[0018] like Figure 1 As shown, the method for generating counterfactual samples for a smart cockpit provided in this embodiment may specifically include the following steps: S101. Obtain at least one original cockpit interaction command.

[0019] In this embodiment, the generated counterfactual samples of the smart cockpit can be used to train, test, fine-tune, and evaluate the smart cockpit model and the smart agent.

[0020] Specifically, the original cockpit interaction commands can be understood as the interaction commands input by the user to the smart cockpit to control the vehicle to perform corresponding operations. These commands can be presented in the form of voice interaction commands, text input commands, etc. This step is used to obtain multiple original cockpit interaction commands. In this embodiment, the original cockpit interaction commands can be obtained by pulling historical user cockpit interaction data, or by setting prompts in a large model to generate corresponding cockpit interaction commands. Other methods for obtaining original cockpit interaction commands can also be used; no specific limitations are made here. For example, original cockpit interaction commands could be: open the rear window, open the trunk, turn on the seat heater, defog the car, etc.

[0021] S102. Determine at least one influencing condition that affects the execution result of the original cockpit interaction command.

[0022] Unlike existing smart cockpit models where training data typically only labels "what the user said" and "what the corresponding intent is," this embodiment considers that the same original cockpit interaction command should be processed differently under different vehicle conditions. Based on this, counterfactual data can be constructed: the same command + different cockpit states + different processing results = counterfactual data.

[0023] To construct counterfactual data, it is first necessary to identify the state conditions that affect the processing results of cockpit commands, denoted as cockpit-related states. Then, it is necessary to determine which cockpit-related states will affect the execution result of the original cockpit interaction command, and these influencing cockpit-related states are recorded as influencing conditions. The states in the vehicle that affect the execution result of the original cockpit interaction command can generally be categorized as follows: vehicle state, environmental state, occupant state, vehicle model capabilities, and safety rules. Vehicle state includes: vehicle speed, gear position, doors, windows, etc.; environmental state includes: rain, fog, high temperature, low temperature, etc.; occupant state includes: driver, children, rear passengers; vehicle model capabilities include: whether a certain function is supported; and safety rules include: whether direct execution is permitted. For example, for "open the rear window," the key influencing conditions are: vehicle speed, child lock status, weather, and whether the window function is available.

[0024] In this embodiment, prompts can be set using a large model to determine whether the above-mentioned cockpit-related states will have different effects on the original command generated in the previous step. Alternatively, a more complex intelligent agent scheme can be used to determine whether the above-mentioned vehicle states will have different effects on the original command generated in the previous step. Or, manual judgment can be used to determine whether the above-mentioned vehicle states will have different effects on the original command generated in the previous step. Any other solution that can achieve the purpose can be used, and no specific limitations are imposed here.

[0025] S103. While keeping the original cockpit interaction commands unchanged, adjust at least one of the influencing conditions to generate at least one counterfactual state.

[0026] Specifically, a counterfactual state can be understood as a vehicle state constructed by setting one or more different influencing conditions, and counterfactual sample data can be understood as sample data consisting of different processing results corresponding to the same original cockpit interaction command and different counterfactual states.

[0027] This step is used to generate a counterfactual state by changing one or more influencing conditions while keeping the original cockpit interaction commands unchanged. In this embodiment, a large model can be used to set prompts and change one or more influencing conditions to generate a counterfactual state. A more complex intelligent agent scheme can also be used to change one or more influencing conditions to generate a counterfactual state. Alternatively, manual judgment can be used to change one or more influencing conditions to generate a counterfactual state. Any other solution that can achieve the purpose can be used; no specific limitations are imposed here.

[0028] For example, the original cockpit interaction command is: open the rear window. The generated counterfactual states may include: State A: parked, child locks off; State B: driving at high speed, child locks off; State C: parked, child locks on; State D: driving in the rain, child locks off.

[0029] S104. For the execution scenario of the original cockpit interaction command in the counterfactual state, determine the corresponding processing result of the intelligent cockpit system.

[0030] In this embodiment, for the execution scenario of the original cockpit interaction commands under the counterfactual state, the intelligent cockpit system is re-determined on how to handle them, and the processing result is regenerated based on the counterfactual state to form comparative sample data. Similarly, prompt words can be set using a large model to determine the corresponding execution or response result. A more complex intelligent agent scheme can also be used to determine the corresponding execution or response result. Alternatively, manual judgment can be used to determine the corresponding execution or response result; any other solution that achieves the purpose can be used, without specific limitations.

[0031] Preferably, the processing result includes at least one of the following: direct execution, inquiry for confirmation, refusal to execute, prompt of unsupported, and prompt of restricted. For example, assuming the original cockpit interaction command is: open the rear window, if the current state is parking + child lock off: the processing result is direct opening; if the current state is high-speed driving + child lock off: the processing result is inquiry; if the current state is confirmed parking + child lock on: the processing result is prompt of restricted; if the current state is vehicle model not supported: the processing result is prompt of unsupported.

[0032] S105. Associate the original cockpit interaction command, the counterfactual state, and the corresponding processing result to form a counterfactual sample data.

[0033] In this embodiment, after determining the counterfactual state and corresponding processing result for each original cockpit interaction command, the same original cockpit interaction command and the processing results under multiple different counterfactual states are combined to form sample data, which is denoted as counterfactual sample data.

[0034] Preferably, the counterfactual sample data is in the format of structured data and includes at least the following fields: cockpit interaction command field, counterfactual status field, and processing result label field.

[0035] For example, the output format of counterfactual sample data can be: user instructions, counterfactual states, processing result labels, etc. Continuing with the example above, where `user` represents the user command, `state` represents the counterfactual state, and `result` represents the processing result, the counterfactual sample data corresponding to the above example can be represented as: [ { "user": "Open rear window", "state": "Park, child lock off", "result": "Execute directly"}, { "user": "Open rear window", "state": "High-speed driving, child lock off", "result": "Ask for confirmation"}, { "user": "Open rear window", "state": "Park, child lock on", "result": "Prompt for restriction"}] The above technical solution can generate sample data of different processing results corresponding to the same cockpit interaction command under different vehicle states. Based on the sample data generated above, the intelligent cockpit model is supervised and fine-tuned, thereby improving the intelligent cockpit model's ability to learn vehicle state constraints and safety strategies, ensuring that the processing results output by the model meet expectations, and improving driving safety. Furthermore, the sample is generated automatically, thereby reducing the cost of manual annotation.

[0036] As an optional embodiment of the present invention, based on the above embodiments, the optional embodiment can be optimized so that when the processing result is the inquiry confirmation, the method further includes: determining and outputting the reply text corresponding to the inquiry confirmation.

[0037] When the processing result is confirmation of the inquiry, the counterfactual sample data also includes a response text field.

[0038] In this embodiment, when the processing result is a confirmation query, a corresponding response text can be generated and output using text or voice. Correspondingly, the format of the counterfactual sample data also includes a system response field.

[0039] The above technical solution adds a way to generate counterfactual sample data when the processing result is a confirmation inquiry.

[0040] Example 2 Figure 2 This is a flowchart illustrating another method for generating counterfactual samples in a smart cockpit according to Embodiment 2 of the present invention. This embodiment is a further optimization of the above embodiment. In this embodiment, the definition of "obtaining at least one original cockpit interaction command" and the definition of "determining at least one influencing condition that affects the execution result of the original cockpit interaction command" are further optimized.

[0041] like Figure 2 As shown in the figure, this embodiment two provides a method for generating counterfactual samples for intelligent cockpits, which specifically includes the following steps: S201. Obtain historical cockpit interaction data as the original cockpit interaction command; or, construct command generation prompts and input the command generation prompts into the basic large model, so that the basic large model generates cockpit interaction commands that conform to the cockpit interaction scenario based on the command generation prompts, and uses them as the original cockpit interaction commands.

[0042] In this embodiment, one way to generate the original cockpit interaction commands is to retrieve the user's historical cockpit interaction data and use the interaction commands in the historical cockpit interaction data as the original cockpit interaction commands. Another way to generate the original cockpit interaction commands is to generate corresponding cockpit interaction commands by setting prompt words in a large model, that is, to construct prompt words for command generation, and then input the prompt words for command generation into the basic large model. The basic large model generates cockpit interaction commands that conform to the cockpit interaction scenario based on the prompt words for command generation, and uses them as the original cockpit interaction commands.

[0043] S202. Using a preset processing strategy, at least one influencing condition that affects the execution result of the original cockpit interaction command is selected from the cockpit association status corresponding to the original cockpit interaction command.

[0044] Preferably, the cockpit-related status includes at least: vehicle status, environmental status, occupant status, vehicle type capability, and safety rules.

[0045] To construct counterfactual data, it is first necessary to identify the state conditions that affect the processing results of cockpit commands, denoted as cockpit-related states. Then, it is necessary to determine which cockpit-related states will affect the execution result of the original cockpit interaction command, and these influencing cockpit-related states are denoted as influencing conditions. Cockpit-related states that affect the execution result of the original cockpit interaction command can generally be categorized as follows: vehicle state, environmental state, occupant state, vehicle model capabilities, and safety rules. Vehicle state includes: vehicle speed, gear position, doors, windows, etc.; environmental state includes: rain, fog, high temperature, low temperature, etc.; occupant state includes: driver, children, rear passengers; vehicle model capabilities include: whether a certain function is supported; and safety rules include: whether direct execution is permitted. For example, for "open the rear window," the key influencing conditions are: vehicle speed, child lock status, weather, and whether the window function is available.

[0046] Preferably, the preset processing strategy includes one or more of the following: processing strategy based on artificial intelligence model, processing strategy based on rules, and processing strategy based on manual annotation.

[0047] Among these, the AI-based processing strategy can be understood as a method of filtering influencing conditions by setting prompts or using more complex intelligent agent schemes through large models. The rule-based processing strategy can be understood as a method of filtering influencing conditions by setting rules. The manual annotation-based processing strategy can be understood as a method of manually annotating whether the cockpit-related states are considered influencing conditions.

[0048] In this embodiment, prompts can be set using a large model to determine whether the associated states of each cockpit will have different effects on the original command generated in the previous step. Alternatively, a more complex intelligent agent scheme can be used to determine whether the associated states of the cockpit will have different effects on the original command generated in the previous step. Alternatively, rules can be set to determine whether the associated states of the cockpit will have different effects on the original command generated in the previous step, or a manual judgment can be used to determine whether the associated states of the cockpit will have different effects on the original command generated in the previous step. Any other solution that can achieve the purpose can be used, and no specific limitations are imposed here.

[0049] S203. While keeping the original cockpit interaction commands unchanged, adjust at least one of the influencing conditions to generate at least one counterfactual state.

[0050] S204. For the execution scenario of the original cockpit interaction command in the counterfactual state, determine the corresponding processing result of the intelligent cockpit system.

[0051] S205. Associate the original cockpit interaction command, the counterfactual state, and the corresponding processing result to form a counterfactual sample data.

[0052] The above technical solution can achieve automated, efficient, and large-scale generation of high-quality cockpit counterfactual sample data by pulling historical cockpit interaction commands or generating original cockpit interaction commands through a large model, and by using a preset strategy to determine the influencing conditions that affect the execution results of the original cockpit interaction commands.

[0053] Example 3 Figure 3 This is a flowchart illustrating an intelligent cockpit interaction method provided in Embodiment 3 of the present invention. This method is applicable to the generation of counterfactual samples in intelligent cockpits, and it is also applicable to the interaction of commands with the countermodel of intelligent cockpits. This method can be executed by an intelligent cockpit interaction device, which can be implemented in hardware and / or software and is generally integrated into the intelligent cockpit interaction system.

[0054] like Figure 3 As shown, the intelligent cockpit interaction method provided in this embodiment may specifically include the following steps: S301. Obtain the cockpit interaction command input by the user and the current counterfactual state of the vehicle corresponding to the cockpit interaction command.

[0055] In this embodiment, when a user wants to interact with the vehicle, they can input a cockpit interaction command. This step is used to obtain the cockpit interaction command input by the user and the current counterfactual state of the vehicle corresponding to the cockpit interaction command. The current counterfactual state can be obtained based on the current cockpit association state.

[0056] S302. Input the cockpit interaction command and the current counterfactual state into the pre-trained intelligent cockpit model to obtain the processing result output by the intelligent cockpit model.

[0057] The intelligent cockpit model is trained using counterfactual sample data generated by the intelligent cockpit counterfactual sample generation method as described in any embodiment of the present invention. The input to the intelligent cockpit model is the cockpit interaction commands and the current counterfactual state; the output is the processing result.

[0058] Specifically, the cockpit interaction commands and the current counterfactual state are input into a pre-trained intelligent cockpit model to obtain the processing results output by the intelligent cockpit model.

[0059] S303. Based on the processing result, control the vehicle to perform corresponding cockpit operations, or output corresponding prompt information to the user.

[0060] In this embodiment, the vehicle is controlled to perform corresponding cockpit operations or to output corresponding prompts to the user based on the processing results output by the intelligent cockpit model.

[0061] The above technical solution uses generated counterfactual sample data to supervise and fine-tune the intelligent cockpit model, thereby improving the intelligent cockpit model's ability to learn vehicle state constraints and safety strategies, making the processing results output by the intelligent cockpit model more in line with expectations, and improving driving safety.

[0062] Example 4 Figure 4 This is a schematic diagram of an electronic device provided in Embodiment 4 of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0063] like Figure 4 As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0064] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0065] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as the smart cockpit counterfactual sample generation method.

[0066] In some embodiments, the smart cockpit counterfactual sample generation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the smart cockpit counterfactual sample generation method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the smart cockpit counterfactual sample generation method by any other suitable means (e.g., by means of firmware).

[0067] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0068] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0069] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0070] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0071] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0072] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0073] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the smart cockpit counterfactual sample generation method provided in any embodiment of this invention.

[0074] In implementing a computer program product, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0075] Example 5 Figure 5 This is a schematic diagram of a smart cockpit system according to Embodiment 5 of the present invention. The smart cockpit system is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The smart cockpit system can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0076] like Figure 5 As shown, the intelligent cockpit system 50 includes at least one processor 51 and a memory, such as a read-only memory (ROM) 52 and a random access memory (RAM) 53, communicatively connected to the at least one processor 51. The memory stores computer programs executable by the at least one processor. The processor 51 can perform various appropriate actions and processes based on the computer program stored in the ROM 52 or loaded from storage unit 58 into the RAM 53. The RAM 53 can also store various programs and data required for the operation of the intelligent cockpit system 50. The processor 51, ROM 52, and RAM 53 are interconnected via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0077] Multiple components in the intelligent cockpit system 50 are connected to the I / O interface 55, including: input units 56, such as a keyboard and mouse; output units 57, such as various types of displays and speakers; storage units 58, such as disks and optical discs; and communication units 59, such as network interface cards, modems, and wireless transceivers. The communication unit 59 allows the intelligent cockpit system 50 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0078] Processor 51 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 51 performs the various methods and processes described above, such as intelligent cockpit interaction methods.

[0079] In some embodiments, the smart cockpit interaction method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed on the smart cockpit system 50 via ROM 52 and / or communication unit 59. When the computer program is loaded into RAM 53 and executed by processor 51, one or more steps of the smart cockpit interaction method described above may be performed. Alternatively, in other embodiments, processor 51 may be configured to perform the smart cockpit interaction method by any other suitable means (e.g., by means of firmware).

[0080] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0081] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0082] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0083] To provide interaction with the user, the systems and technologies described herein can be implemented on a smart cockpit system having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the smart cockpit system. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0084] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0085] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0086] This invention also provides a vehicle including the intelligent cockpit system as described in any embodiment of the invention.

[0087] In implementing a computer program product, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0088] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0089] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for generating counterfactual samples in an intelligent cockpit, characterized in that, include: Obtain at least one original cockpit interaction command; Determine at least one influencing condition that affects the execution result of the original cockpit interaction command; While keeping the original cockpit interaction commands unchanged, at least one of the influencing conditions is adjusted to generate at least one counterfactual state; For the execution scenario of the original cockpit interaction command in the counterfactual state, determine the corresponding processing result of the intelligent cockpit system; The original cockpit interaction command, the counterfactual state, and the corresponding processing result are associated to form a counterfactual sample data.

2. The method according to claim 1, characterized in that, The acquisition of at least one original cockpit interaction command includes: Obtain historical cockpit interaction data as the original cockpit interaction command; or, Construct instruction generation prompts and input them into the basic large model so that the basic large model can generate cockpit interaction instructions that conform to the cockpit interaction scenario based on the instruction generation prompts, which are then used as the original cockpit interaction instructions.

3. The method according to claim 1, characterized in that, The determination of at least one influencing condition that affects the execution result of the original cockpit interaction command includes: By using a preset processing strategy, at least one influencing condition that affects the execution result of the original cockpit interaction command is selected from the cockpit association status corresponding to the original cockpit interaction command.

4. The method according to claim 3, characterized in that, The cockpit-related states include at least: vehicle status, environmental status, occupant status, vehicle type capabilities, and safety rules.

5. The method according to claim 3, characterized in that, The preset processing strategy includes one or more of the following: processing strategy based on artificial intelligence model, processing strategy based on rules, and processing strategy based on manual annotation.

6. The method according to claim 1, characterized in that, The processing result includes at least one of the following: direct execution, inquiry for confirmation, refusal to execute, prompt for support, and prompt for restriction.

7. The method according to claim 6, characterized in that, The counterfactual sample data is in the format of structured data and includes at least the following fields: cockpit interaction command field, counterfactual status field, and processing result label field.

8. The method according to claim 7, characterized in that, When the processing result is a confirmation of the query, the method further includes: Determine and output the corresponding response text for the inquiry; The counterfactual sample data also includes a response text field.

9. A smart cockpit interaction method, characterized in that, include: Obtain the cockpit interaction commands input by the user and the current counterfactual state of the vehicle corresponding to the cockpit interaction commands; The cockpit interaction commands and the current counterfactual state are input into a pre-trained intelligent cockpit model to obtain the processing result output by the intelligent cockpit model. The intelligent cockpit model is trained using counterfactual sample data generated by the intelligent cockpit counterfactual sample generation method as described in any one of claims 1-8. Based on the processing result, the vehicle is controlled to perform corresponding cockpit operations, or corresponding prompts are output to the user.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the smart cockpit counterfactual sample generation method as described in any one of claims 1-8.

11. An intelligent cockpit system, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the intelligent cockpit interaction method as described in any one of claims 9.

12. A vehicle, characterized in that, Including the intelligent cockpit system as described in claim 11.

Images