Interaction control method and device in vehicle cabin, electronic equipment and storage medium

By obtaining and analyzing the instructions and biometric information of personnel in the cockpit, and combining historical data, precise control of vehicle equipment functions is achieved, the problem of insufficient adaptability of smart cockpit systems to individual users is solved, and voice information interpretation and equipment control are improved accuracy.

CN119089907APending Publication Date: 2024-12-06CHINA FAW CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411085883.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing smart cockpit system is not adaptable enough to provide customized service experiences in individual users, mainly because the user model is relatively simple to establish and lacks long-term learning and self-optimization mechanisms, making it difficult to effectively accumulate and utilize user personalized data.

Method used

By obtaining the instruction information and biometric information of the personnel in the cockpit, retrieving historical data information, analyzing the instruction intentions of the personnel, and generating control information based on the intention and equipment function status, precise control of the functions of the vehicle equipment is achieved.

Benefits of technology

Enhance semantic analysis capabilities through historical data, use known semantic fill questioning module to continuously analyze unknown semantics, complete the interpretation of speech information, lock the functions of the vehicle equipment to be controlled, and realize the specific functional control of complex semantic instructions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089907A_ABST
    Figure CN119089907A_ABST
Patent Text Reader

Abstract

The invention discloses an interaction control method and device in a vehicle cabin, electronic equipment and a storage medium, and relates to the field of man-machine interaction, and the method comprises the steps: obtaining the instruction information of a person in the cabin; according to the acquired instruction information of the personnel in the cabin, acquiring biological recognition information of the personnel in the cabin; according to the acquired instruction information of the personnel in the cabin and the acquired biological recognition information of the personnel in the cabin, historical data information of the personnel in the cabin is called; according to the acquired instruction information of the personnel in the cockpit, acquiring biological recognition information of the personnel in the cockpit and historical data information of the personnel in the cockpit, and analyzing instruction intentions of the personnel in the cockpit; according to the instruction intention of the personnel in the cabin, obtaining vehicle equipment function state information; according to the instruction intention of the personnel in the cabin and the vehicle equipment function state information, generating vehicle equipment function control information; according to the vehicle equipment function control information, the vehicle equipment function state is controlled. Through multiple rounds of semantic interpretation, vehicle equipment function related information is interpreted.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction, and in particular to an in-cabin interactive control method for a vehicle, an in-cabin interactive control device for a vehicle, an electronic device, a storage medium, and a vehicle. Background Art

[0002] At present, the technology of in-vehicle intelligent cockpit is developing rapidly, aiming to provide drivers and passengers with a safer, more convenient and personalized travel experience by integrating multiple sensing and interaction technologies. Among them, multimodal intention understanding is one of the core technologies of the intelligent cockpit, which integrates multiple input methods such as facial expression analysis, gesture recognition, and multimodal scene information to achieve accurate understanding and rapid response to the intentions of drivers and passengers.

[0003] Existing smart cockpit systems often adopt a universal design, which is not adaptable enough to the individual differences of different users. The ability to identify the individual habits, preferences, and emotional states of users is limited, making it difficult to provide a truly customized service experience. The reason behind this is that the system's user model is relatively simple to establish, lacks long-term learning and self-optimization mechanisms, and is difficult to effectively accumulate and utilize users' personalized data.

[0004] Therefore, a solution for interactive control within the vehicle cabin is needed, which introduces historical data to assist in parsing user intentions so that indirect user commands can be accurately parsed into commands corresponding to the functions of the vehicle equipment. Summary of the invention

[0005] The purpose of the present invention is to provide a vehicle cabin interactive control method, a vehicle cabin interactive control device, an electronic device, a storage medium and a vehicle, which at least solve the problem of analyzing the driver's intention when it is unclear, and solve one technical problem in the problem of analyzing user intention based on the user's personal characteristics.

[0006] The present invention provides the following scheme:

[0007] According to one aspect of the present invention, a vehicle cabin interactive control method is provided, the vehicle cabin interactive control method comprising:

[0008] Obtain command information from personnel in the cockpit;

[0009] According to the acquisition of the command information of the person in the cockpit, the biometric information of the person in the cockpit is collected;

[0010] According to the acquisition of command information of the person in the cockpit and the collection of biometric information of the person in the cockpit, historical data information of the person in the cockpit is retrieved;

[0011] According to the acquisition of the command information of the person in the cockpit, the collection of the biometric information of the person in the cockpit and the historical data information of the person in the cockpit, the command intention of the person in the cockpit is analyzed;

[0012] Acquiring vehicle equipment function status information according to the command intention of the person in the cockpit;

[0013] Generate vehicle equipment function control information according to the analyzed command intention of the person in the cockpit and the vehicle equipment function status information;

[0014] The state of the vehicle equipment function is controlled according to the vehicle equipment function control information.

[0015] Furthermore, obtaining the command information of the person in the cockpit includes:

[0016] Collect voice information of people in the cockpit;

[0017] Interpret semantics based on speech, including classifying unknown semantics and known semantics based on the intent database corresponding to the preset slot text;

[0018] Filling in the questioning module for unknown semantics according to the known semantics;

[0019] Performing voice interaction according to the questioning module;

[0020] The semantics in the voice interaction process are interpreted, and the unknown semantics and known semantics are classified in another round. Based on the known semantics, the question module is filled in for the unknown semantics in this round.

[0021] Further, the acquiring of the instruction information of the person in the cabin and collecting the biometric information of the person in the cabin comprises:

[0022] Obtain visual recognition information;

[0023] Analyze facial features and behavioral characteristics based on visual recognition information;

[0024] The facial features are analyzed to include identity information and facial status information of cabin environment discomfort recognized according to the facial features;

[0025] Parsing the behavior characteristics includes identifying behavior status information indicating cabin environment discomfort according to the behavior characteristics.

[0026] Further, the obtaining of the command information of the person in the cockpit and the collection of the biometric information of the person in the cockpit and the retrieval of the historical data information of the person in the cockpit includes:

[0027] Retrieve historical data information of personnel behavior, historical data information of instructions and historical data information of vehicle equipment function control generated accordingly based on identity information;

[0028] According to the historical data information, corresponding to the current personnel instructions and the personnel biometric information, analyzing the personnel's instruction intention;

[0029] The analyzing the personnel's instruction intention according to the historical data information, corresponding to the current personnel's instructions and the personnel's biometric information, includes assisting in filling in the question module for unknown semantics according to the known semantics.

[0030] Further, the obtaining of vehicle equipment function status information according to the analysis of the command intention of the person in the cockpit includes:

[0031] According to the semantics of voice interpretation, it is determined whether the command intention of the person in the cockpit is to control the vehicle equipment;

[0032] If yes, then judging whether the current functional state of the device satisfies the control target of the vehicle device in the instruction intention of the personnel according to the pointing control vehicle device;

[0033] If,satisfied, then the current vehicle equipment function status is fed back;

[0034] If it is not satisfied, then after setting the target parameters for controlling the vehicle equipment according to the instruction intention, the state of the vehicle equipment function is controlled according to the target parameters.

[0035] Furthermore, after setting the target parameters for controlling the vehicle equipment according to the command intention, controlling the state of the vehicle equipment function according to the target parameters includes:

[0036] Send feedback information according to the target parameters set for controlling vehicle equipment;

[0037] Determine whether a voice message of response is received within a preset delay threshold;

[0038] If,received, then interpret the semantics according to the voice information of the response, and adjust the parameters according to the semantic interpretation result;

[0039] If,not received, the state of the vehicle equipment function is controlled according to the target parameter.

[0040] According to two aspects of the present invention, there is provided an interactive control device in a vehicle cabin, the interactive control device in the vehicle cabin comprising:

[0041] The command information module is used to obtain the command information of the personnel in the cockpit;

[0042] A biometric information module, used to collect biometric information of people in the cockpit according to the command information of obtaining people in the cockpit;

[0043] A historical information module, used to retrieve historical data information of the people in the cockpit according to the acquisition of command information of the people in the cockpit and the collection of biometric information of the people in the cockpit;

[0044] A command parsing module, used to parse the command intention of the person in the cockpit according to the command information of the person in the cockpit, the biometric information of the person in the cockpit and the historical data information of the person in the cockpit;

[0045] The equipment information module is used to obtain the functional status information of the vehicle equipment according to the command intention of the personnel in the cockpit;

[0046] A control information module, used to generate vehicle equipment function control information according to the command intention of the personnel in the cockpit and the vehicle equipment function status information;

[0047] The device control module is used to control the state of the vehicle device function according to the vehicle device function control information.

[0048] According to three aspects of the present invention, there is provided an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0049] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the vehicle cabin interactive control method.

[0050] According to four aspects of the present invention, there is provided a computer-readable storage medium, comprising: a computer program executable by an electronic device is stored therein, and when the computer program runs on the electronic device, the electronic device executes the steps of the vehicle cabin interactive control method.

[0051] According to five aspects of the present invention, there is provided a vehicle, comprising:

[0052] An electronic device for implementing the steps of the vehicle cabin interactive control method;

[0053] A processor, the processor runs a program, and when the program runs, the steps of the vehicle cabin interactive control method are executed from the data output by the electronic device;

[0054] The storage medium is used to store a program, and when the program is running, the steps of the vehicle cabin interactive control method are executed for the data output from the electronic device.

[0055] Through the above scheme, the following beneficial technical effects are obtained:

[0056] This application uses historical data to enhance semantic parsing capabilities, uses known semantics to fill in the question module to continuously parse unknown semantics, and ultimately completes the interpretation of the entire voice information.

[0057] The present application locks the vehicle equipment function to be controlled by parsing complex semantic information, so that complex semantic instructions can point to specific vehicle equipment functions.

[0058] This application finally parses a sentence of voice information through multiple rounds of semantic interpretation, so that the information related to the vehicle equipment function in the voice information can be interpreted. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a flow chart of a vehicle cabin interactive control method provided by one or more embodiments of the present invention.

[0060] Figure 2 It is a structural diagram of an interactive control device in a vehicle cabin provided by one or more embodiments of the present invention.

[0061] Figure 3 A schematic diagram of a vehicle cabin interaction system according to a specific embodiment of the present invention.

[0062] Figure 4 A schematic diagram of constructing prompt words for an intent recognition task according to a specific embodiment of the present invention.

[0063] Figure 5 A schematic diagram of obtaining a prompt word inference result according to a specific embodiment of the present invention.

[0064] Figure 6 A schematic diagram of a Prompt template for filling in a question task according to a specific embodiment of the present invention.

[0065] Figure 7 A schematic diagram of asking questions about unknown information in user intention according to a specific embodiment of the present invention.

[0066] Figure 8 A schematic diagram of a prompt for constructing a multimodal information extraction task based on user intent according to a specific embodiment of the present invention.

[0067] Fig. 9 A schematic diagram of using a large graphic model to answer questions raised by the large model according to a specific embodiment of the present invention.

[0068] Fig.10 A structural block diagram of an electronic device of a vehicle cabin interactive control method provided by one or more embodiments of the present invention. DETAILED DESCRIPTION

[0069] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] Figure 1 It is a flow chart of a vehicle cabin interactive control method provided by one or more embodiments of the present invention.

[0071] like Figure 1 The vehicle cabin interactive control method shown includes:

[0072] Step S1, obtaining command information of personnel in the cockpit;

[0073] Step S2, collecting biometric information of the person in the cabin according to the command information of the person in the cabin;

[0074] Step S3, retrieving historical data information of the people in the cockpit according to obtaining command information of the people in the cockpit and collecting biometric information of the people in the cockpit;

[0075] Step S4, analyzing the command intention of the person in the cockpit based on obtaining the command information of the person in the cockpit, collecting the biometric information of the person in the cockpit and the historical data information of the person in the cockpit;

[0076] Step S5, obtaining vehicle equipment function status information according to analyzing the command intention of the person in the cockpit;

[0077] Step S6, generating vehicle equipment function control information according to analyzing the command intention of the person in the cockpit and the vehicle equipment function status information;

[0078] Step S7, controlling the state of the vehicle equipment function according to the vehicle equipment function control information.

[0079] Specifically, in one embodiment, the audio recognition device identifies the direction of the sound source in the cabin and locates the cabin position where the command is issued, such as the main and co-pilot seats, front and rear passenger seats, etc. The person in the corresponding cabin position collects the biometric information of the person, such as identifying the age and gender of the passenger through the audio recognition device, and also includes using the visual recognition state to perform facial recognition and behavioral feature recognition on the passenger, and combining multiple biometric features to determine the identity, possible preferences, behavioral intentions, etc. of the passenger. After knowing the identity of the passenger, the historical data information of the passenger is retrieved according to the identity. Combined with the historical data information, the preferences and habits of the passenger are obtained, including the operation of vehicle functions, the sentence pattern of issuing voice commands, and the preferences and habits of the passenger's sitting position, etc. With the help of historical data information, the audio recognition device is assisted in parsing the voice commands of the person, so that the weight of the historical data information is added in the parsing process. Through the parsing of the voice command, the semantics or intentions related to the control of the vehicle equipment function, as well as the target parameters, function status, effect status, etc. of the vehicle equipment function control are extracted, thereby forming an input and output process from voice command to vehicle function control.

[0080] In this embodiment, obtaining the command information of the person in the cabin includes:

[0081] Collect voice information of people in the cockpit;

[0082] Interpret semantics based on speech, including classifying unknown semantics and known semantics based on the intent database corresponding to the preset slot text;

[0083] Based on known semantics, fill in the question module for unknown semantics;

[0084] Conduct voice interaction based on the question module;

[0085] The semantics in the voice interaction process are interpreted, and then the unknown semantics and known semantics are classified in another round. Based on the known semantics, the question module is filled in for the unknown semantics in this round.

[0086] Specifically, in one embodiment, a speech recognition device is internally loaded with speech recognition software, which parses a sentence of speech and classifies unknown semantics and known semantics according to the intent database corresponding to the preset slot text. The known semantics is the part that is successfully compared with the preset slot text, and the corresponding intent database has a clear meaning. For other parts, due to lack of data, they are listed as unknown semantics. Through further human-computer voice interaction question and answer, the unknown semantics are gradually clarified as data with clear meaning. A sentence of speech is regarded as a whole, and the known semantics are used as a clue, which are input into the question module together with the unknown semantics. With the help of known semantics, extended voice interaction is carried out to generate directionality and accelerate the parsing of unknown semantics. In voice interaction, all unknown semantics can be gradually parsed through one or more rounds of voice interaction question and answer.

[0087] In addition to using known semantics to accelerate the analysis of unknown semantics, historical data information can also be input. For example, in the past, passengers were accustomed to using the color of their clothes to express the color of the ambient light they wanted to adjust. For the unknown semantics about the color of the clothes in the voice, the priority is to conduct voice interaction questions and answers in the direction of whether to control the ambient color.

[0088] Usually, when a type of speech is semantically analyzed for the first time, it will progress slowly due to the lack of reference in the direction of voice interactive question and answer. After historical data is generated, machine learning can be used to process the historical data to improve the efficiency of obtaining user intent.

[0089] For example, the voice analysis of the user is, "I want to find a place to relax." "I want" and "find a place" are known semantics. Usually, when the navigation software is activated, "relax" is an unknown voice. Based on the known semantics of "I want" and "find a place", the voice interaction question is whether "relax" is the target address. Based on the historical data, the user's behavioral habits of not being able to ride in a vehicle for a long time and the habit of expressing voice commands, the voice interaction question is whether to locate the nearest toilet address. This realizes the voice interaction extended by known semantics, generates directionality, and accelerates the technical effect of parsing unknown semantics.

[0090] In this embodiment, according to obtaining the instruction information of the person in the cabin, collecting the biometric information of the person in the cabin includes:

[0091] Obtain visual recognition information;

[0092] Analyze facial features and behavioral characteristics based on visual recognition information;

[0093] Facial feature analysis includes identity information based on facial features and facial status information of cabin environment discomfort;

[0094] Analyzing the behavior characteristics includes identifying behavior status information indicating cabin environment discomfort based on the behavior characteristics.

[0095] Specifically, in one embodiment, in addition to audio recognition (voiceprint recognition), biometric information can also be collected by visual recognition. Through biometric recognition, identity information and historical data and authority data under the corresponding identity information can be obtained.

[0096] Visual recognition also includes identifying the behavior and health status of people. For example, if the infrared signal of a person's face is enhanced and the amplitude and frequency of shaking increase, it can be judged that the passenger is uncomfortable in the current cabin environment, may be nervous, and want to go to the toilet. According to similar behaviors in historical data information, the instructions issued by the person are to control the navigation to find the nearest toilet. In the next step of voice recognition, this information is used as background information, and voice interaction questions and answers for unknown semantics such as finding the nearest toilet are preferred.

[0097] In this embodiment, according to obtaining the command information of the person in the cockpit and collecting the biometric information of the person in the cockpit, retrieving the historical data information of the person in the cockpit includes:

[0098] Retrieve historical data information of personnel behavior, historical data information of instructions and historical data information of vehicle equipment function control generated accordingly based on identity information;

[0099] According to historical data information, corresponding to the current personnel instructions and personnel biometric information, analyze the personnel's instruction intentions;

[0100] Based on historical data information, corresponding to the current personnel instructions and personnel biometric information, the personnel's instruction intentions are parsed, including assisting in filling in the question module based on known semantics and unknown semantics.

[0101] Specifically, in one embodiment, the identity information of the person is obtained through biometrics, and the historical data corresponding to the identity of the person is retrieved, and the retrieved historical data is for the position and status of the person. For example, when the person is in the driving seat, the historical data retrieved is the historical data when the person is in the driver's identity. When parsing the intention of the person's command, the control needs that may appear in the driver's identity are given priority. For example, the driver's command to close the window is equivalent to the rear passenger's command to close the window. The range of controlled windows is wider. In the driver's identity, it is necessary to confirm whether all windows are closed in the first round of questions and answers through voice interaction questions and answers. If not, then confirm which specific window to close in the second round of questions and answers. In the rear passenger's identity, it is necessary to confirm whether only the rear left or right window is closed in the first round of questions and answers through voice interaction questions and answers. If not, then confirm which specific window to close in the second round of questions and answers.

[0102] Using historical data information to assist in filling in the question module based on known semantics for unknown semantics can cause slight changes in the classification stage of known semantics and unknown semantics. For example, the voice of the voice analysis issued by the user is, "I want to find a place to relax." After issuing this voice command, the person recorded in the historical data information repeatedly chose to navigate to a place where there must be a toilet. After issuing this voice command, another person recorded in the historical data information repeatedly chose to navigate to a place where there must be a supermarket selling cigarettes. Therefore, when the historical data information is used to assist in analyzing the personnel's instruction intentions, when the known semantics and unknown semantics are further input into the question module, the parameters of the question module are different.

[0103] In this embodiment, according to analyzing the command intention of the person in the cockpit, obtaining the vehicle equipment function status information includes:

[0104] According to the semantics of voice interpretation, it is determined whether the command intention of the person in the cockpit is to control the vehicle equipment;

[0105] If yes, then judging whether the current functional state of the device satisfies the control target of the vehicle device in the instruction intention of the personnel according to the pointing control vehicle device;

[0106] If,satisfied, then the current vehicle equipment function status is fed back;

[0107] If it is not satisfied, then after setting the target parameters for controlling the vehicle equipment according to the instruction intention, the state of the vehicle equipment function is controlled according to the target parameters.

[0108] Specifically, in one embodiment, based on the semantics of voice interpretation, it is determined whether the intention of the person in the cabin to control the vehicle equipment is included in the command intention, including determining which function of the vehicle equipment is intended to be controlled. For example, if the voice analysis voice sent by the user is "I want to find a place to relax", it can be determined that the intention is to control the navigation software and to set the navigation location to the toilet.

[0109] According to the pointing control vehicle equipment, it is judged whether the current functional state of the equipment meets the control target of the vehicle equipment in the personnel's instruction intention. For example, if the navigation location has been selected as a gas station, and the gas station has a toilet function, there is no need to modify the navigation location. The user can be informed through voice interaction that the current navigation location is a gas station, which includes toilet facilities. If the gas station at the current navigation location does not include toilet facilities, after setting the target parameters for controlling the vehicle equipment according to the instruction intention (setting the passing location with toilets), the state of the vehicle equipment function is controlled according to the target parameters (such as recommending the route, or reporting the congestion status according to the road conditions, etc.).

[0110] In this embodiment, after setting the target parameters for controlling the vehicle equipment according to the command intention, the state of controlling the vehicle equipment function according to the target parameters includes:

[0111] Send feedback information according to the target parameters set for controlling vehicle equipment;

[0112] Determine whether a voice message of response is received within a preset delay threshold;

[0113] If,received, then interpret the semantics according to the voice information of the response, and adjust the parameters according to the semantic interpretation result;

[0114] If,not received, the state of the vehicle equipment function is controlled according to the target parameter.

[0115] Specifically, in one embodiment, there is also a waiting process before executing the command, which is used to confirm the execution of the command. Due to the special vehicle environment, the shift of attention to the voice interaction response with the vehicle poses a certain risk to the vehicle driving safety. For example, the vehicle issues a voice interaction response to inform that there are toilet facilities at the current navigation destination and asks whether to maintain the current navigation settings. At this time, the driver can continue to concentrate on driving and ignore the inquiry of the voice interaction response. After the window time has passed, the original parameters can be maintained unchanged, and the current vehicle equipment function status is maintained by default.

[0116] Figure 2 It is a structural diagram of an interactive control device in a vehicle cabin provided by one or more embodiments of the present invention.

[0117] like Figure 2 The interactive control device in the vehicle cabin shown includes: a command information module, a biological information module, a historical information module, a command parsing module, a device information module, a control information module, and a device control module;

[0118] The command information module is used to obtain the command information of the personnel in the cockpit;

[0119] A biometric information module is used to collect biometric information of people in the cockpit based on the command information of people in the cockpit;

[0120] A historical information module is used to retrieve historical data information of the people in the cockpit based on obtaining command information of the people in the cockpit and collecting biometric information of the people in the cockpit;

[0121] The command parsing module is used to parse the command intention of the person in the cockpit based on obtaining the command information of the person in the cockpit, collecting the biometric information of the person in the cockpit and the historical data information of the person in the cockpit;

[0122] The equipment information module is used to obtain the functional status information of the vehicle equipment according to the command intention of the personnel in the cockpit;

[0123] A control information module is used to generate vehicle equipment function control information based on the analysis of the command intention of the personnel in the cockpit and the vehicle equipment function status information;

[0124] The device control module is used to control the state of the vehicle device function according to the vehicle device function control information.

[0125] It is worth noting that although the present system only discloses the instruction information module, the biological information module, the historical information module, the instruction parsing module, the device information module, the control information module, and the device control module, it does not mean that the present device is limited to the above-mentioned basic functional modules. Rather, what the present invention wants to express is that, based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with the prior art to form an infinite number of embodiments or technical solutions. In other words, the present system is open rather than closed. Just because the present embodiment only discloses individual basic functional modules, it cannot be considered that the scope of protection of the claims of the present invention is limited to the above-mentioned basic functional modules.

[0126] Through the above scheme, the following beneficial technical effects are obtained:

[0127] This application uses historical data to enhance semantic parsing capabilities, uses known semantics to fill in the question module to continuously parse unknown semantics, and ultimately completes the interpretation of the entire voice information.

[0128] The present application locks the vehicle equipment function to be controlled by parsing complex semantic information, so that complex semantic instructions can point to specific vehicle equipment functions.

[0129] This application finally parses a sentence of voice information through multiple rounds of semantic interpretation, so that the information related to the vehicle equipment function in the voice information can be interpreted.

[0130] Figure 3 A schematic diagram of a vehicle cabin interaction system according to a specific embodiment of the present invention.

[0131] Figure 4 A schematic diagram of constructing prompt words for an intent recognition task according to a specific embodiment of the present invention.

[0132] Figure 5 A schematic diagram of obtaining a prompt word inference result according to a specific embodiment of the present invention.

[0133] Figure 6 A schematic diagram of a Prompt template for filling in a question task according to a specific embodiment of the present invention.

[0134] Figure 7 A schematic diagram of asking questions about unknown information in user intention according to a specific embodiment of the present invention.

[0135] Figure 8 A schematic diagram of a prompt for constructing a multimodal information extraction task based on user intent according to a specific embodiment of the present invention.

[0136] Fig. 9 A schematic diagram of using a large graphic model to answer questions raised by the large model according to a specific embodiment of the present invention.

[0137] In a specific embodiment, Figure 3 The vehicle in-cabin interactive system shown includes several functional blocks: information layer, information bus, intent recognition, intent understanding, and information extraction.

[0138] 1. First, the information layer integrates the in-vehicle scene semantic information, in-vehicle multi-modal information, and environmental information and transmits them to the information bus;

[0139] 2. Then use the scene semantic information, in-car multimodal information, and environmental information of the information bus to fill in the large model prompt template and construct a prompt for intent recognition;

[0140] 3. Input the prompt into the big model to predict the result of intent recognition (intent and slot);

[0141] 4. If multimodal intent understanding is involved, this step will return an unknown slot;

[0142] 5. Then use all the returned unknown slots to fill in the prompt word template of the question task, input the obtained prompt words into the big model, and get the question method (interactive answer);

[0143] 6. Then, based on the question (interactive answer) and the in-car camera information, a multimodal prompt word is constructed and a large picture and text model is input to obtain the information of all unknown slots;

[0144] 7. Use the complete intent understanding results to build intelligent commands and pass them to the execution layer to execute the behavioral actions corresponding to the user's intent on the vehicle side.

[0145] In this embodiment, if Figure 4 As shown in , construct the prompt words for the intention recognition task. Figure 3 In the embodiment, the multimodal information is collected by the information bus to construct the prompt word of the intention recognition task.

[0146] In this embodiment, if Figure 5 As shown, obtain the prompt word reasoning result. Figure 3 In the consciousness recognition task Prompt uses the Owen 7B text model for multimodal intent understanding.

[0147] In this embodiment, if Figure 6 As shown in the following figure, fill in the prompt template of the question task. Figure 3 In the above, based on the consciousness recognition task Prompt uses the Owen 7B text model to perform multimodal intent understanding, uses all the returned unknown slots to fill the prompt word template of the question task, and inputs the obtained prompt words into the Owen7B text model to obtain the interactive answering question.

[0148] In this embodiment, if Figure 7 As shown in , questions are asked about unknown information in the user's intention. Figure 3 In the example, the Owen 7B text model is used to ask questions about unknown information in the user's intention based on the constructed prompt words.

[0149] In this embodiment, if Figure 8 As shown in , a multi-modal information extraction task is constructed for user intention. Figure 3 In the process, the inference result is used to analyze whether each intention is a multi-modal task. If it is a multi-modal task, the in-vehicle camera data is passed in.

[0150] In this embodiment, if Fig. 9 As shown in the figure, use the picture and text large model to answer the questions raised by the large model. Figure 3 In the example, based on the collected information, a prompt for a multimodal information extraction task is constructed for each user intention.

[0151] like Figures 4 to 9 , taking the voice command processing of a specific embodiment as an example, the atmosphere light is controlled.

[0152] Fig.10 A structural block diagram of an electronic device of a vehicle cabin interactive control method provided by one or more embodiments of the present invention.

[0153] like Fig.10 As shown, the present application provides an electronic device, including: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0154] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of a method for interactive control in a vehicle cabin.

[0155] The present application also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of a method for interactive control in a vehicle cabin.

[0156] The present application also provides a vehicle, comprising:

[0157] An electronic device for implementing the steps of the interactive control method in a vehicle cabin;

[0158] A processor, the processor runs a program, and when the program runs, data output from the electronic device executes the steps of the vehicle cabin interactive control method;

[0159] The storage medium is used to store a program, and when the program is running, it executes the steps of the vehicle cabin interactive control method for the data output from the electronic device.

[0160] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0161] The electronic device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory. The operating system can be any one or more computer operating systems that implement electronic device control through a process, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. In addition, in an embodiment of the present invention, the electronic device can be a handheld device such as a smart phone or a tablet computer, or can be an electronic device such as a desktop computer or a portable computer, which is not particularly limited in the embodiment of the present invention.

[0162] The execution subject of the electronic device control in the embodiment of the present invention may be an electronic device, or a functional module in the electronic device that can call and execute a program. The electronic device may obtain the firmware corresponding to the storage medium. The firmware corresponding to the storage medium is provided by the supplier. The firmware corresponding to different storage media may be the same or different, which is not limited here. After the electronic device obtains the firmware corresponding to the storage medium, the firmware corresponding to the storage medium may be written into the storage medium, specifically, the firmware corresponding to the storage medium may be burned into the storage medium. The process of burning the firmware into the storage medium may be implemented using existing technology, which will not be described in detail in the embodiment of the present invention.

[0163] The electronic device may also obtain a reset command corresponding to the storage medium. The reset command corresponding to the storage medium is provided by the supplier. The reset commands corresponding to different storage media may be the same or different, and are not limited here.

[0164] At this time, the storage medium of the electronic device is a storage medium in which the corresponding firmware is written, and the electronic device can respond to the reset command corresponding to the storage medium in the storage medium in which the corresponding firmware is written, so that the electronic device resets the storage medium in which the corresponding firmware is written according to the reset command corresponding to the storage medium. The process of resetting the storage medium according to the reset command can be implemented by the existing technology and will not be described in detail in the embodiments of the present invention.

[0165] For the convenience of description, the above devices are described in terms of functions and are divided into various units and modules. Of course, when implementing the present application, the functions of each unit and module can be implemented in the same or multiple software and / or hardware.

[0166] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0167] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0168] It can be known from the description of the above implementation modes that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute the methods described in the various implementation modes of the present application or certain parts of the implementation modes.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle cabin interactive control method, characterized in that: The vehicle cabin interactive control method comprises: Obtain command information from personnel in the cockpit; According to the acquisition of the command information of the person in the cockpit, the biometric information of the person in the cockpit is collected; According to the acquisition of command information of the person in the cockpit and the collection of biometric information of the person in the cockpit, historical data information of the person in the cockpit is retrieved; According to the acquisition of the command information of the person in the cockpit, the collection of the biometric information of the person in the cockpit and the historical data information of the person in the cockpit, the command intention of the person in the cockpit is analyzed; Acquiring vehicle equipment function status information according to the command intention of the person in the cockpit; Generate vehicle equipment function control information according to the analyzed command intention of the person in the cockpit and the vehicle equipment function status information; The state of the vehicle equipment function is controlled according to the vehicle equipment function control information.

2. The vehicle cabin interactive control method according to claim 1, characterized in that: The obtaining of cabin personnel command information includes: Collect voice information of people in the cockpit; Interpret semantics based on speech, including classifying unknown semantics and known semantics based on the intent database corresponding to the preset slot text; Filling in the questioning module for unknown semantics according to the known semantics; Performing voice interaction according to the questioning module; The semantics in the voice interaction process are interpreted, and the unknown semantics and known semantics are classified in another round. Based on the known semantics, the question module is filled in for the unknown semantics in this round.

3. The vehicle cabin interactive control method according to claim 2, characterized in that: The acquiring of the command information of the person in the cabin and collecting the biometric information of the person in the cabin comprises: Obtain visual recognition information; Analyze facial features and behavioral characteristics based on visual recognition information; The facial features are analyzed to include identity information and facial status information of cabin environment discomfort recognized according to the facial features; Parsing the behavior characteristics includes identifying behavior status information indicating cabin environment discomfort according to the behavior characteristics.

4. The vehicle cabin interactive control method according to claim 3, characterized in that: The obtaining of the command information of the person in the cockpit and the collection of the biometric information of the person in the cockpit and the retrieval of the historical data information of the person in the cockpit includes: Retrieve historical data information of personnel behavior, historical data information of instructions and historical data information of vehicle equipment function control generated accordingly based on identity information; According to the historical data information, corresponding to the current personnel instructions and the personnel biometric information, analyzing the personnel's instruction intention; The analyzing the personnel's instruction intention according to the historical data information, corresponding to the current personnel's instructions and the personnel's biometric information, includes assisting in filling in the question module for unknown semantics according to the known semantics.

5. The vehicle cabin interactive control method according to claim 4, characterized in that: The obtaining of vehicle equipment function status information according to the analysis of the command intention of the person in the cockpit includes: According to the semantics of voice interpretation, it is determined whether the command intention of the person in the cockpit is to control the vehicle equipment; If yes, then judging whether the current functional state of the device satisfies the control target of the vehicle device in the instruction intention of the personnel according to the pointing control vehicle device; If,satisfied, then the current vehicle equipment function status is fed back; If it is not satisfied, then after setting the target parameters for controlling the vehicle equipment according to the instruction intention, the state of the vehicle equipment function is controlled according to the target parameters.

6. The vehicle cabin interactive control method according to claim 5, characterized in that: After the target parameters for controlling the vehicle equipment are set according to the command intention, the state of the vehicle equipment function is controlled according to the target parameters, including: Send feedback information according to the target parameters set for controlling vehicle equipment; Determine whether a voice message of response is received within a preset delay threshold; If,received, then interpret the semantics according to the voice information of the response, and adjust the parameters according to the semantic interpretation result; If,not received, the state of the vehicle equipment function is controlled according to the target parameter.

7. An interactive control device in a vehicle cabin, characterized in that: The vehicle cabin interactive control device comprises: The command information module is used to obtain the command information of the personnel in the cockpit; A biometric information module, used to collect biometric information of people in the cockpit according to the command information of obtaining people in the cockpit; A historical information module, used to retrieve historical data information of the people in the cockpit according to the acquisition of command information of the people in the cockpit and the collection of biometric information of the people in the cockpit; A command parsing module, used to parse the command intention of the person in the cockpit according to the command information of the person in the cockpit, the biometric information of the person in the cockpit and the historical data information of the person in the cockpit; The equipment information module is used to obtain the functional status information of the vehicle equipment according to the command intention of the personnel in the cockpit; A control information module, used to generate vehicle equipment function control information according to the command intention of the person in the cockpit and the vehicle equipment function status information; The device control module is used to control the state of the vehicle device function according to the vehicle device function control information.

8. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; A computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the vehicle cabin interactive control method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: include: It stores a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the vehicle cabin interactive control method as described in any one of claims 1 to 6.

10. A vehicle, characterized in that: include: An electronic device for implementing the steps of the vehicle cabin interactive control method according to any one of claims 1 to 6; A processor, the processor runs a program, and when the program runs, data output from the electronic device executes the steps of the vehicle cabin interactive control method according to any one of claims 1 to 6; A storage medium for storing a program, wherein when the program is run, the program executes the steps of the vehicle cabin interactive control method according to any one of claims 1 to 6 for data output from an electronic device.