Multi-mode intelligent cabin interaction control method and system considering priority switching

Abstract

The invention belongs to the technical field of intelligent cabin interaction control, and particularly relates to a multi-mode intelligent cabin interaction control method and system considering priority switching. Comprising the steps of performing behavior recognition on a user in the intelligent cabin, and obtaining recognized first instruction information; performing voice recognition on a user in the intelligent cabin to obtain recognized second instruction information; the vehicle driving scene is recognized, whether a dangerous scene exists in the vehicle driving scene or not is judged based on a preset judgment rule, if yes, the priority of the second instruction information is set to be the highest priority, and the second instruction information is adopted as a final control instruction to control the vehicle; and if not, performing weighted fusion on the first instruction information and the second instruction information to obtain a final control instruction, and controlling the vehicle. According to the method, the priority is set according to the dangerous scene identified and judged by the vehicle driving scene, so that safer vehicle control is realized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent cockpit interaction control technology, and particularly relates to a multimodal intelligent cockpit interaction control method and system that considers priority switching. Background Technology

[0002] In the field of intelligent cockpit interaction, fully recognizing user emotions can enable users to receive the most suitable functional services during driving, providing a comfortable and relaxing atmosphere for passengers, thus achieving a perfect driving and riding experience and meeting market demands.

[0003] Currently, existing technologies include user emotion recognition using infrared recognition modules. These modules can recognize hand movement trajectories, allowing the system to understand the user's intentions based on hand movement paths and speeds. Simultaneously, a speech recognition mode is available, allowing users to issue various commands via voice, which the system can then convert into corresponding operational instructions. Furthermore, a visual control mode is provided, capturing and analyzing visual information such as the user's eye contact and facial expressions to achieve specific functions. The system also features a mode-switching function, allowing users to flexibly switch between different recognition modes according to their usage habits and the specific scenario.

[0004] However, current upgrades generally focus on more comprehensive and accurate detection of user emotions, neglecting the identification and handling of hazardous factors and scenarios during vehicle driving. When hazardous factors and scenarios occur, the driver and passengers inside the vehicle often undergo significant changes in body, hand, and head positions due to instantaneous reactions. If the system still relies on infrared recognition modules to identify user hand movement trajectories or visual control modes to achieve intelligent cockpit control, it will lead to erroneous control command transmission and will be unable to effectively handle sudden emergencies or dangerous situations, such as suddenly appearing pedestrians, vehicles, moving obstacles, or sudden changes in road conditions. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the present invention provides a multimodal intelligent cockpit interactive control method and system that considers priority switching. By combining the dangerous scenarios identified by vehicle driving scenario recognition, priorities are set, which is more in line with the directness of voice control commands sent by the occupants in dangerous scenarios, thereby achieving safer vehicle control.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of the present invention provides a multimodal intelligent cockpit interaction control method that takes into account priority switching.

[0007] A multimodal intelligent cockpit interaction control method considering priority switching includes the following steps: The system identifies user behavior within the smart cockpit and obtains the first identified command information. Voice recognition is performed on users inside the smart cockpit to obtain the recognized second command information; The system identifies vehicle driving scenarios and, based on pre-defined judgment rules, determines whether any dangerous situations exist within those scenarios. If it exists, the priority of the second instruction information is set to the highest priority, and the second instruction information is used as the final control instruction to control the vehicle; If it does not exist, the first instruction information and the second instruction information are weighted and fused to obtain the final control instruction to control the vehicle.

[0008] A second aspect of the present invention provides a multimodal intelligent cockpit interaction control system that takes into account priority switching.

[0009] A multimodal intelligent cockpit interaction control system that considers priority switching includes: The first instruction acquisition module is configured to: perform behavior recognition on the user in the smart cockpit and acquire the recognized first instruction information; The second command module is configured to: perform voice recognition on the user in the smart cockpit and obtain the recognized second command information; The priority setting module is configured to: identify vehicle driving scenarios and, based on preset judgment rules, determine whether any dangerous scenarios exist within the vehicle driving scenario. If it exists, the priority of the second instruction information is set to the highest priority, and the second instruction information is used as the final control instruction to control the vehicle; If it does not exist, the first instruction information and the second instruction information are weighted and fused to obtain the final control instruction to control the vehicle.

[0010] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the multimodal intelligent cockpit interaction control method considering priority switching as described in the first aspect of the present invention.

[0011] The fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal intelligent cockpit interaction control method considering priority switching as described in the first aspect of the present invention.

[0012] The above one or more technical solutions have the following beneficial effects: This invention provides a multimodal intelligent cockpit interactive control method and system that considers priority switching. It combines the dangerous scenarios identified by vehicle driving scenario recognition to set priorities, assigning the highest priority to control commands obtained from in-vehicle voice recognition results under dangerous scenarios. Control commands obtained from in-vehicle behavior recognition results and in-vehicle voice recognition results under non-dangerous scenarios are weighted and fused to obtain the final control command for vehicle control. By switching the priority of multimodal intelligent cockpit control recognition commands, safer vehicle control is achieved.

[0013] This invention takes into account the actual driving situation of vehicles. Based on the directness of the driver's voice control mode output control commands in dangerous vehicle scenarios, the control mode in dangerous scenarios is switched to voice control as the highest priority, thereby improving the efficiency and safety of control command recognition.

[0014] In the process of recognizing user behavior in the smart cockpit and obtaining the first command information, both driver eye image data and gesture image data are taken into account to obtain a clearer driving intention.

[0015] When identifying vehicle driving scenarios, road conditions, weather conditions, related vehicle environments, and the driving conditions of surrounding vehicles are considered, making the scenario coverage more comprehensive.

[0016] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0018] Figure 1 This is a flowchart of the method in Example 1. Detailed Implementation

[0019] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0020] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0021] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0022] Example 1 This embodiment discloses a multimodal intelligent cockpit interaction control method that takes into account priority switching.

[0023] like Figure 1 As shown, the multimodal intelligent cockpit interaction control method considering priority switching includes the following steps: The system identifies user behavior within the smart cockpit and obtains the first identified command information. Voice recognition is performed on users inside the smart cockpit to obtain the recognized second command information; The system identifies vehicle driving scenarios and, based on pre-defined judgment rules, determines whether any dangerous situations exist within those scenarios. If it exists, the priority of the second instruction information is set to the highest priority, and the second instruction information is used as the final control instruction to control the vehicle; If it does not exist, the first instruction information and the second instruction information are weighted and fused to obtain the final control instruction to control the vehicle.

[0024] The first and second instruction information are weighted and fused to obtain the final control instruction, which can be represented as:

[0025] Where α and β are the fusion weights.

[0026] It is understood that the users in the aforementioned smart cockpit include the driver and passengers.

[0027] This embodiment combines the dangerous scenarios identified by vehicle driving scenario recognition and sets priorities. Control commands obtained from in-vehicle voice recognition results under dangerous scenarios are given the highest priority. Control commands obtained from in-vehicle behavior recognition results and in-vehicle voice recognition results under non-dangerous scenarios are weighted and fused to obtain the final control command to control the vehicle. By switching the priority of control and recognition commands in a multimodal intelligent cockpit, safer vehicle control is achieved.

[0028] Since controlling the vehicle using the first command information obtained from the user behavior recognition results in the smart cockpit cannot guarantee accuracy when dangerous situations occur during vehicle driving (when dangerous factors and dangerous scenarios occur, the driver and passengers in the vehicle will undergo significant changes in the position of their body, hands, head, etc. due to instantaneous reactions), and sending control commands via voice mode is more direct and efficient, this embodiment sets priorities to achieve more scientific and safer control.

[0029] Of course, even when voice control is set as the highest priority, it still needs to be supplemented by data from the vehicle's own sensors and integrated with intelligent detection algorithms to verify the current vehicle and road conditions. When the vehicle executes control according to the highest-priority voice recognition result, the intelligent detection algorithm simultaneously verifies the vehicle's operation. If the verification fails, a voice reminder is issued to the driver, prompting them to reconsider the command, thus providing a higher level of driving protection for safe driving.

[0030] Therefore, this embodiment takes into account the actual driving situation of the vehicle. Based on the directness of the driver's voice control mode output control commands in dangerous vehicle scenarios, the control mode in dangerous scenarios is switched to voice control as the highest priority, thereby improving the efficiency and safety of control command recognition.

[0031] For example, if a pedestrian or other obstacle suddenly appears in front of the vehicle while it is in normal driving, the occupants may make a sound to remind the driver to "stop, stop quickly" in order to avoid a collision. At this time, the voice recognition mode can quickly obtain the second instruction information, namely the control instruction to stop, and control the vehicle more safely.

[0032] Similarly, in the aforementioned emergency scenarios, vehicle occupants may lose control of their physical behavior due to sudden events, resulting in unconscious and large movements, including head movements, hand movements, and body movements. If the vehicle control first command information is still obtained according to the behavior recognition mode at this time, there will be a deviation between the actual control intention and the first command information, and safety control cannot be effectively achieved.

[0033] Furthermore, user behavior is recognized within the smart cockpit to obtain the first identified command information, specifically including: Acquire behavioral data of occupants inside the vehicle, including eye image data and gesture image data; Behavioral data is preprocessed to obtain the driver's intentional tendencies; Based on the driver's intentions, a decision is made to generate the first instruction information for vehicle control.

[0034] In the above process, based on the driver's intention, a decision is made to generate the first instruction information for vehicle control. Existing technologies can be adopted, and this embodiment will not elaborate further.

[0035] Furthermore, the behavioral data is preprocessed to obtain the driver's intention tendency, specifically including: The eye image data is preprocessed to identify the position of the pupil, the direction of gaze, and the duration of fixation, thereby obtaining the first intention tendency; The gesture image data is preprocessed to identify the gesture type and gesture parameters, thereby obtaining the second intention tendency; Based on preset fusion rules and weighting coefficients, the first intention tendency and the second intention tendency are fused to obtain the driver's intention tendency.

[0036] It can be understood that when performing gesture image recognition, it is necessary to pre-store a mapping table of gesture type, gesture parameters and driving intention. After analyzing and recognizing the gesture image, the corresponding driving intention is retrieved from the mapping table, thereby obtaining the second intention tendency.

[0037] Furthermore, voice recognition is performed on the user inside the smart cockpit to obtain the recognized second command information, specifically including: Acquire voice data from occupants in the smart cockpit; The voice data is preprocessed, including noise reduction, feature extraction, speech recognition, and text conversion, to obtain the second command information for vehicle control.

[0038] When identifying vehicle driving scenarios, road conditions, weather conditions, related vehicle environments, and the driving situations of surrounding vehicles are considered, resulting in more comprehensive scenario coverage. (1) Further, vehicle driving scenarios are identified, specifically including: Acquire vehicle driving scenarios, specifically including road conditions, weather conditions, and relevant vehicle conditions; By analyzing road conditions, we can obtain information on the current road's gradient, curvature, and roughness. By checking the weather conditions, we can determine whether there is rain, snow, fog, dust, or sandstorm during the current driving period. By examining the relevant vehicle environment, it can be determined whether there are any vehicle collisions or accidents on the current road.

[0039] Furthermore, based on preset judgment rules, it is determined whether a dangerous situation exists in the vehicle driving scenario, specifically including: When any one of the following occurs: road gradient greater than preset gradient value, road curvature greater than preset curvature value, or road roughness greater than preset roughness threshold, it is determined to be a dangerous situation. A dangerous situation is defined as one of the following weather conditions during the current driving period: rain, snow, fog, dust, or sandstorm. When a vehicle collision or accident occurs on the current road, it is considered a dangerous situation.

[0040] (2) Furthermore, determining whether a dangerous situation exists in the vehicle driving scenario also includes: Identify whether the vehicle ahead on the current road is experiencing a sudden braking situation; if so, it is determined to be a dangerous situation. Identify whether there are any illegal lane changes or overtaking situations; if so, determine them as dangerous situations.

[0041] Example 2 This embodiment discloses a multimodal intelligent cockpit interaction control system that takes into account priority switching.

[0042] A multimodal intelligent cockpit interaction control system that considers priority switching includes: The first instruction acquisition module is configured to: perform behavior recognition on the user in the smart cockpit and acquire the recognized first instruction information; The second command module is configured to: perform voice recognition on the user in the smart cockpit and obtain the recognized second command information; The priority setting module is configured to: identify vehicle driving scenarios and, based on preset judgment rules, determine whether any dangerous scenarios exist within the vehicle driving scenario. If it exists, the priority of the second instruction information is set to the highest priority, and the second instruction information is used as the final control instruction to control the vehicle; If it does not exist, the first instruction information and the second instruction information are weighted and fused to obtain the final control instruction to control the vehicle.

[0043] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0044] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the multimodal intelligent cockpit interactive control method considering priority switching as described in Embodiment 1 of this disclosure.

[0045] Example 4 The purpose of this embodiment is to provide an electronic device.

[0046] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal intelligent cockpit interaction control method considering priority switching as described in Embodiment 1 of this disclosure.

[0047] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0048] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0049] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A multimodal intelligent cockpit interaction control method considering priority switching, characterized in that, Includes the following steps: The system identifies user behavior within the smart cockpit and obtains the first identified command information. Voice recognition is performed on users inside the smart cockpit to obtain the recognized second command information; The system identifies vehicle driving scenarios and, based on pre-defined judgment rules, determines whether any dangerous situations exist within those scenarios. If it exists, the priority of the second instruction information is set to the highest priority, and the second instruction information is used as the final control instruction to control the vehicle; If it does not exist, the first instruction information and the second instruction information are weighted and fused to obtain the final control instruction to control the vehicle.

2. The multimodal intelligent cockpit interaction control method considering priority switching as described in claim 1, characterized in that, The system performs behavior recognition on users within the smart cockpit and obtains the first recognized command information, specifically including: Acquire behavioral data of occupants inside the vehicle, including eye image data and gesture image data; Behavioral data is preprocessed to obtain the driver's intentional tendencies; Based on the driver's intended tendencies, a decision is made to generate first instruction information for vehicle control.

3. The multimodal intelligent cockpit interaction control method considering priority switching as described in claim 2, characterized in that, The behavioral data is preprocessed to obtain the driver's intention tendency, specifically including: The eye image data is preprocessed to identify the position of the pupil, the direction of gaze, and the duration of fixation, thereby obtaining the first intention tendency; The gesture image data is preprocessed to identify the gesture type and gesture parameters, thereby obtaining the second intention tendency; Based on preset fusion rules and weighting coefficients, the first intention tendency and the second intention tendency are fused to obtain the driver's intention tendency.

4. The multimodal intelligent cockpit interaction control method considering priority switching as described in claim 1, characterized in that, Voice recognition is performed on users inside the smart cockpit to obtain the recognized second command information, specifically including: Acquire voice data from occupants in the smart cockpit; The voice data is preprocessed, including noise reduction, feature extraction, speech recognition, and text conversion, to obtain the second command information for vehicle control.

5. The multimodal intelligent cockpit interaction control method considering priority switching as described in claim 1, characterized in that, Vehicle driving scenario recognition specifically includes: Acquire vehicle driving scenarios, specifically including road conditions, weather conditions, and relevant vehicle conditions; By analyzing road conditions, we can obtain information on the current road's gradient, curvature, and roughness. By checking the weather conditions, we can determine whether there is rain, snow, fog, dust, or sandstorm during the current driving period. By examining the relevant vehicle environment, it can be determined whether there are any vehicle collisions or accidents on the current road.

6. The multimodal intelligent cockpit interaction control method considering priority switching as described in claim 5, characterized in that, Based on preset judgment rules, determine whether a dangerous situation exists in the vehicle driving scenario, specifically including: When any one of the following occurs: road gradient greater than preset gradient value, road curvature greater than preset curvature value, or road roughness greater than preset roughness threshold, it is determined to be a dangerous situation. A dangerous situation is defined as one of the following weather conditions during the current driving period: rain, snow, fog, dust, or sandstorm. When a vehicle collision or accident occurs on the current road, it is considered a dangerous situation.

7. The multimodal intelligent cockpit interaction control method considering priority switching as described in claim 6, characterized in that, Determining whether a dangerous situation exists in a vehicle driving scenario also includes: Identify whether the vehicle ahead on the current road is experiencing a sudden braking situation; if so, it is determined to be a dangerous situation. Identify whether there are any illegal lane changes or overtaking situations; if so, determine them as dangerous situations.

8. A multimodal intelligent cockpit interaction control system considering priority switching, characterized in that, include: The first instruction acquisition module is configured to: perform behavior recognition on the user in the smart cockpit and acquire the recognized first instruction information; The second command module is configured to: perform voice recognition on the user in the smart cockpit and obtain the recognized second command information; The priority setting module is configured to: identify vehicle driving scenarios and, based on preset judgment rules, determine whether any dangerous scenarios exist within the vehicle driving scenario. If it exists, the priority of the second instruction information is set to the highest priority, and the second instruction information is used as the final control instruction to control the vehicle; If it does not exist, the first instruction information and the second instruction information are weighted and fused to obtain the final control instruction to control the vehicle.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multimodal intelligent cockpit interaction control method that takes priority switching into account as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multimodal intelligent cockpit interaction control method that considers priority switching as described in any one of claims 1-7.

Images