A cockpit system control method based on multi-modal information interaction and a cockpit system

Abstract

The present invention discloses a cockpit system control method and a cockpit system based on multimodal information interaction, the method comprising: a vehicle-mounted host generates a driving event, and sends the driving event to a multimodal information interaction controller based on a signal switching module; the multimodal information interaction controller analyzes the driving event, generates an initial instruction, sends it to a peripheral module, and activates a gesture recognition module; the peripheral module generates an initial response based on the initial instruction; the gesture recognition module recognizes the gesture information of the driver in the cockpit, generates a gesture recognition instruction, and feeds back to the multimodal information interaction controller; the multimodal information interaction controller generates an event processing instruction based on the gesture recognition instruction, and sends it to the peripheral module; the peripheral module generates an event processing response based on the event processing instruction. The present invention processes multimodal information of devices of different interaction dimensions based on the multimodal information interaction controller, realizes multimodal intelligent interaction between the driver and the vehicle, and effectively optimizes the driver's driving experience.

Description

Technical Field

[0001] The present invention relates to the technical field of cockpit information interaction, and in particular to a cockpit system control method and a cockpit system based on multi-modal information interaction. Background Art

[0002] At present, with the progress of society and the development of science and technology, in the field of smart cars, with the improvement of smart car product technology, more and more attention is paid to the intelligent interaction between the driver and the vehicle cockpit. During the driving process of the vehicle, the driver will receive information from multi-directional, multi-modal, and different interactive dimension devices, and needs to interact and process this information. The most common intelligent interaction methods include voice interaction and screen information interaction. For voice interaction, the driver can obtain information through hearing, but when people are highly concentrated, there is a probability that they will ignore the information brought by hearing. When the driver is driving the vehicle, when a driving event occurs, the driver is also prone to ignore the information brought by hearing. If screen information interaction is used, the driver's eyes need to move to the screen and cannot continue to pay attention to the road. It is also easy to cause safety problems, affecting the safety and stability of vehicle driving. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention provides a cockpit system control method and a cockpit system based on multimodal information interaction. The multimodal information interaction controller processes the multimodal information of devices with different interaction dimensions, realizes multimodal intelligent interaction between the driver and the vehicle, and effectively optimizes the driver's driving experience.

[0004] The present invention provides a cockpit system control method based on multimodal information interaction, wherein the cockpit system based on multimodal information interaction includes a vehicle-mounted host, a multimodal information interaction controller, a gesture recognition module, a signal switching module, and a peripheral module, and the method includes:

[0005] The vehicle-mounted host generates a driving event, and sends the driving event to the multimodal information interaction controller based on the signal switching module;

[0006] The multimodal information interaction controller analyzes the driving event, generates an initial instruction, sends the initial instruction to the peripheral module, and activates the gesture recognition module;

[0007] The peripheral module generates an initial response based on the initial instruction;

[0008] The gesture recognition module recognizes the gesture information of the driver in the cockpit, generates a gesture recognition instruction, and feeds back the gesture recognition instruction to the multimodal information interaction controller;

[0009] The multimodal information interaction controller generates an event processing instruction based on the gesture recognition instruction, and sends the event processing instruction to the peripheral module;

[0010] The peripheral module generates an event processing response based on the event processing instruction.

[0011] Furthermore, the vehicle-mounted host generates a driving event, and sends the driving event to the multimodal information interaction controller based on the signal switching module, including:

[0012] The vehicle-mounted host receives a driving event establishment application and generates a driving event;

[0013] The vehicle-mounted host transfers the driving event to the first signal transfer unit in the signal transfer module arranged on one side of the vehicle-mounted host based on CAN-FD communication;

[0014] The first signal switching unit sends the driving event to a second signal switching unit in the signal switching module which is arranged on one side of the multimodal information interaction controller based on Bluetooth communication and / or Star Flash communication;

[0015] The second signal transfer unit transfers the driving event to the multimodal information interaction controller based on UART serial port communication.

[0016] Furthermore, the multimodal information interaction controller analyzes the driving event and generates an initial instruction including:

[0017] The multimodal information interaction controller analyzes the driving event to determine whether a driver in the cockpit needs to make a decision, and if it is determined that the driver needs to make a decision, generates an initial instruction based on the driving event.

[0018] Further, the peripheral module generates an initial response based on the initial instruction, including:

[0019] The dynamic LED light strip in the cockpit generates a corresponding initial response light effect based on the initial instruction;

[0020] The vibration actuator on the steering wheel generates a corresponding initial response vibration effect based on the initial instruction;

[0021] The audio equipment in the cockpit generates a corresponding initial response indication sound effect based on the initial instruction.

[0022] Furthermore, the gesture recognition module recognizes the gesture information of the driver in the cockpit, and generates the gesture recognition instruction including:

[0023] The gesture recognition module collects the driver's gesture image, obtains the spatial coordinates of the fingers, palm and wrist in each frame of the gesture image, and generates the spatial codes of the fingers, palm and wrist in each frame of the image;

[0024] Marking the time point corresponding to each frame of the image on the time axis of the gesture image to generate a time code corresponding to each frame of the image;

[0025] Based on the self-attention mechanism model, the spatial encoding of the fingers, palm and wrist in each frame of the image is combined with the time encoding corresponding to each frame of the image to generate the spatiotemporal position encoding of the gesture image;

[0026] The spatiotemporal position coding is matched with coding data in a database to obtain a gesture recognition instruction corresponding to the gesture image.

[0027] Furthermore, after the gesture recognition instruction corresponding to the gesture image is matched, the gesture recognition module stops recognizing the driver's gesture information.

[0028] Furthermore, the step of feeding back the gesture recognition instruction to the multimodal information interaction controller includes:

[0029] The gesture recognition module feeds back the gesture recognition instruction to the multimodal information interaction controller based on I²C communication.

[0030] Furthermore, the multimodal information interaction controller generates an event processing instruction based on the gesture recognition instruction, and sends the event processing instruction to the peripheral module, including:

[0031] The multimodal information interaction controller analyzes the meaning of the gesture recognition instruction in the driving event, generates a corresponding event processing instruction for the driving event, and sends the event processing instruction to the peripheral module.

[0032] Furthermore, the peripheral module generates an event processing response based on the event processing instruction, including:

[0033] The dynamic LED light strip in the cockpit generates a corresponding event processing response lighting effect based on the event processing instruction;

[0034] The vibration actuator on the steering wheel generates a corresponding event processing response vibration effect based on the event processing instruction;

[0035] The audio equipment in the cockpit generates a corresponding event processing response indication sound effect based on the event processing instruction;

[0036] The dot matrix display screen module outside the cockpit generates a corresponding event processing response image based on the event processing instruction.

[0037] The present invention also provides a cockpit system based on multimodal information interaction, the cockpit system based on multimodal information interaction comprising a vehicle-mounted host, a multimodal information interaction controller, a gesture recognition module, a first signal switching unit, a second signal, and a peripheral module, and the cockpit system based on multimodal information interaction is used to execute the above method.

[0038] The present invention provides a cockpit system control method and a cockpit system based on multimodal information interaction. The multimodal information interaction controller processes multimodal information of devices with different interaction dimensions, and adopts information interaction modes represented by light, vibration, and dot matrix display, which can give the driver richer feedback, realize multimodal intelligent interaction between the driver and the vehicle, and effectively optimize the driver's timely experience; Bluetooth or Star Flash communication is adopted between the vehicle-mounted host and the multimodal information interaction control, which has better security in addition to the advantages of wireless communication, and is not easy to be detected and cracked. The communication signal improves the safety and stability of vehicle driving; by adopting gesture recognition technology, it can be well used in intelligent driving scenarios, reduce the impact on the driver's driving process, improve the driver's operation convenience, and has high practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 is a cockpit system architecture diagram based on multimodal information interaction in Embodiment 1 of the present invention;

[0041] Figure 2 is a flow chart of a cockpit system control method based on multimodal information interaction in Embodiment 2 of the present invention;

[0042] Figure 3 is a flow chart of driving event generation in Embodiment 2 of the present invention;

[0043] Figure 4 This is a flow chart of the gesture recognition module in the second embodiment of the present invention recognizing and generating a gesture recognition instruction. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In the present invention, it should be understood that terms such as "include" or "have" are intended to indicate the existence of features, numbers, steps, behaviors, components, parts or a combination thereof disclosed in the specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts or a combination thereof exist or are added.

[0046] It should also be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] Embodiment 1

[0048] Embodiment 1 of the present invention provides a cockpit system based on multimodal information interaction, the system comprising a vehicle-mounted host, a multimodal information interaction controller, a gesture recognition module, a signal switching module, and a peripheral module.

[0049] In an optional implementation of this embodiment, Figure 1 As shown, Figure 1 The figure shows an architecture diagram of a cockpit system based on multimodal information interaction in the first embodiment of the present invention, wherein the system comprises a vehicle-mounted host, a multimodal information interaction controller, a gesture recognition module, a signal switching module, and a peripheral module, wherein the vehicle-mounted host is connected to the signal switching module, and the multimodal information interaction controller is respectively connected to the signal switching module, the gesture recognition module, and the peripheral module.

[0050] In an optional implementation of this embodiment, the vehicle-mounted host includes a central control host, an OBU, and a BCM.

[0051] Specifically, the central control host is the central controller of the vehicle, which controls all component modules in the vehicle.

[0052] Furthermore, the OBU stands for On Board Unit, which is a vehicle-mounted device used to achieve real-time communication between the vehicle and the surrounding traffic environment. It communicates and exchanges data with other vehicles, traffic infrastructure and mobile devices through on-board communication technologies (such as radio spectrum, Wi-Fi, cellular networks, Bluetooth, Star Flash Protocol, etc.). Its main functions include traffic safety alarms, traffic flow optimization and autonomous driving support.

[0053] Furthermore, the BCM is a body control module, which is an electronic control unit for controlling the body electrical system.

[0054] In an optional implementation of this embodiment, the signal switching module includes a first signal switching unit arranged on the vehicle-mounted host side, and a second signal switching unit arranged on the multimodal information interaction controller side, and the first signal switching unit and the second signal switching unit communicate via Bluetooth communication and / or StarFlash communication.

[0055] Specifically, at present, most of the control components inside the vehicle communicate with each other through wired communication, such as LIN communication or CAN communication. The wired communication network is usually limited by the layout of the wiring harness, and the layout length of the wiring harness needs to be considered. The flexibility of its layout is poor, and both LIN communication and CAN communication can detect network signals through other LIN power-saving devices or CAN node devices, thereby reverse cracking the signal; if wireless communication is adopted, the security of conventional wireless communication methods is slightly insufficient. In the sensitive field of automobile driving, it is related to the interaction and safety of oneself or even other road participants or traffic facilities and equipment, and the requirements for security are higher. Conventional wireless communication methods are easier to detect and crack, and are not suitable for driving scenarios. In this embodiment, SparkLink communication is adopted. SparkLink is a new generation of wireless short-distance communication technology with the advantages of low latency, high reliability, precise synchronization, high speed, multi-concurrency, high information security and low power consumption. It is more suitable for wireless short-distance communication within 10m-20m within the vehicle and between vehicles, effectively improving the safety and stability of driving and optimizing the driving experience of the driver.

[0056] In an optional implementation of this embodiment, the vehicle-mounted host and the first signal switching unit communicate based on a CAN-FD communication method.

[0057] In an optional implementation of this embodiment, the multimodal information interaction controller communicates with the second signal switching unit based on a UART serial port communication method.

[0058] In an optional implementation of this embodiment, the gesture recognition module communicates with the multimodal information interaction controller based on an I²C communication method.

[0059] In an optional implementation of this embodiment, the peripheral module includes a dynamic LED light strip arranged in the cockpit, a vibration actuator arranged on the steering wheel, an audio device arranged in the cockpit, and a dot matrix display screen module arranged outside the cockpit.

[0060] In an optional implementation of this embodiment, the cockpit system based on multimodal information interaction also includes a power module, which is connected to the vehicle-mounted host, the multimodal information interaction controller, and the peripheral module, and the power module is used to provide power for the system operation.

[0061] In summary, Embodiment 1 of the present invention provides a cockpit system based on multimodal information interaction, which is provided with a vehicle-mounted host, a multimodal information interaction controller, a gesture recognition module, a signal switching module, and a peripheral module. The multimodal information of devices of different interaction dimensions can be processed based on the multimodal information interaction controller, and the information interaction mode represented by light, vibration, and dot matrix display can be adopted to give the driver richer feedback, realize the multimodal intelligent interaction between the driver and the vehicle, and effectively optimize the driver's timely experience; Bluetooth or Star Flash communication is adopted between the vehicle-mounted host and the multimodal information interaction control. In addition to having the advantages of wireless communication, it has better security and is not easy to detect and crack the communication signal, thereby improving the safety and stability of vehicle driving; by setting a gesture recognition module and adopting gesture recognition technology, it can be well used in intelligent driving scenarios, reduce the impact on the driver's driving process, improve the driver's operation convenience, and has high practical application value.

[0062] Embodiment 2

[0063] Embodiment 2 of the present invention provides a cockpit system control method based on multimodal information interaction, the control method is used to control the cockpit system based on multimodal information interaction in embodiment 1, the method comprising: the vehicle-mounted host generates a driving event, and sends the driving event to the multimodal information interaction controller based on the signal switching module; the multimodal information interaction controller analyzes the driving event, generates an initial instruction, sends the initial instruction to the peripheral module, and activates the gesture recognition module; the peripheral module generates an initial response based on the initial instruction; the gesture recognition module recognizes the gesture information of the driver in the cockpit, generates a gesture recognition instruction, and feeds the gesture recognition instruction back to the multimodal information interaction controller; the multimodal information interaction controller generates an event processing instruction based on the gesture recognition instruction, and sends the event processing instruction to the peripheral module; the peripheral module generates an event processing response based on the event processing instruction.

[0064] In an optional implementation of this embodiment, Figure 2 As shown, Figure 2 The flowchart of the cockpit system control method based on multimodal information interaction in the second embodiment of the present invention is shown, which includes the following steps:

[0065] S201, the vehicle-mounted host generates a driving event, and sends the driving event to the multimodal information interaction controller based on the signal switching module;

[0066] In an optional implementation of this embodiment, Figure 3 As shown, Figure 3 The flowchart of driving event generation in the second embodiment of the present invention is shown, which includes the following steps:

[0067] S301, the vehicle-mounted host receives a driving event establishment application and generates a driving event;

[0068] In an optional implementation of this embodiment, the vehicle-mounted host receives a driving event establishment application sent by other vehicles or devices, and generates a corresponding driving event based on the application.

[0069] Specifically, if a request for overtaking is received from a rear vehicle, a driving event is established to determine whether the vehicle agrees to overtake the rear vehicle.

[0070] Furthermore, when a driving event establishment application is received from a mobile device, such as a mobile phone, a smart watch, or other mobile device, a corresponding driving event is also generated based on the application, such as making or receiving calls, and reminders of important matters.

[0071] S302, the vehicle-mounted host transfers the driving event to the first signal transfer unit in the signal transfer module arranged on the vehicle-mounted host side based on CAN-FD communication;

[0072] In an optional implementation of this embodiment, the first signal switching unit in the signal switching module is arranged on the side of the vehicle-mounted host. After the vehicle-mounted host generates a driving event, the relevant information of the driving event is transferred to the first signal switching unit based on CAN-FD communication.

[0073] Specifically, the CAN-FD communication is a new standard (CAN with Flexible Data Rate) for improving the communication rate and payload of traditional CAN communication. It integrates most of the characteristics of CAN, uses the same physical layer, two-wire serial communication protocol, and is based on non-destructive arbitration technology, distributed real-time control, etc., and makes up for the shortcomings of traditional CAN communication in terms of bus bandwidth and data length. It is applied to the communication between the vehicle-mounted host and the first signal switching unit in this embodiment, and can effectively improve the communication efficiency between the vehicle-mounted host and the first signal switching unit.

[0074] S303, the first signal switching unit sends the driving event to the second signal switching unit arranged at one side of the multimodal information interaction controller in the signal switching module based on Bluetooth communication and / or Star Flash communication;

[0075] In an optional implementation of this embodiment, the second signal switching unit of the signal switching module is arranged on one side of the multimodal information interaction controller, and the first signal switching unit sends the driving event to the second signal switching unit based on wireless communication.

[0076] Specifically, the wireless communication methods include Bluetooth communication and SparkLink communication, among which SparkLink is a new generation of wireless short-range communication technology with the advantages of low latency, high reliability, precise synchronization, high rate, multi-concurrency, high information security and low power consumption. It is more suitable for wireless short-range communication within vehicles and between vehicles within 10m-20m, effectively improving driving safety and stability, optimizing the driver's driving experience, and is applied in this embodiment to effectively improve the communication efficiency and safety between the vehicle-mounted host and the multimodal information interaction controller.

[0077] S304: The second signal transfer unit transfers the driving event to the multimodal information interaction controller based on UART serial port communication.

[0078] In an optional implementation of this embodiment, the second signal transfer unit transfers the driving event to the multimodal information interaction controller based on a serial communication method of a UART universal asynchronous receiver / transmitter.

[0079] S202, the multimodal information interaction controller analyzes the driving event, generates an initial instruction, sends the initial instruction to the peripheral module, and activates the gesture recognition module;

[0080] In an optional implementation of this embodiment, the multimodal information interaction controller analyzes the driving event to determine whether the driver in the cockpit needs to make a decision, and if it is determined that the driver needs to make a decision, an initial instruction is generated based on the driving event.

[0081] Specifically, when the multimodal information interaction controller determines that the driving event requires the driver in the cockpit to make a decision, it generates a corresponding initial instruction according to the driving event, and sends an activation instruction to the gesture recognition module to activate and start the gesture recognition module to enter a working state.

[0082] S203, the peripheral module generates an initial response based on the initial instruction;

[0083] In an optional implementation of this embodiment, the peripheral module generates a corresponding initial response according to different peripheral devices based on the initial instruction sent by the multimodal information interaction controller.

[0084] Specifically, the dynamic LED light strip in the cockpit, the vibration actuator on the steering wheel, and the audio equipment in the cockpit make corresponding initial responses.

[0085] In an optional implementation of this embodiment, the dynamic LED light strip in the cockpit generates a corresponding initial response lighting effect based on the initial instruction.

[0086] Specifically, the initial corresponding lighting effects include corresponding lighting flow effects, lighting flashing effects, lighting gradient effects, etc.

[0087] In an optional implementation of this embodiment, a vibration actuator on the steering wheel generates a corresponding initial response vibration effect based on the initial instruction.

[0088] Specifically, the vibration actuator disposed on the steering wheel generates a vibration effect with a vibration intensity of 2G or more.

[0089] In an optional implementation of this embodiment, the audio equipment in the cockpit generates a corresponding initial response indication sound effect based on the initial instruction.

[0090] Specifically, the audio equipment installed in the cockpit generates an indication sound corresponding to the initial instruction, specifically prompting the driver of the details of the driving event that needs to be handled.

[0091] S204, the gesture recognition module recognizes the gesture information of the driver in the cockpit, generates a gesture recognition instruction, and feeds back the gesture recognition instruction to the multimodal information interaction controller;

[0092] In an optional implementation of this embodiment, the gesture recognition module recognizes the gesture information of the driver in the cockpit according to actual setting requirements.

[0093] Specifically, when set to simple mode, the recognition principle of the gesture recognition module is to detect the direction of hand movement through infrared light of a specific wavelength. When the hand is swung in different directions, the infrared light in the corresponding direction is reflected, generating infrared light reflection with direction information, thereby identifying the direction of hand movement. When the hand is identified to move to the left, it means rejection, and when the hand is identified to move to the right, it means consent.

[0094] Furthermore, when set to complex mode, the recognition principle of the gesture recognition module is to capture the driver's hand movement image, and then analyze the specific gesture to obtain more specific gesture instructions, such as clockwise rotation, counterclockwise rotation, left and right reciprocating swing, etc.

[0095] In an optional implementation of this embodiment, Figure 4 As shown, Figure 4The flowchart of the gesture recognition module in the second embodiment of the present invention identifying and generating a gesture recognition instruction is shown, including the following steps:

[0096] S401, the gesture recognition module collects the driver's gesture image, obtains the spatial coordinates of the fingers, palm and wrist in each frame of the gesture image, and generates the spatial codes of the fingers, palm and wrist in each frame of the image;

[0097] In an optional implementation of this embodiment, the gesture recognition module collects gesture images of the driver and obtains the spatial coordinates of the fingers, palm and wrist in each frame of the image.

[0098] Specifically, the joints of the hand are used as positioning points. For example, a single palm has four joints in each of the five fingers, including the top, upper middle, lower middle, and bottom. Four positioning points are set, and a total of 20 positioning points are set for the five fingers. One is set at the intersection of the extension lines of the index and middle fingers on the palm, one is set at the intersection of the extension lines of the ring finger and the last one is set at the wrist. A total of 23 positioning points are set. After obtaining the spatial coordinates of these 23 positioning points, they are compiled into a data set. , , , is a positive integer, Refers to Frame image, Refers to Anchor points, ,in,( ) represents the spatial coordinates of the top, upper middle, lower middle, and bottom of the thumb, ( ) represents the spatial coordinates of the top, upper middle, lower middle, and bottom of the index finger, ( ) represents the spatial coordinates of the top, upper middle, lower middle, and bottom of the middle finger, ( ) represents the spatial coordinates of the top, upper middle, lower middle, and bottom of the ring finger, ( ) represents the spatial coordinates of the top, upper middle, lower middle, and bottom of the little finger, ( ) represents the spatial coordinates of the intersection of the extended lines of the index and middle fingers on the palm and the intersection of the extended lines of the ring finger and the last finger on the palm, ( ) represents the spatial coordinates of the wrist.

[0099] In an optional implementation of this embodiment, after the spatial coordinates are acquired, two-dimensional spatial coding is performed on them to generate corresponding spatial coding, and the calculation formula includes:

[0100] ;

[0101] In the formula, For the Frame image The spatial encoding of the anchor points, For the Frame image The spatial coordinates of the positioning points, For the Frame image The unit space vector of the anchor point, is the average size of the grid cells in two-dimensional space, is the maximum value of the grid cell in two-dimensional space divided by the minimum value, is the total number of grid cells in two-dimensional space, , is a positive integer. In this embodiment, .

[0102] S402, marking the time point corresponding to each frame of the image on the time axis of the gesture image to generate a time code corresponding to each frame of the image;

[0103] In an optional implementation of this embodiment, the position of each frame of image in the time axis of the gesture image is extracted and annotated to generate a time code corresponding to the frame of image.

[0104] Specifically, set The time code corresponding to the frame image is .

[0105] S403, based on the self-attention mechanism model, the spatial coding of the fingers, palm and wrist in each frame of the image is combined with the time coding corresponding to each frame of the image to generate the spatiotemporal position coding of the gesture image;

[0106] In an optional implementation of this embodiment, a self-attention mechanism model is constructed based on the preset correlation change information between different positioning points in the palm, and the spatial encoding of the fingers, palm and wrist in each frame image is merged with the time encoding corresponding to each frame image to generate the spatiotemporal position encoding of each frame image.

[0107] Specifically, a spatial map specialized by spatial coding and a temporal map specialized by temporal coding in each frame of image are generated based on the self-attention mechanism model, feature information is extracted, and the feature information is merged to generate a spatiotemporal map of each frame of image, and the spatiotemporal unit vector in the spatiotemporal map is extracted, and the spatiotemporal unit vector is combined with the spatial coding and the temporal coding to generate the spatiotemporal position coding of each frame of image. The calculation formula includes:

[0108] ;

[0109] ;

[0110] In the formula, For the Frame image The spatiotemporal position encoding of the anchor points, is the space-time unit vector, For the Frame image The spatial encoding of the anchor points, For the The time code corresponding to the frame image, is the spatiotemporal position encoding of the gesture image, , is a positive integer. In this embodiment, .

[0111] S404: Match the spatiotemporal position code with the code data in the database to obtain a gesture recognition instruction corresponding to the gesture image.

[0112] In an optional implementation of this embodiment, the spatiotemporal position code obtained in step S403 is matched with the code data in the database to match the gesture recognition instruction corresponding to the gesture image.

[0113] It should be noted that, after the gesture recognition instruction corresponding to the gesture image is matched, the gesture recognition module stops recognizing the driver's gesture information.

[0114] Specifically, after matching the corresponding gesture recognition instruction, the gesture recognition module generates a recognition interrupt signal and stops recognizing the driver's gesture information.

[0115] In an optional implementation of this embodiment, the gesture recognition module feeds back the gesture recognition instruction to the multimodal information interaction controller based on I²C communication.

[0116] S205, the multimodal information interaction controller generates an event processing instruction based on the gesture recognition instruction, and sends the event processing instruction to the peripheral module;

[0117] In an optional implementation of this embodiment, the multimodal information interaction controller analyzes the meaning of the gesture recognition instruction in the driving event, generates an event processing instruction corresponding to the driving event, and sends the event processing instruction to the peripheral module.

[0118] S206: The peripheral module generates an event processing response based on the event processing instruction.

[0119] In an optional implementation of this embodiment, the peripheral module generates corresponding event processing responses according to different peripheral devices based on the event processing instructions sent by the multimodal information interaction controller.

[0120] Specifically, the dynamic LED light strip in the cockpit, the vibration actuator on the steering wheel, the audio equipment in the cockpit, and the dot matrix display screen module outside the cockpit perform corresponding event processing responses.

[0121] In an optional implementation of this embodiment, the dynamic LED light strip in the cockpit generates a corresponding event processing response lighting effect based on the initial instruction.

[0122] Specifically, the event processing response lighting effects include corresponding lighting flow effects, lighting flashing effects, lighting gradient effects, etc.

[0123] In an optional implementation of this embodiment, a vibration actuator on the steering wheel generates a corresponding event processing response vibration effect based on the event processing instruction.

[0124] Specifically, the vibration actuator disposed on the steering wheel generates a vibration effect with a vibration intensity of 2G or more.

[0125] In an optional implementation of this embodiment, the audio equipment in the cockpit generates a corresponding event processing response indication sound effect based on the event processing instruction.

[0126] Specifically, the audio equipment installed in the cockpit generates an indication sound corresponding to the event processing instruction, specifically prompting the driver to complete the details of the driving event for the decision.

[0127] In an optional implementation of this embodiment, the dot matrix display screen module outside the cockpit generates a corresponding event processing response image based on the event processing instruction.

[0128] Specifically, a dot matrix display screen module outside the cockpit is arranged to generate a corresponding event processing response image, and feed back the event processing response to other drivers.

[0129] In an optional implementation of the present embodiment, in a specific driving event, such as when a vehicle 10 meters behind the vehicle makes a request to overtake, the on-board host generates an overtaking request driving event, and the multimodal information interaction controller determines that the driver needs to make a decision. At this time, the dynamic LED light strip of the peripheral module triggers the corresponding flow effect, the vibration actuator on the steering wheel vibrates, the audio device generates an overtaking request prompt sound, and the gesture recognition module is activated to start recognizing the driver's gesture information. When the driver's gesture is recognized to move to the left, indicating a refusal to overtake, the dynamic LED light strip generates an orange-red light flowing to the left, the audio device generates a prompt sound of disagreeing with the overtaking request, and the dot matrix display screen module displays the corresponding rejection image. When the driver's gesture is recognized to move to the right, indicating an agreement to overtake, the dynamic LED light strip generates a green light flowing to the right, the audio device generates a prompt sound of agreeing to the overtaking request, and the dot matrix display screen module displays the corresponding agreement image.

[0130] In an optional implementation of this embodiment, if it rains on the driving road, the road test unit RSU sends a request to close the windows, the on-board host generates a driving event of requesting to close the windows, the multimodal information interaction controller determines that the driver does not need to make a decision, and automatically closes the windows through the BCM body controller.

[0131] In summary, the second embodiment of the present invention provides a cockpit system control method based on multimodal information interaction, which is used to control the cockpit system based on multimodal information interaction in the first embodiment. The multimodal information of devices of different interaction dimensions is processed based on the multimodal information interaction controller, and the information interaction mode represented by light, vibration, and dot matrix display is adopted, which can give the driver richer feedback, realize the multimodal intelligent interaction between the driver and the vehicle, and effectively optimize the driver's timely experience; Bluetooth or Star Flash communication is adopted between the vehicle-mounted host and the multimodal information interaction control. In addition to the advantages of wireless communication, it has better security and is not easy to detect and crack the communication signal, thereby improving the safety and stability of vehicle driving; by adopting gesture recognition technology, it can be well used in intelligent driving scenarios, reduce the impact on the driver's driving process, improve the driver's operation convenience, and has high practical application value.

[0132] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0133] In addition, the embodiments of the present invention are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A cockpit system control method based on multimodal information interaction, characterized in that: The cockpit system based on multimodal information interaction includes a vehicle-mounted host, a multimodal information interaction controller, a gesture recognition module, a signal switching module, and a peripheral module. The method includes: The vehicle-mounted host generates a driving event, and sends the driving event to the multimodal information interaction controller based on the signal switching module; The vehicle-mounted host generates a driving event, and sends the driving event to the multimodal information interaction controller based on the signal transfer module, including: the vehicle-mounted host receives a driving event establishment application and generates a driving event; the vehicle-mounted host transfers the driving event to a first signal transfer unit in the signal transfer module arranged on the vehicle-mounted host side based on CAN-FD communication; the first signal transfer unit sends the driving event to a second signal transfer unit in the signal transfer module arranged on the multimodal information interaction controller side based on Bluetooth communication and / or Star Flash communication; the second signal transfer unit transfers the driving event to the multimodal information interaction controller based on UART serial port communication; The multimodal information interaction controller analyzes the driving event, generates an initial instruction, sends the initial instruction to the peripheral module, and activates the gesture recognition module; The peripheral module generates an initial response based on the initial instruction; The gesture recognition module recognizes the gesture information of the driver in the cockpit, generates a gesture recognition instruction, and feeds back the gesture recognition instruction to the multimodal information interaction controller; The gesture recognition module recognizes the gesture information of the driver in the cockpit, and generates the gesture recognition instruction, including: the gesture recognition module collects the gesture image of the driver, obtains the spatial coordinates of the fingers, palms and wrists in each frame of the gesture image, and generates the spatial coding of the fingers, palms and wrists in each frame; marks the time point corresponding to each frame on the time axis of the gesture image, and generates the time coding corresponding to each frame; based on the self-attention mechanism model, the spatial coding of the fingers, palms and wrists in each frame is combined with the time coding corresponding to each frame, and the spatiotemporal position coding of the gesture image is generated; the spatiotemporal position coding is matched with the coding data in the database, and the gesture recognition instruction corresponding to the gesture image is matched; The generating of the spatial coding of the fingers, palm and wrist in each frame of the image comprises: after acquiring the spatial coordinates of the fingers, palm and wrist in each frame of the gesture image, performing two-dimensional spatial coding to generate corresponding spatial coding, the calculation formula comprises: ； In the formula, For the Frame image The spatial encoding of the anchor points, For the Frame image The spatial coordinates of the positioning points, For the Frame image The unit space vector of the anchor point, is the average size of the grid cells in two-dimensional space, is the maximum value of the grid cell in two-dimensional space divided by the minimum value, is the total number of grid cells in two-dimensional space, , is a positive integer; The step of marking the time point corresponding to each frame of the image on the time axis of the gesture image to generate the time code corresponding to each frame of the image comprises: Set The time code corresponding to the frame image is ; The self-attention mechanism model combines the spatial encoding of fingers, palms and wrists in each frame of the image with the time encoding corresponding to each frame of the image to generate the spatiotemporal position encoding of the gesture image, including: Based on the self-attention mechanism model, a spatial map specialized by spatial coding and a temporal map specialized by temporal coding are generated in each frame of the image, and the feature information is extracted. After merging the feature information, a spatiotemporal map of each frame of the image is generated, and the spatiotemporal unit vector in the spatiotemporal map is extracted. The spatiotemporal unit vector is combined with the spatial coding and the temporal coding to generate the spatiotemporal position coding of each frame of the image. The calculation formula includes: ； ； In the formula, For the Frame image The spatiotemporal position encoding of the anchor points, is the space-time unit vector, For the Frame image The spatial encoding of the anchor points, For the The time code corresponding to the frame image, is the spatiotemporal position encoding of the gesture image, , is a positive integer; The multimodal information interaction controller generates an event processing instruction based on the gesture recognition instruction, and sends the event processing instruction to the peripheral module; The peripheral module generates an event processing response based on the event processing instruction.

2. The cockpit system control method based on multimodal information interaction according to claim 1, characterized in that: The multimodal information interaction controller analyzes the driving event and generates an initial instruction, including: The multimodal information interaction controller analyzes the driving event to determine whether a driver in the cockpit needs to make a decision, and if it is determined that the driver needs to make a decision, generates an initial instruction based on the driving event.

3. The cockpit system control method based on multimodal information interaction according to claim 1, characterized in that: The peripheral module generating an initial response based on the initial instruction includes: The dynamic LED light strip in the cockpit generates a corresponding initial response light effect based on the initial instruction; The vibration actuator on the steering wheel generates a corresponding initial response vibration effect based on the initial instruction; The audio equipment in the cockpit generates a corresponding initial response indication sound effect based on the initial instruction.

4. The cockpit system control method based on multimodal information interaction according to claim 1, characterized in that: When the gesture recognition instruction corresponding to the gesture image is matched, the gesture recognition module stops recognizing the driver's gesture information.

5. The cockpit system control method based on multimodal information interaction according to claim 1, characterized in that: Feeding back the gesture recognition instruction to the multimodal information interaction controller includes: The gesture recognition module feeds back the gesture recognition instruction to the multimodal information interaction controller based on I²C communication.

6. The cockpit system control method based on multimodal information interaction according to claim 1, characterized in that: The multimodal information interaction controller generates an event processing instruction based on the gesture recognition instruction, and sends the event processing instruction to the peripheral module, including: The multimodal information interaction controller analyzes the meaning of the gesture recognition instruction in the driving event, generates a corresponding event processing instruction for the driving event, and sends the event processing instruction to the peripheral module.

7. The cockpit system control method based on multimodal information interaction according to claim 1, characterized in that: The peripheral module generating an event processing response based on the event processing instruction includes: The dynamic LED light strip in the cockpit generates a corresponding event processing response lighting effect based on the event processing instruction; The vibration actuator on the steering wheel generates a corresponding event processing response vibration effect based on the event processing instruction; The audio equipment in the cockpit generates a corresponding event processing response indication sound effect based on the event processing instruction; The dot matrix display screen module outside the cockpit generates a corresponding event processing response image based on the event processing instruction.

8. A cockpit system based on multimodal information interaction, characterized in that: The cockpit system based on multimodal information interaction includes an on-board host, a multimodal information interaction controller, a gesture recognition module, a first signal switching unit, a second signal, and a peripheral module. The cockpit system based on multimodal information interaction is used to execute the method described in any one of claims 1 to 7.

Images