A vehicle cabin interaction method and device, a storage medium and an equipment

Abstract

The application provides a vehicle cabin interaction method and device, a storage medium and equipment. In the method, multi-modal data of a driver's hand region is collected, gesture features are extracted from the multi-modal data, the gesture features are processed based on a gesture recognition model to obtain a gesture category, a cabin interaction instruction is generated according to the recognized gesture category, the cabin interaction instruction is sent to a cabin subsystem to drive it to perform corresponding operations, and vehicle cabin interaction based on gesture control is realized. In addition to image data, the multi-modal data also includes radar data and / or depth data, so that gesture features can be effectively captured even in extreme light and when the hands are blocked. In this way, the gesture recognition accuracy and interaction reliability can be improved.

Description

Technical Field

[0001] This application relates to the field of automotive control technology, and more specifically, to an automotive cockpit interaction method, device, storage medium, and equipment. Background Technology

[0002] As the level of automotive intelligence continues to increase, the innovation and optimization of human-machine interaction methods have become crucial for improving user experience. Currently, smart cockpits are gradually shifting from traditional mechanical interaction to multimodal interaction methods such as touch and voice. In practical applications, while touch interaction is intuitive, the vibrations of a moving vehicle can easily lead to accidental touches, and complex functions require multiple levels of operation, causing the driver's gaze to deviate from the road for extended periods, posing a safety hazard. While voice interaction can free up the hands, limitations in voice recognition and semantic understanding technologies result in inconsistent user experiences, and both interaction methods lack sufficient intelligence. Among related technologies, some smart cockpit systems can improve the intelligence of smart cockpit interaction by recognizing the facial expressions of passengers and adjusting the interaction methods accordingly. However, this method relies on cameras capturing clear and complete frontal facial images, which is easily affected by factors such as lighting and facial occlusion, leading to low recognition accuracy and consequently affecting the precision of interaction decisions. Summary of the Invention

[0003] The purpose of this application is to provide an automotive cockpit interaction method, device, storage medium, and equipment, aiming to solve the problem that the intelligent cockpit interaction methods in related technologies rely on cameras to capture clear and complete frontal facial images, which are easily affected by factors such as lighting and facial occlusion, resulting in low recognition accuracy and thus affecting the accuracy of interaction decisions.

[0004] In a first aspect, this application provides a method for automotive cockpit interaction, comprising: collecting multimodal data of the driver's hand area; the multimodal data including image data and at least one of the following: radar data and depth data; extracting gesture features from the multimodal data, processing the gesture features based on a gesture recognition model to obtain a gesture category; generating cockpit interaction commands according to the gesture category, and controlling the cockpit subsystem of the automotive cockpit according to the cockpit interaction commands.

[0005] In the above implementation process, multimodal data of the driver's hand area is collected, gesture features are extracted from the multimodal data, and the gesture features are processed based on a gesture recognition model to obtain the gesture category. Then, a cockpit interaction command is generated based on the recognized gesture category and sent to the cockpit subsystem to drive it to perform the corresponding operation, thereby realizing gesture-based car cockpit interaction. The multimodal data includes not only image data but also radar data and / or depth data. Therefore, even in extreme lighting conditions or when the hand is obstructed, gesture features can be effectively captured, thus improving gesture recognition accuracy and interaction reliability.

[0006] Furthermore, in some examples, before extracting gesture features from the multimodal data, the process includes: preprocessing the multimodal data; and spatiotemporally aligning the preprocessed multimodal data.

[0007] In the above implementation process, after collecting the raw data of the driver's hand area, the quality of the raw data can be improved through preprocessing and spatiotemporal alignment, laying a good data foundation for subsequent accurate recognition of gesture categories.

[0008] Furthermore, in some examples, the image data includes an image sequence; the extraction of gesture features from the multimodal data includes: extracting a first gesture feature from the image sequence; the first gesture feature represents the spatial coordinates and temporal sequence changes of key points of the hand; extracting a second gesture feature from the radar data and / or depth data; the second gesture feature includes spatial static features representing hand contour and distance information, and temporal dynamic features representing hand movement speed and direction.

[0009] In the above implementation process, during the feature extraction stage, key hand points, such as the spatial coordinates of joints and their temporal changes, can be extracted from the image sequence. Furthermore, features such as the hand's contour shape, movement speed, movement direction, and distance information to the sensor can be extracted from radar data and / or depth data. Thus, by extracting the spatial static features and temporal dynamic features of the gesture, the driver's current hand gesture can be accurately identified.

[0010] Furthermore, in some examples, the process of processing the gesture features based on the gesture recognition model to obtain the gesture category includes: weighted fusion of the first gesture feature and the second gesture feature to obtain a fused feature vector; the weights of the first gesture feature and the second gesture feature are determined based on the image quality of the image sequence; and the fused feature vector is input into the gesture recognition model to obtain the gesture category output by the gesture recognition model.

[0011] In the above implementation process, after feature extraction is completed, a dynamic weighting strategy is used to fuse feature vectors from different sensors, thereby effectively improving the accuracy of gesture recognition.

[0012] Furthermore, in some examples, the gesture category includes static gesture categories and dynamic gesture trajectory categories; generating cockpit interaction commands based on the gesture category includes: mapping the gesture category to cockpit interaction commands according to a preset mapping rule library; the mapping rule library records cockpit interaction commands corresponding to different static gesture categories and cockpit interaction commands corresponding to different dynamic gesture trajectory categories.

[0013] In the above implementation process, the gesture recognition model recognizes not only static gestures but also dynamic gestures, which makes the number of mappable interactive commands greater, thereby enabling a richer and more intuitive set of interactive commands and improving the accuracy of control.

[0014] Furthermore, in some examples, the method also includes: obtaining the gesture recognition confidence score output by the gesture recognition model; if the gesture recognition confidence score is less than the confidence score threshold, determining the cockpit interaction command as an invalid command, and notifying the driver to re-enter the gesture.

[0015] In the above implementation process, by introducing a confidence judgment mechanism, low confidence recognition results caused by brief interference or invalid actions are automatically filtered out, thereby effectively reducing the probability of false triggering and improving the reliability of interaction.

[0016] Furthermore, in some examples, it also includes: providing multi-channel feedback to the driver based on the execution result of the cockpit interaction command; the multi-channel feedback includes at least two of the following: visual feedback, auditory feedback, and vibration feedback.

[0017] In the above implementation process, after the command is sent, operation confirmation or status prompts are provided in real time through at least two of the following: visual feedback, auditory feedback, and vibration feedback. This allows the driver to clearly perceive whether the command has been successfully received and executed, thereby improving the user experience.

[0018] Secondly, this application provides an automotive cockpit interaction device, comprising: a data acquisition module for acquiring multimodal data of the driver's hand area; the multimodal data including image data and at least one of the following: radar data and depth data; a recognition module for extracting gesture features from the multimodal data, processing the gesture features based on a gesture recognition model to obtain a gesture category; and a control module for generating cockpit interaction commands according to the gesture category, and controlling the cockpit subsystem of the automotive cockpit according to the cockpit interaction commands.

[0019] Thirdly, this application provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method described in any of the first aspects.

[0020] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in any of the first aspects.

[0021] Fifthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the method described in any of the first aspects.

[0022] Other features and advantages disclosed in this application will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the above-described technology disclosed in this application.

[0023] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating an automotive cockpit interaction method provided in this application embodiment; Figure 2 A schematic diagram illustrating the architecture of a gesture-controlled intelligent cockpit interaction system provided in this application embodiment; Figure 3 A block diagram of an automotive cockpit interaction device provided in an embodiment of this application; Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0027] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] With the rapid advancement of automotive technology and the increasing demand from consumers for intelligent vehicles, in-vehicle infotainment has become a significant trend in the automotive industry. This trend is particularly evident in cockpit interaction methods. Intelligent cockpits, by integrating technologies such as artificial intelligence and the Internet of Things, upgrade the traditional car cockpit into a mobile intelligent space with sensing, interaction, and service capabilities. Currently, intelligent cockpits are gradually shifting from traditional mechanical interaction to multimodal interaction methods such as touch and voice. In practical applications, while touch interaction is intuitive, the bumps and vibrations during vehicle movement can easily lead to accidental touches, and complex functions require multi-level operations, causing the driver's gaze to deviate from the road for extended periods, posing a safety hazard. While voice interaction can free up the hands, limitations in voice recognition and semantic understanding technologies result in inconsistent user experiences, and both of these interaction methods lack sufficient intelligence. Among related technologies, some intelligent cockpit systems can recognize the facial expressions of drivers and passengers to adjust the interaction methods, thereby improving the intelligence of intelligent cockpit interaction. However, this interaction method relies on the camera to capture clear and complete frontal facial images. When there is insufficient light or the driver is wearing a mask, the system cannot effectively extract facial features, resulting in low recognition accuracy and thus affecting the accuracy of interaction decisions.

[0029] To address the aforementioned issues, this application provides an automotive cockpit interaction solution. It collects multimodal data of the driver's hand area, extracts gesture features from the multimodal data, processes these features based on a gesture recognition model to determine gesture categories, and then generates cockpit interaction commands based on the identified gesture categories to control the corresponding cockpit subsystems for cockpit interaction. The multimodal data includes not only image data but also radar data and / or depth data, enabling effective capture of gesture features even in extreme lighting conditions or when the hand is obscured, thus improving gesture recognition accuracy and interaction reliability.

[0030] The embodiments of this application will be described below: like Figure 1 As shown, Figure 1 This is a flowchart of an automotive cockpit interaction method provided in an embodiment of this application. The automotive cockpit mentioned in this embodiment can refer to a smart cockpit, and the method can be applied to a smart cockpit interaction system, which includes a sensor group arranged in the cockpit and a cockpit domain controller, etc.

[0031] The method includes: Step 101: Collect multimodal data of the driver's hand area; the multimodal data includes image data and at least one of the following: radar data and depth data; The multimodal data mentioned in this step includes not only image data but also radar data and / or depth data. Radar data can include radar reflection signals and / or point cloud data, while depth data can include depth maps or 3D point clouds. In implementation, multimodal data can be acquired using at least two different types of sensors. Optionally, image data can be acquired using a visible light camera, radar data can be acquired using millimeter-wave radar, and depth data can be acquired using a depth sensor. Millimeter-wave radar and depth sensors are unaffected by visible light intensity, and even when the driver's hands are partially obscured by the steering wheel, clothing, or other objects, the strong penetrating power of millimeter-wave radar or the contour perception capability of the depth sensor can still effectively capture the main motion features of the gestures.

[0032] In addition, in some embodiments, an event camera can be used instead of a visible light camera. An event camera is a new type of bionic vision sensor that achieves microsecond-level response by asynchronously detecting changes in pixel brightness. It has advantages such as high dynamic range and low power consumption. Therefore, using an event camera to capture pixel changes in hand movement can achieve effects such as ultra-high-speed response and resistance to strong light.

[0033] Step 102: Extract gesture features from the multimodal data, process the gesture features based on the gesture recognition model, and obtain the gesture category; The gesture recognition model mentioned in this step can be a pre-trained neural network model, which can be trained based on a gesture recognition dataset. This dataset can contain multimodal data of different gestures and gesture category labels. During training, the gesture recognition model can learn to identify the corresponding gesture category based on the gesture features extracted from the multimodal data.

[0034] In some embodiments, this step may precede: preprocessing the multimodal data; and spatiotemporally aligning the preprocessed multimodal data. That is, after acquiring multimodal data of the driver's hand area, preprocessing can be performed to improve the quality of the original data and provide a reliable foundation for subsequent analysis. In implementation, for image data in the multimodal data, the preprocessing process may include denoising, illumination compensation, and ROI (Region of Interest) segmentation; while for radar data and / or depth data, the preprocessing process may include noise filtering, point cloud clustering, and preliminary motion trajectory calculation. After preprocessing, timestamp alignment and spatial coordinate system unification can be performed on data acquired from different sensors at the same time, thereby establishing a precise temporal correspondence and spatial mapping relationship between the multimodal data. In implementation, the timestamp of one sensor can be used as a reference. Data from other sensors is then interpolated based on time offset and clock drift rate to generate data aligned with the reference timestamp, thus achieving time alignment. Using calibrated extrinsic parameters, all modal data are transformed into a unified cockpit coordinate system, achieving spatial alignment. These extrinsic parameters describe the transformation relationship between the sensor's own coordinate system and the cockpit coordinate system, typically including a rotation matrix and a translation vector. Thus, through preprocessing and spatiotemporal alignment, a solid data foundation can be laid for subsequent accurate gesture category recognition.

[0035] In some embodiments, the aforementioned image data may include image sequences; the extraction of gesture features from the multimodal data mentioned in this step may include: extracting a first gesture feature from the image sequence; the first gesture feature represents the spatial coordinates and temporal changes of key hand points; and extracting a second gesture feature from the radar data and / or depth data; the second gesture feature includes spatial static features representing hand contours and distance information, and temporal dynamic features representing hand movement speed and direction. That is, in the feature extraction stage, key hand points, such as the spatial coordinates of joints and their temporal changes, can be extracted from the image sequence. Furthermore, features such as the hand contour shape, movement speed, movement direction, and distance information with the sensor can be extracted from the radar data and / or depth data. Thus, by extracting the spatial static and temporal dynamic features of the gesture, the driver's current gesture can be accurately identified. Extracting target contours and target movement speeds from radar data and / or depth data are mature technologies and will not be elaborated upon here.

[0036] Further, in some embodiments, the processing of the gesture features based on the gesture recognition model mentioned in this step to obtain the gesture category may include: weighted fusion of the first gesture feature and the second gesture feature to obtain a fused feature vector; the weights of the first gesture feature and the second gesture feature are determined based on the image quality of the image sequence; the fused feature vector is input into the gesture recognition model to obtain the gesture category output by the gesture recognition model. That is, after feature extraction, a dynamic weighting strategy can be used to fuse feature vectors from different sensors. Specifically, the weights can be determined based on the image quality of the image sequence. When the image quality is poor, the weight of the first gesture feature can be reduced, so that the fused feature vector focuses on the precise motion trajectory and distance information provided by millimeter-wave radar and / or depth sensors. In implementation, the image quality of the image sequence can be evaluated using at least one of the following metrics: signal-to-noise ratio, local contrast, sharpness, pixel oversaturation ratio, and inter-frame difference. Then, the fused feature vector is input into the gesture recognition model, which identifies the gesture category corresponding to the driver's current hand based on the fused feature vector. This effectively improves the gesture recognition accuracy. In other embodiments, the weights of the first gesture feature and the second gesture feature can also be determined based on the judgment result of whether the radar data is affected by metal interference.

[0037] Optionally, the gesture recognition model may include an ST-GCN (Spatial Temporal Graph Convolutional Networks) model. The ST-GCN model is a deep learning model for processing data with spatiotemporal structure. In its implementation, the ST-GCN model constructs a spatiotemporal graph based on the fused feature vectors. For example, it uses the spatial coordinates of 21 key points on the hand to establish nodes in the graph and defines edges based on the natural connections of the hand. Then, the ST-GCN model transforms the spatiotemporal graph into a high-level feature map through a multi-layer spatiotemporal graph convolutional module, and finally uses a Softmax classifier for classification to obtain the gesture category. Of course, in other embodiments, the gesture recognition model can also be other types of deep learning models, such as 3D-CNN (3D-Convolutional Neural Networks) models, SVM (Support Vector Machine) models, etc. For specific model structures and training processes, please refer to the descriptions of the corresponding algorithms in related technologies; this application will not elaborate further.

[0038] Step 103: Generate cockpit interaction commands based on the gesture category, and control the cockpit subsystem of the vehicle cockpit based on the cockpit interaction commands.

[0039] In this embodiment, the driver displays a gesture, the system collects multimodal data of the driver's hand area, identifies the gesture category using a gesture recognition model, maps the gesture category to a cockpit interaction command, and then sends the cockpit interaction command to the corresponding cockpit subsystem, such as the infotainment system, air conditioning control system, and window control system, to drive them to perform corresponding operations, such as playing music, adjusting the temperature, or opening and closing windows. This achieves intelligent cockpit interaction based on gesture control.

[0040] In some embodiments, the gesture categories mentioned in this step may include static gesture categories and dynamic gesture trajectory categories. Correspondingly, generating cockpit interaction commands based on the gesture categories mentioned in this step may include: mapping the gesture categories to cockpit interaction commands according to a preset mapping rule library; the mapping rule library records cockpit interaction commands corresponding to different static gesture categories and cockpit interaction commands corresponding to different dynamic gesture trajectory categories. That is, the output of the gesture recognition model includes static gesture categories, such as raising a thumb or clenching a fist, and also includes dynamic gesture trajectory categories, such as drawing circles or pinching fingers. The system pre-sets cockpit interaction commands corresponding to various static gesture categories and dynamic gesture trajectory categories, such as sliding to adjust volume, pushing the palm outward to reject a call, and clenching a fist to make an emergency call, thus forming a mapping rule library. After recognizing a gesture category, the system maps the recognized gesture category to a specific cockpit interaction command according to this mapping rule library. In this way, the user experience is closer to natural physical operation, thereby improving the user experience and reducing the user's learning cost.

[0041] In practical applications, a driver's hand may unintentionally brush past the sensor, easily triggering a misoperation. Therefore, some embodiments may further include: obtaining the gesture recognition confidence score output by the gesture recognition model; if the gesture recognition confidence score is less than a confidence threshold, the cockpit interaction command is determined to be an invalid command, and the driver is notified to re-enter the gesture. That is, the output of the gesture recognition model also includes the gesture recognition confidence score, which represents the reliability of the gesture category recognition result. When the gesture recognition confidence score is greater than or equal to the confidence threshold, it indicates that the gesture category recognition result is reliable, and the cockpit subsystem executes the corresponding operation according to the cockpit interaction command. When the gesture recognition confidence score is less than the confidence threshold, it indicates that the gesture category recognition result is unreliable, and the generated cockpit interaction command is determined to be an invalid command, the cockpit subsystem does not execute the corresponding operation, and the user is notified to repeat the gesture. Thus, by introducing a confidence score judgment mechanism, low-confidence recognition results caused by brief interference or invalid actions are automatically filtered out, thereby effectively reducing the probability of false triggers and improving interaction reliability.

[0042] In addition, some embodiments may further include: providing multi-channel feedback to the driver based on the execution result of the cockpit interaction command; the multi-channel feedback includes at least two of the following: visual feedback, auditory feedback, and vibration feedback. That is, after the command is sent, operation confirmation or status prompts are provided in real time through at least two of the visual, auditory, and vibration feedback, so that the driver can clearly perceive whether the command has been successfully received and executed, thereby improving the user experience.

[0043] In this embodiment, multimodal data of the driver's hand area is collected, gesture features are extracted from the multimodal data, and the gesture features are processed based on a gesture recognition model to obtain gesture categories. Then, cockpit interaction commands are generated based on the recognized gesture categories and sent to the cockpit subsystem to drive it to perform corresponding operations, thereby realizing gesture-based car cockpit interaction. The multimodal data includes not only image data but also radar data and / or depth data, which can effectively capture gesture features even in extreme lighting conditions or when the hand is obscured, thus improving gesture recognition accuracy and interaction reliability.

[0044] To provide a more detailed explanation of the solution in this application, a specific embodiment is described below: This embodiment provides a gesture-based intelligent cockpit interaction system, the architecture of which is as follows: Figure 2 As shown, the system includes a gesture acquisition module 21, a data processing module 22, an interaction control module 23, and a feedback module 24. The gesture acquisition module 21 is a sensor group deployed in the smart cockpit, which may include a camera, millimeter-wave radar, and depth sensor, etc. This embodiment uses a camera combined with millimeter-wave radar as an example. The data processing module 22 may be a processor integrated into the cockpit domain controller or a dedicated computing unit and the gesture recognition software running on it. The interaction control module 23 is a controller that communicates with the cockpit subsystem 25. The feedback module 24 is a device that provides visual feedback, auditory feedback, and tactile feedback.

[0045] The system's workflow includes: S201. The gesture acquisition module 21 captures raw data of the driver's hand area, including image sequences, point cloud data and radar reflection signals. S202. The data processing module 22 preprocesses the collected raw data and performs spatiotemporal alignment on the preprocessed data. Specifically, in implementation, for image sequences, a preprocessing process of denoising, illumination compensation, and ROI segmentation is performed; for point cloud data and radar reflection signals, a preprocessing process of noise filtering, point cloud clustering, and preliminary calculation of motion trajectory is performed; then, data collected by different sensors at the same time are time-stamp aligned and spatial coordinate system unified. S203, The gesture category is identified by the data processing module 22; Specifically, key hand points, such as the spatial coordinates of joints and their temporal changes, are extracted from the preprocessed image sequence. Physical features such as the outline shape, movement speed, movement direction, and distance to the sensor are extracted from the preprocessed point cloud data and radar reflection signals. Then, the feature vectors from different sensors are fused. Under normal conditions, the feature vectors from different sensors are used to perform complementary verification. When the image quality of the image sequence is poor, the feature fusion focuses on the accurate motion trajectory and distance information provided by the millimeter-wave radar. When the radar data is affected by metal interference, the feature fusion focuses on the key point information extracted from the image sequence. The fused feature vector is input into a pre-trained gesture recognition model to obtain the static gesture category, dynamic gesture trajectory category, and gesture recognition confidence output by the gesture recognition model; the gesture recognition model can be any one of the ST-GCN model, 3D-CNN model, and SVM model; S204. Generate cockpit interaction commands through the interaction control module 23, and send valid cockpit interaction commands to the corresponding cockpit subsystem 25 to drive it to perform corresponding operations. Specifically, the interactive control module 23 maps the recognized gesture categories to specific cockpit interactive commands according to the preset gesture-command mapping rule library. For example, the "OK" gesture is mapped to the command to confirm / answer a phone call, the thumb gesture to the left / right is mapped to the command to adjust the volume down / up, and the gesture of pinching fingers and moving them up is mapped to the command to increase the air conditioning temperature. If the confidence level of gesture recognition is greater than or equal to the confidence threshold, the cockpit interaction command is confirmed as a valid command, and the interaction control module 23 sends it to the corresponding cockpit subsystem 25. If the confidence level of gesture recognition is less than the confidence threshold, it is considered invalid recognition or the feedback module 24 is triggered to require the user to repeat the gesture. S205. Provide operation confirmation or status prompts in real time through feedback module 24; Specifically, the feedback module 24 provides visual feedback, auditory feedback, and tactile feedback. Visual feedback may be displaying the currently recognized function icon or operation result on the HUD (Head-Up Display) or instrument panel, such as displaying changes in the volume bar and the air conditioning temperature value. Auditory feedback may include playing a short prompt tone, such as playing a "beep" sound to indicate that the command has been successfully sent. Tactile feedback may be applying a slight vibration to the steering wheel or seat to indicate that the operation was successful or that attention is needed.

[0046] This embodiment has at least the following advantages: First, millimeter-wave radar and depth sensors are unaffected by the intensity of visible light, and can still provide reliable hand movement trajectory and position information even when the camera image quality is severely degraded, ensuring that core interactive functions are not interrupted. When the driver's hand is obstructed, the strong penetration capability of millimeter-wave radar and / or the contour perception capability of depth sensors can still effectively capture the main motion features of the gesture, and combine them with local information from the image data for fusion judgment, effectively reducing the recognition failure rate caused by occlusion. The system can work stably in complex physical environments such as rain, fog, dust, and changes in temperature and humidity in the cockpit. Secondly, the gesture recognition model can recognize not only static gestures but also dynamic gestures, which allows for a greater number of mappable interactive commands, thus enabling a richer and more intuitive set of interactive commands. Dynamic gestures themselves carry information about direction, speed, and amplitude, making control more precise and intuitive. Third, by filtering out low-confidence recognition results caused by brief interference or invalid actions, it ensures that only clear and effective gestures will trigger commands, effectively reducing the false trigger rate; through multi-channel real-time feedback, the driver can clearly perceive whether the command has been successfully received and executed, and when the recognition confidence is insufficient, it provides an opportunity for error correction, forming a reliable human-machine interaction closed loop, significantly improving the controllability of operation and user trust. Fourth, drivers do not need to precisely aim at specific devices, such as physical buttons or touchscreens. They can complete the operation with their hands within their natural range of motion, such as near the steering wheel or in front of the center console. This reduces the time the driver's gaze is off the road, thus improving driver safety. The gesture command design conforms to human natural semantics and operating habits, eliminating the need for complex memorization or menu navigation, effectively reducing the driver's cognitive burden. Through reliable multimodal recognition and clear feedback, it reduces repeated attempts and frustration caused by interaction failures, allowing drivers to focus more on driving tasks.

[0047] Corresponding to the embodiments of the aforementioned methods, this application also provides embodiments of automotive cockpit interaction devices and terminals for their applications: like Figure 3 As shown, Figure 3 This is a block diagram of an automotive cockpit interaction device provided in an embodiment of this application. The device includes: The acquisition module 31 is used to acquire multimodal data of the driver's hand area; the multimodal data includes image data and at least one of the following: radar data and depth data; The recognition module 32 is used to extract gesture features from the multimodal data, process the gesture features based on the gesture recognition model, and obtain the gesture category. The control module 33 is used to generate cockpit interaction commands based on the gesture category, and control the cockpit subsystem of the vehicle cockpit based on the cockpit interaction commands.

[0048] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0049] This application also provides an electronic device, please refer to [link to application]. Figure 4 , Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. The electronic device may include a processor 410, a communication interface 420, a memory 430, and at least one communication bus 440. The communication bus 440 is used to enable direct communication between these components. In this embodiment, the communication interface 420 of the electronic device is used for signaling or data communication with other node devices. The processor 410 may be an integrated circuit chip with signal processing capabilities.

[0050] The processor 410 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor, or the processor 410 can be any conventional processor.

[0051] The memory 430 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory 430 stores computer-readable instructions. When these computer-readable instructions are executed by the processor 410, the electronic device can perform the aforementioned operations. Figure 1 The various steps involved in the method implementation examples.

[0052] Alternatively, the electronic device may also include a storage controller and an input / output unit.

[0053] The memory 430, storage controller, processor 410, peripheral interface, and input / output unit are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses 440. The processor 410 is used to execute executable modules stored in the memory 430, such as software function modules or computer programs included in electronic devices.

[0054] The input / output unit is used to provide users with the ability to create tasks and to set optional start periods or preset execution times for those tasks, thereby enabling user-server interaction. The input / output unit may be, but is not limited to, a mouse and keyboard.

[0055] Understandable. Figure 4 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown. Figure 4 The components shown can be implemented using hardware, software, or a combination thereof.

[0056] This application also provides a storage medium storing instructions. When the instructions are run on a computer, the computer program is executed by a processor to implement the method described in the method embodiment. To avoid repetition, the method will not be described again here.

[0057] This application also provides a computer program product that, when run on a computer, causes the computer to perform the method described in the method embodiment.

[0058] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0059] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0060] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0061] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0062] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0063] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for interaction in a car cockpit, characterized in that, include: Collect multimodal data of the driver's hand area; the multimodal data includes image data and at least one of the following: radar data and depth data; Gesture features are extracted from the multimodal data, and the gesture features are processed based on the gesture recognition model to obtain the gesture category; Generate cockpit interaction commands based on the gesture category, and control the cockpit subsystem of the vehicle cockpit based on the cockpit interaction commands.

2. The method according to claim 1, characterized in that, Before extracting gesture features from the multimodal data, the process includes: The multimodal data is preprocessed; Spatiotemporal alignment is performed on the preprocessed multimodal data.

3. The method according to claim 1, characterized in that, The image data includes an image sequence; the extraction of gesture features from the multimodal data includes: A first gesture feature is extracted from the image sequence; the first gesture feature represents the spatial coordinates and temporal changes of key points of the hand. A second gesture feature is extracted from the radar data and / or depth data; the second gesture feature includes spatial static features representing hand contour and distance information, and temporal dynamic features representing hand movement speed and direction.

4. The method according to claim 3, characterized in that, The gesture recognition model is used to process the gesture features to obtain gesture categories, including: The first gesture feature and the second gesture feature are weighted and fused to obtain a fused feature vector; the weights of the first gesture feature and the second gesture feature are determined based on the image quality of the image sequence. The fused feature vector is input into the gesture recognition model to obtain the gesture category output by the gesture recognition model.

5. The method according to claim 1, characterized in that, The gesture categories include static gesture categories and dynamic gesture trajectory categories; The step of generating cockpit interaction commands based on the gesture category includes: According to a preset mapping rule library, the gesture categories are mapped to cockpit interaction commands; The mapping rule base records the cockpit interaction commands corresponding to different static gesture categories, as well as the cockpit interaction commands corresponding to different dynamic gesture trajectory categories.

6. The method according to claim 1, characterized in that, Also includes: Obtain the gesture recognition confidence score output by the gesture recognition model; If the confidence level of the gesture recognition is less than the confidence level threshold, the cockpit interaction command is determined to be an invalid command, and the driver is notified to re-enter the gesture.

7. The method according to claim 1, characterized in that, Also includes: Based on the execution result of the cockpit interaction command, multi-channel feedback is provided to the driver; the multi-channel feedback includes at least two of the following: visual feedback, auditory feedback, and vibration feedback.

8. A car cockpit interaction device, characterized in that, include: The acquisition module is used to acquire multimodal data of the driver's hand area; the multimodal data includes image data and at least one of the following: radar data and depth data; The recognition module is used to extract gesture features from the multimodal data, process the gesture features based on the gesture recognition model, and obtain the gesture category. The control module is used to generate cockpit interaction commands based on the gesture category, and to control the cockpit subsystem of the vehicle cockpit based on the cockpit interaction commands.

9. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any one of claims 1 to 7.

Images