Vehicle on-board function control method, vehicle and readable storage medium

Abstract

The embodiment of the application provides a vehicle-mounted function control method of a vehicle, a vehicle and a readable storage medium, the method comprises the following steps: fusing multi-modal sensing data of at least one driving and riding object in the vehicle to obtain fusion features, wherein the fusion features are used for representing operations performed by the driving and riding object on a vehicle-mounted function of the vehicle; identifying the fusion features to obtain an operation category to which the operations belong and a confidence degree of the operation category; correcting the operation category in response to the confidence degree being lower than a confidence degree threshold; generating a control instruction of the vehicle based on the corrected operation category; and controlling the vehicle to perform the vehicle-mounted function in response to the control instruction. The application solves the technical problem of low accuracy of vehicle-mounted function control of the vehicle.

Description

Technical Field

[0001] This application relates to the field of vehicle control, and more specifically, to a method for controlling onboard functions of a vehicle, a vehicle, and a readable storage medium. Background Technology

[0002] Currently, vehicle in-vehicle function control mainly relies on physical buttons, central control touch screens, and voice commands. However, physical buttons have a fixed layout, poor expandability, and can easily distract the driver; central control touch screens require eye shift and physical operation, posing safety hazards and operational delays in bumpy or emergency situations.

[0003] Furthermore, the lack of multimodal collaboration and real-time error correction mechanisms results in low accuracy of voice command recognition in complex driving environments, failing to provide a continuous, smooth, and safe immersive interactive experience. Therefore, the technical issue of low accuracy in vehicle in-vehicle function control persists.

[0004] There is currently no good solution to the above problems. Summary of the Invention

[0005] This application provides a vehicle onboard function control method, a vehicle, and a readable storage medium to at least solve the technical problem of low accuracy in vehicle onboard function control.

[0006] According to one aspect of the embodiments of this application, a method for controlling the in-vehicle function of a vehicle is provided. The method may include: fusing multimodal perception data of at least one driver or passenger in the vehicle to obtain fused features, wherein the fused features are used to characterize the operation performed by the driver or passenger in response to the in-vehicle function of the vehicle; identifying the fused features to obtain the operation category to which the operation belongs and the confidence level of the operation category; correcting the operation category in response to the confidence level being lower than a confidence level threshold; generating a control command for the vehicle based on the corrected operation category; and controlling the vehicle to execute the in-vehicle function in response to the control command.

[0007] Furthermore, in response to the confidence level being lower than the confidence level threshold, the operation category is corrected, including: in response to the confidence level being lower than the confidence level threshold, obtaining the vehicle's status information, wherein the status information is used to represent the vehicle's operating status and / or the working status of the on-board function; matching the operation category and the status information to obtain a matching result; and correcting the operation category based on the matching result.

[0008] Furthermore, based on the matching results, the operation category is corrected, including: in response to the matching result indicating that the operation category and status information failed to match, obtaining the operation information performed by the driver / passenger on the in-vehicle function in a historical time period, wherein the operation information includes multiple historical operation categories and execution frequencies; and correcting the operation category based on the operation information.

[0009] Furthermore, based on the operation information, the operation categories are corrected, including: filtering the operation information to obtain historical operation categories with an execution frequency higher than a frequency threshold; in response to the fact that there are multiple types of historical operation categories, sound source localization information and semantic features are extracted from the audio data in the multimodal perception data; and the operation categories are corrected based on the sound source localization information and semantic features.

[0010] Furthermore, based on the operation information, the operation category is corrected, including: in response to detecting that the sound source localization information in the audio data of the multimodal perception data comes from the driver or passenger, the semantic features in the audio data of the multimodal perception data are parsed to obtain the initial operation category corresponding to the semantic features; in response to the inconsistency between the initial operation category and the operation category, the operation category is corrected.

[0011] Furthermore, the multimodal perception data of at least one driver / passenger object in the vehicle is fused to obtain fused features, including: acquiring multimodal perception data using multi-source sensors deployed in the vehicle; determining target multimodal perception data corresponding to the target part of the driver / passenger object from the multimodal perception data; and calling a fusion model to fuse the target multimodal perception data to obtain fused features, wherein the fusion model is obtained by training a neural network using multimodal perception data samples of the driver / passenger object samples.

[0012] Furthermore, the multimodal perception data includes at least one of the following: positional features, attitude features, three-dimensional structural features, micro-motion change features, and kinematic features of the target part. The kinematic features include acceleration features and angular velocity features. The multimodal perception data is acquired using multi-source sensors deployed in the vehicle, including: extracting positional and attitude features from the vehicle's image data using visual sensors deployed in the vehicle; extracting three-dimensional structural features and micro-motion change features from the vehicle's point cloud data using perception sensors deployed in the vehicle; and extracting kinematic features from the vehicle's motion data using inertial sensors deployed in the vehicle.

[0013] Furthermore, the fused features are identified to obtain the operation category to which the operation belongs, as well as the confidence level of the operation category. This includes: determining the operation function area to which the target part belongs based on the location information of the target part of the driver / passenger object in the vehicle, wherein the operation function area is used to represent the orientation of the target part in the vehicle, and the orientation corresponds to the function module of the vehicle function; determining the candidate operation category set of the driver / passenger object based on the operation function area; calling the recognition model to identify the fused features and the candidate operation category set to obtain the operation category and confidence level. The recognition model is obtained by training a convolutional neural network using multimodal perception data samples of driver / passenger object samples and state information samples of vehicle samples.

[0014] Furthermore, based on the corrected operation category, control commands for the vehicle are generated, including: determining the identification information corresponding to the corrected operation category; determining the vehicle function and its control mode according to the identification information; and generating control commands based on the current state and control mode of the vehicle function.

[0015] Furthermore, in response to a control command, controlling the vehicle to perform onboard functions includes: in response to the control mode being an adjustment mode, adjusting the onboard function parameters corresponding to the current state according to the target parameters in the control command; in response to the control mode being a switching mode, switching the onboard function from the current state to the target state according to the target state in the control command; and in response to the control mode being a confirmation mode, performing the operation corresponding to the onboard function in the current state.

[0016] According to another aspect of the embodiments of this application, a vehicle in-vehicle function control device is also provided, comprising: a fusion unit, configured to perform fusion processing on multimodal perception data of at least one driver / passenger object in the vehicle to obtain fusion features, wherein the fusion features are used to characterize the operation performed by the driver / passenger object for the vehicle's in-vehicle functions; an identification unit, configured to identify the fusion features to obtain the operation category to which the operation belongs and the confidence level of the operation category; a correction unit, configured to correct the operation category in response to the confidence level being lower than a confidence level threshold; a generation unit, configured to generate a vehicle control command based on the corrected operation category; and a control unit, configured to control the vehicle to perform in-vehicle functions in response to the control command.

[0017] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0018] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0019] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0020] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0021] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0022] In this embodiment, multimodal perception data of at least one passenger in the vehicle are fused to obtain fused features, overcoming the limitation of incomplete data obtained from a single sensor. The fused features are then identified to determine the operation category and its confidence level. If the confidence level is below a threshold, the operation category is corrected to avoid safety hazards caused by misoperation. Control commands are then generated based on the corrected, high-confidence operation category to drive the execution of in-vehicle functions. This solves the technical problem of low accuracy in in-vehicle function control and improves the accuracy of in-vehicle function control. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0024] Figure 1 This is a flowchart of a vehicle on-board function control method according to an embodiment of this application;

[0025] Figure 2 This is a flowchart of a method for recognizing and correcting gesture commands from multiple vehicle sensors according to an embodiment of this application;

[0026] Figure 3 This is a flowchart of a gesture recognition and error correction method according to an embodiment of this application;

[0027] Figure 4 This is a flowchart of a multimodal fusion determination method according to an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of an on-board function control device for a vehicle according to an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] According to an embodiment of this application, an embodiment of a vehicle in-vehicle function control method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0033] This embodiment provides a method for controlling vehicle onboard functions. Figure 1 This is a flowchart of a vehicle on-board function control method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps.

[0034] Step S102: Perform fusion processing on the multimodal perception data of at least one driver / passenger in the vehicle to obtain fused features.

[0035] In the technical solution provided by step S102 of this application, the fusion feature can be used to characterize the operation performed by the driver / passenger on the vehicle's onboard functions.

[0036] In this embodiment, multimodal perception data can refer to raw or primary feature data about the actions (e.g., hand gestures) of the driving and riding objects (e.g., the driver, the front passenger, or the rear passenger) and the environmental state collected by different types of sensors deployed inside the vehicle.

[0037] Optionally, multimodal perception data may include visual information, depth and structural information, inertial motion information, and audio and sound source localization, etc., for illustrative purposes only, and no specific limitations are imposed here.

[0038] Optionally, visual information can be acquired via a camera, capturing the position coordinates (X, Y) of the driver's hand on a two-dimensional image plane and posture information (e.g., finger spread angle, wrist rotation angle). Depth and structural information can be acquired via a depth camera or millimeter-wave radar sensor, capturing the three-dimensional spatial trajectory (X, Y, Z coordinates) of the driver's hand, the three-dimensional structural features of the hand contour, and subtle movement changes (e.g., minute tremors or tapping movements of the fingers).

[0039] Optionally, by using a hand-worn device with inertial sensors (such as a smartwatch or wristband), or by utilizing onboard sensors to capture the relative displacement of the hand caused by vehicle vibrations, kinematic characteristics such as acceleration and angular velocity can be obtained to reflect the dynamic trend of the gesture. Voice signals can also be combined to assist in locating the sound source of the gesture or confirming the operational intent, thereby obtaining audio and sound source localization.

[0040] In this embodiment of the application, multimodal perception data of at least one driver or passenger in the vehicle are fused. By using multi-source complementarity, the problems that a single sensor cannot solve, such as changes in illumination (night / strong light), gesture occlusion (steering wheel occlusion, clothing occlusion), and hand angle deviation, are overcome. This results in high-quality fused features, providing an accurate data foundation for the subsequent recognition of fused features and the correction of operation categories.

[0041] Step S104: Identify the fused features to obtain the operation category to which the operation belongs and the confidence level of the operation category.

[0042] In the technical solution provided by step S104 of this application, the fusion feature is a high-dimensional data representation that can comprehensively characterize the spatial structure of gestures, dynamic trajectories, and temporal dependencies after multi-source sensor fusion, dimensionality reduction, and abstraction.

[0043] In this embodiment, the operation category can refer to the specific semantic intent or in-vehicle function command represented by the operation (e.g., a gesture). The confidence level of the operation category can be a probability value indicating the degree of certainty regarding the operation category to which the identified operation belongs.

[0044] Optionally, recognition models, such as convolutional neural networks, skeletal point sequence models, graph convolutional neural networks, and multimodal fusion classifiers, can be used to identify the fused features, obtain the operation category to which the operation belongs, and the confidence level of the operation category. This is only for illustrative purposes and no specific restrictions are imposed here.

[0045] Optionally, convolutional neural networks can be used to process data with a grid structure (e.g., image data) to extract spatial local features. Skeletal point sequence models can be used to process time-series data, capturing continuous trends and long-range dependencies in hand gestures. Graph convolutional neural networks can be used to process the topological relationships of hand skeletal points, understanding the relative positions between hand joints. Multimodal fusion classifiers can receive fused features and output class probabilities through fully connected layers or attention mechanisms.

[0046] In this embodiment, recognition based on high-quality fused features can effectively resist changes in illumination, occlusion, and noise interference. Compared with single-sensor recognition, the classification decision boundary under fused features is clearer, significantly reducing the false recognition rate.

[0047] Step S106: In response to the confidence level being lower than the confidence level threshold, the operation category is corrected.

[0048] In the technical solution provided by step S106 of this application, it can be determined whether the operation category needs to be modified based on the confidence level and the confidence level threshold.

[0049] In this embodiment, if the confidence level is lower than the confidence level threshold, it can be determined that the identification result obtained by identifying the fusion feature (that is, the operation category to which the operation belongs and the confidence level of the operation category) is ambiguous or uncertain, and there is a risk of misoperation. At this time, correction can be triggered.

[0050] For example, if the confidence level is below the confidence threshold, other auxiliary information (such as vehicle status information, user habits, and multimodal redundant data) can be used to re-evaluate the operation category. The goal of the correction is to find a more reasonable operation category that fits the current context, rather than simply discarding the gesture.

[0051] Optionally, the operation category can be corrected through contextual logic comparison, historical habit matching, and multimodal auxiliary confirmation. Contextual logic comparison can determine the reasonableness of the initially identified operation category by combining the vehicle's current operating status and task scenario. Historical habit matching utilizes the historical operation database of the driver and passengers to infer the most likely operation category. Multimodal auxiliary confirmation can involve introducing data from other sensors or interaction channels for cross-validation.

[0052] Optionally, if the confidence level is higher than the confidence level threshold, it can be determined that the current recognition result is correct. In this case, control commands can be directly output to control the vehicle to perform onboard functions.

[0053] In this embodiment, when the confidence level is lower than the confidence level threshold, the operation category is corrected; when the confidence level is higher than the confidence level threshold, control commands are directly output to control the vehicle to perform onboard functions. This effectively filters out noise and abnormal identification results and reduces the error rate.

[0054] Step S108: Based on the modified operation category, generate control commands for the vehicle.

[0055] In the technical solution provided by step S108 of this application, the semantic intent that can be modified (e.g., error correction) can be transformed into a low-level signal that can be understood and executed by the vehicle's electronic control unit or the functional module of the vehicle's functions.

[0056] In this embodiment, the control commands can be specific parameters or signals that can be directly parsed and executed by various subsystems inside the vehicle (such as the audio head unit, air conditioning controller, navigation system, window motor, etc.).

[0057] Optionally, by querying a preset function mapping table, the corrected operation category can be converted into a specific in-vehicle function and control mode. For example, if the control mode is "song skipping," it is mapped to an in-vehicle function for multimedia playback, and the operation type can be "previous track" or "next track."

[0058] Optionally, specific parameters can be determined based on the current state of the vehicle function. For example, if the control mode is adjustment mode, the vehicle function parameters corresponding to the current state can be adjusted according to the target parameters in the control command. If the control mode is switching mode, the vehicle function can be switched from the current state to the target state according to the target state in the control command. If the control mode is confirmation mode, the operation corresponding to the vehicle function in the current state can be executed.

[0059] Optionally, if multiple gestures are triggered simultaneously or nearly simultaneously, or if gesture commands in the control instructions conflict with voice commands or physical button commands, the final control command can be generated according to a preset priority strategy. For example, the preset priority strategy could be: emergency safety commands > driver commands > passenger commands > entertainment commands.

[0060] In this embodiment, control commands are generated based on the correction results, which reduces misoperations caused by recognition errors (such as accidentally opening the window or accidentally switching the navigation), so that the feedback felt by the user is accurate and in line with expectations.

[0061] Step S110: In response to a control command, control the vehicle to perform onboard functions.

[0062] In the technical solution provided by step S110 of this application, after generating the vehicle control command based on the modified operation category, the vehicle can be controlled to perform onboard functions according to the control command.

[0063] In this embodiment, the control command may include the target functional module (e.g., audio host, air conditioner controller, central control screen), control mode (e.g., adjust, switch, confirm), and specific execution parameters (e.g., volume value, temperature value, on / off status).

[0064] Optionally, in-vehicle functions can be various functions managed by electronic control units or domain controllers within the vehicle, including but not limited to entertainment systems, navigation systems, environmental control systems, vehicle settings, and driver assistance systems. These are merely examples and no specific limitations are imposed here.

[0065] Optionally, the entertainment system can control volume adjustment, song switching, and media pause / play. The navigation system can control route confirmation, map zooming, and point of interest (POI) search confirmation. The environmental control system can control air conditioning temperature adjustment, fan speed adjustment, and seat heating / ventilation on / off. Vehicle settings can control driving mode switching (e.g., Sport / Comfort mode), lighting control, and window / sunroof control. Driver assistance can control cruise control settings and automatic parking confirmation (subject to safety regulations).

[0066] Optionally, control commands can be sent via the vehicle network to the electronic control unit or domain controller where the target functional module is located, in order to control the vehicle to perform vehicle functions.

[0067] In this embodiment of the application, the above steps can ensure that the operation is transformed into actual vehicle behavior within a safe, compliant, and efficient range, thereby improving the human-machine interaction experience of the smart cockpit.

[0068] Steps S102 to S110 above involve fusing multimodal perception data of at least one passenger in the vehicle to obtain fused features, overcoming the limitation of incomplete data obtained from a single sensor. Then, the fused features are identified to determine the operation category and its confidence level. If the confidence level is lower than a confidence threshold, the operation category is corrected to avoid safety hazards caused by misoperation. Control commands are then generated based on the corrected, high-confidence operation category to drive the execution of in-vehicle functions. This solves the technical problem of low accuracy in vehicle in-vehicle function control and achieves the technical effect of improving the accuracy of vehicle in-vehicle function control.

[0069] The above-mentioned method of this application will be further described below.

[0070] As an optional implementation, step S106, in response to the confidence level being lower than the confidence level threshold, corrects the operation category, including: in response to the confidence level being lower than the confidence level threshold, obtaining the vehicle's status information, wherein the status information is used to represent the vehicle's operating status and / or the working status of the vehicle's functions; matching the operation category and the status information to obtain a matching result; and correcting the operation category based on the matching result.

[0071] In this embodiment, during the process of correcting the operation category in response to a confidence level below a confidence threshold, if the confidence level is below the threshold, a contextual logic comparison can be performed. That is, the vehicle's state information is obtained, and the operation categories with confidence levels below the confidence threshold are matched with the current vehicle state information, for example, by performing a logical comparison and a reasonableness assessment.

[0072] Optionally, after matching the operation category and status information to obtain the matching result, if the matching result indicates that the operation category and status information are compatible, for example, in navigation mode, the gesture "confirm" and "navigation confirmation" have a high degree of matching, then there is no need to modify the operation category.

[0073] Optionally, if the matching result indicates that the operation category and status information fail to match, for example, the operation category conflicts with or is unreasonable with the current status information (for example, when driving at high speed, the gesture "switch navigation destination" is considered high risk or low priority), then the correction of the operation category is triggered.

[0074] In this embodiment of the application, by introducing vehicle status information as the key basis for correction (e.g., error correction), isolated operation categories (e.g., gestures) can be transformed into intelligent interactions with scene awareness.

[0075] As an optional implementation, the operation category is corrected based on the matching result, including: in response to the matching result indicating that the operation category and status information failed to match, obtaining the operation information performed by the driver / passenger on the vehicle function in a historical time period, wherein the operation information includes multiple historical operation categories and execution frequencies; and correcting the operation category based on the operation information.

[0076] In this embodiment, if the matching result indicates that the operation category and status information fail to match, a correction of the operation category is triggered. Historical habit matching can be performed, that is, obtaining the operation information performed by the driver / passenger on in-vehicle functions within a historical time period, thereby correcting the operation category based on the operation information.

[0077] For example, if the current vehicle speed is 0 (parked), the gesture recognition is a vague "swipe left". In this case, "swipe left" could mean "change song", "switch radio station", or "return to the previous menu". Since the scene information (parked) cannot rule out any possibilities, the logical matching "failed" (i.e., it could not make a unique or high-confidence judgment). Therefore, it is necessary to introduce information from other dimensions to assist in decision-making.

[0078] Optionally, the operation information performed by the driver / passenger on in-vehicle functions during a historical time period is a record of gesture operations performed by the driver / passenger (e.g., driver or passenger) over a past period (e.g., the past day, week, or month). The operation category can be the specific name of the gesture performed by the user (e.g., "clench fist," "two-finger tap," "wave"). The execution frequency can be the number of times or probability distribution of a particular operation category performed by the user. For example, a user performed the "clench fist" gesture 10 times in the past week, with 8 times used for "confirming navigation" and 2 times used for "answering a phone call."

[0079] Optionally, by utilizing users' personal habit data, probabilistic inferences can be made about the current ambiguous gesture intent. That is, in the absence of strong contextual constraints, users are most likely to perform the actions they have most frequently performed in the past.

[0080] In this embodiment of the application, when the vehicle information is insufficient to distinguish the user's intention, the operation information of the driver and passenger on the vehicle function in a historical time period, that is, the user's personal behavioral preferences, can be used as a second judgment basis to solve the problem of ambiguity.

[0081] As an optional implementation, the operation category is corrected based on the operation information, including: filtering the operation information to obtain historical operation categories with an execution frequency higher than a frequency threshold; in response to the fact that there are multiple types of historical operation categories, extracting sound source localization information and semantic features from the audio data in the multimodal perception data; and correcting the operation category based on the sound source localization information and semantic features.

[0082] In this embodiment, if the operation information performed by the driver or passenger on the vehicle function in a historical time period is still ambiguous (i.e., the user has multiple high-frequency gesture habits and cannot be determined by frequency alone), multimodal auxiliary confirmation can be performed to correct the operation category.

[0083] Optionally, after filtering the operation information to obtain historical operation categories with execution frequencies higher than a frequency threshold, if there are multiple historical operation categories—for example, if the user has multiple high-frequency, mutually exclusive operation habits for the current ambiguous gesture—the intention cannot be determined solely by the operation information. In this case, a voice-assisted error correction mechanism can be activated. Sound source localization information and semantic features can be extracted from the audio data in the multimodal perception data, thereby correcting the operation category based on the sound source localization information and semantic features.

[0084] Optionally, during the process of correcting the action category based on sound source localization information and semantic features, if the sound source localization information is detected to originate from a driver or passenger, the semantic features can be parsed to obtain the initial action category corresponding to the semantic features. If the initial action category is inconsistent with the actual action category, the action category can be corrected.

[0085] In this embodiment, when the confidence level of a single modality (visual / inertial only) is low, and the intent cannot be uniquely determined solely by historical habits, audio data can be introduced as an arbitrator. Audio data has clear semantic directionality, which can effectively eliminate the ambiguity of gestures. By locating the sound source, it can be ensured that the correction logic is based on the correct operator's habits and context, avoiding command mismatch in multi-occupant scenarios.

[0086] As an optional implementation, the operation category is corrected based on the operation information, including: in response to detecting that the sound source localization information in the audio data of the multimodal perception data comes from the driver or passenger, the semantic features in the audio data of the multimodal perception data are parsed to obtain the initial operation category corresponding to the semantic features; in response to the inconsistency between the initial operation category and the operation category, the operation category is corrected.

[0087] In this embodiment, a microphone array can detect the presence of sound and calculate the sound source location information to confirm that the sound originates from a legitimate operator inside the vehicle (e.g., the driver, front passenger, or rear passenger), rather than external noise or broadcast sounds. The captured semantic features can be automatically converted into text through speech recognition, and the user's specific intent can be extracted through natural language processing to obtain the initial operation category corresponding to the semantic features.

[0088] Optionally, if the initial operation category is inconsistent with the operation category, the operation category can be corrected.

[0089] In this embodiment, by making corrections based on explicit semantic features, erroneous execution caused by gesture recognition errors is avoided. For example, it prevents users from accidentally triggering "close the window" when saying "confirm navigation".

[0090] As an optional implementation, step S102 involves fusing the multimodal perception data of at least one driver / passenger object in the vehicle to obtain fused features. This includes: acquiring multimodal perception data using multi-source sensors deployed in the vehicle; determining target multimodal perception data corresponding to the target part of the driver / passenger object from the multimodal perception data; and calling a fusion model to fuse the target multimodal perception data to obtain fused features. The fusion model is obtained by training a neural network using multimodal perception data samples of the driver / passenger object samples.

[0091] In this embodiment, the multimodal perception data of at least one driver or passenger in the vehicle are fused to obtain fused features. This process can transform the raw, heterogeneous sensor data into a unified, high-dimensional feature representation, providing a high-quality data foundation for subsequent identification and error correction.

[0092] Optionally, multimodal perception data refers to raw data containing different physical attributes collected by multi-source sensors deployed inside the vehicle. The target body part can be the hand, for example only.

[0093] Optionally, after acquiring the multimodal perception data, the target multimodal perception data corresponding to the target part of the driving and riding object can be determined from the multimodal perception data. Feature extraction can be performed on the target multimodal perception data, and then a fusion model can be called to perform fusion processing on the target multimodal perception data to obtain fused features.

[0094] In this embodiment, the fused features contain more information than single-modal features, enabling more accurate differentiation of similar gestures (e.g., "swipe left" and "swipe right" differ significantly in depth information), thereby reducing the false recognition rate. By extracting the target region, only data from the region of interest is processed, significantly reducing computational load and improving the speed of feature extraction and fusion, thus meeting the real-time requirements of in-vehicle applications.

[0095] As an optional implementation, the multimodal perception data includes at least one of the following: position features, attitude features, three-dimensional structural features, micro-motion change features, and kinematic features of the target part. The kinematic features include acceleration features and angular velocity features. The multimodal perception data is acquired using multi-source sensors deployed in the vehicle, including: extracting position features and attitude features from the vehicle's image data using visual sensors deployed in the vehicle; extracting three-dimensional structural features and micro-motion change features from the vehicle's point cloud data using perception sensors deployed in the vehicle; and extracting kinematic features from the vehicle's motion data using inertial sensors deployed in the vehicle.

[0096] In this embodiment, multi-source sensors deployed in the vehicle can acquire multimodal perception data.

[0097] Optionally, positional features refer to the hand's coordinates (X, Y, Z) in three-dimensional space or its relative position to the vehicle body / driver. Postural features refer to the hand's orientation, rotation angle, and the opening / closing state of the fingers (e.g., open, clenched fist, "V" sign). Three-dimensional structural features refer to the geometry, volume, and depth profile of the hand and arm, used to distinguish the spatial forms of different hand gestures. Micro-motion variation features refer to extremely subtle vibrations, tremors, or high-frequency shaking of the hand. Among kinematic features, acceleration features are the rate of change of hand movement, reflecting the force and the instant of initiation and termination of the action. Angular velocity features are the rate of hand rotation, reflecting the torsional movements of the wrist or arm.

[0098] Alternatively, visual sensors, such as cameras, are adept at capturing image data (e.g., appearance and 2D / 3D projection information), thus enabling the extraction of position and pose.

[0099] Alternatively, perception sensors, such as depth cameras / millimeter-wave radar / LiDAR, are adept at capturing point cloud data (e.g., depth and spatial point cloud information), and thus can extract 3D structure and micro-motion changes (through small changes in point cloud density or the Doppler effect of radar).

[0100] Alternatively, inertial sensors can directly measure physical motion quantities, thus extracting acceleration and angular velocity features from the motion data.

[0101] In this embodiment, a comprehensive, multi-layered gesture perception system is constructed through precise mapping of sensor and feature. By utilizing vision, perception, and inertia, high-precision and highly reliable gesture feature extraction is achieved in complex in-vehicle environments.

[0102] As an optional implementation, step S104 involves identifying the fused features to obtain the operation category to which the operation belongs and the confidence level of the operation category. This includes: determining the operation function area to which the target part belongs based on the location information of the target part of the driver / passenger object in the vehicle, wherein the operation function area is used to represent the orientation of the target part in the vehicle, and the orientation corresponds to the function module of the vehicle function; determining the candidate operation category set of the driver / passenger object based on the operation function area; and calling the recognition model to identify the fused features and the candidate operation category set to obtain the operation category and confidence level. The recognition model is obtained by training a convolutional neural network using multimodal perception data samples of the driver / passenger object sample and state information samples of the vehicle sample.

[0103] In this embodiment, the operating function area is a functional partition divided according to the position information (e.g., three-dimensional spatial position) of the target part of the driver / passenger (e.g., hand) within the vehicle. Each area corresponds to a specific set of in-vehicle function modules.

[0104] Optionally, the operating function areas can be the central control screen area, the steering wheel area, the door / armrest area, and the rear seat area. The central control screen area can be located on the center console in front of the driver / passenger, corresponding to multimedia, navigation, and air conditioning control functions. The steering wheel area can be located in front of the driver, corresponding to cruise control, volume adjustment, and voice assistant functions. The door / armrest area can correspond to window and seat adjustment functions. The rear seat area can correspond to rear entertainment and independent air conditioning control functions.

[0105] Optionally, the candidate operation category set is a predefined set of all possible gesture commands within a specific operation function area. For example, when a hand is detected in the "central control screen area", it is not necessary to judge among all possible gestures (such as "shifting gears" or "unlocking the door"), but only to filter from the gesture set corresponding to the "central control screen area" (such as "swiping left or right to change songs", "swiping up or down to adjust volume", or "using two fingers to zoom the map").

[0106] Optionally, after determining the operating function area to which the target part belongs based on the location information of the target part of the driver or passenger in the vehicle, the recognition model can be invoked to identify the fused features and candidate operation category set, thereby obtaining the operation category and confidence level.

[0107] In this embodiment, the strategy of "spatial positioning -> candidate set screening -> local recognition" is used to cleverly combine geometric spatial information with semantic recognition tasks, thereby achieving efficient, accurate and low-latency vehicle gesture recognition.

[0108] As an optional implementation, step S108, generating vehicle control commands based on the modified operation category, includes: determining the identification information corresponding to the modified operation category based on the modified operation category; determining the vehicle function and the control mode of the vehicle function according to the identification information; and generating control commands based on the current state and control mode of the vehicle function.

[0109] In this embodiment, the identification information refers to the unique logical code or instruction information bound to a specific operation category, which serves as a mapping bridge between gesture semantics and the vehicle's onboard functions.

[0110] Optionally, based on the corrected operation category, the corresponding identification information can be determined. Then, according to the identification information, the vehicle function and its control mode can be determined. The control mode is the specific method or parameter type for executing the vehicle function. For example, it can be a control mode, a switching mode, a confirmation mode, etc. Then, based on the current state of the vehicle function and the control mode, control commands can be generated.

[0111] Optionally, the current state of the in-vehicle function is the real-time state of the in-vehicle function before the command is executed, such as the current volume being 50; the current air conditioning being in "automatic" mode; and the current navigation being guided.

[0112] Optionally, for adjustment mode, the current state is the baseline value to be modified (e.g., the current temperature is 20℃). For switching mode, the current state is the basis for determining whether the state needs to be flipped (e.g., whether the current light is on or off; if it is a switching mode and the command is ambiguous, the current state needs to be referenced). For confirmation mode, the current state determines the specific behavior of the confirmation operation (e.g., confirming the route in the navigation interface, confirming playback in the music interface).

[0113] In the embodiments of this application, different vehicle functions have different interaction characteristics (some require continuous adjustment, some require switching on and off, and some require confirmation). By distinguishing the modes, the processing logic can be unified to adapt to the requirements of different vehicle functions.

[0114] As an optional implementation, step S110, in response to a control command, controls the vehicle to perform onboard functions, including: in response to the control mode being an adjustment mode, adjusting the onboard function parameters corresponding to the current state according to the target parameters in the control command; in response to the control mode being a switching mode, switching the onboard function from the current state to the target state according to the target state in the control command; and in response to the control mode being a confirmation mode, performing the operation corresponding to the onboard function in the current state.

[0115] In this embodiment, the target parameter / target state is the desired result specified in the control command.

[0116] Optionally, the adjustment mode is used for continuous or discrete value changes, and the control command includes specific target parameters (e.g., volume value, temperature value) or adjustment amount (e.g., +1, -5%).

[0117] Optionally, the switching mode is used to change a binary state or a group of finite states, and the control command contains a clear target state (e.g., "on", "off", "on", "off").

[0118] Optionally, the confirmation mode is used to trigger a one-time action or confirm the current interface state without changing the parameter value, but instead executes specific operation logic (such as "Save", "Next", "Apply").

[0119] In this embodiment, the accuracy and real-time performance of control commands are ensured through structured mapping and a control command generation mechanism based on the current state, thereby achieving stable and reliable execution of vehicle functions.

[0120] The vehicle in-vehicle function control method of this application embodiment overcomes the limitation of incomplete data obtained from a single sensor by fusing multimodal perception data of at least one driver and passenger in the vehicle to obtain fused features. Then, the fused features are identified to obtain the operation category to which the operation belongs and the confidence level of the operation category. When the confidence level is lower than a confidence level threshold, the operation category is corrected to avoid safety hazards caused by misoperation. Control commands are then generated based on the corrected high-confidence operation category to drive the execution of the in-vehicle function, solving the technical problem of low accuracy in vehicle in-vehicle function control and achieving the technical effect of improving the accuracy of vehicle in-vehicle function control.

[0121] The above technical solutions of the embodiments of this application will be further illustrated below with reference to preferred embodiments.

[0122] Currently, with the development of intelligent vehicles and the popularization of human-machine interaction technology, in-vehicle gesture recognition is gradually becoming an important way to assist drivers. Through gesture recognition, drivers can operate functions such as volume adjustment, navigation switching, and air conditioning control without touching the instrument panel or central control screen while driving.

[0123] In related technologies, a single visual sensor (such as an in-vehicle camera) is often used for gesture capture and recognition. For example, gesture classification is achieved by extracting key features of the hand, or image recognition algorithms are used to judge gesture actions.

[0124] Currently, the aforementioned methods primarily rely on visual information, which is easily affected by factors such as changes in lighting, hand gestures obstructing the view, and deviations in the driver's hand angle, leading to recognition errors. Furthermore, some systems have attempted to introduce depth cameras or infrared sensors to assist in recognition, but lack effective fusion of multi-source information and real-time error correction mechanisms, making it difficult to guarantee recognition accuracy and stability in complex driving environments.

[0125] In real-world scenarios, single-sensor approaches are highly susceptible to environmental factors. For example, infrared sensor-based solutions experience a significant drop in recognition rate under direct sunlight; camera-based solutions lack accuracy at night and in backlit environments; and millimeter-wave radar-based solutions suffer from limited gesture differentiation. These shortcomings lead to frequent user experience issues such as misrecognition and delayed feedback during actual interactions, and can even pose safety hazards while driving.

[0126] To address the aforementioned issues, some technologies have incorporated deep learning models for fusion judgment. However, due to the lack of systematic modeling of the complex in-vehicle environment (such as bumps, strong electromagnetic interference, and occupant occupancy), robustness remains insufficient. Especially in scenarios of "false triggering" and "command ambiguity," traditional error correction methods often rely on secondary confirmation, increasing the risk of driver distraction.

[0127] To address the aforementioned issues, this application provides a method for recognizing and correcting gesture commands using multi-source sensors in vehicles. Driver gesture information is collected using multiple sensors (e.g., cameras, depth cameras, and inertial sensors). Feature extraction is performed on the data collected by each sensor, and a multimodal fusion algorithm is used to generate a comprehensive gesture feature vector. Preliminary recognition is performed based on the gesture feature vector and a preset gesture model to obtain candidate gesture categories. The driver's historical operation records and the vehicle's current state are used as correction references to dynamically correct the preliminary recognition results, improving recognition accuracy. The final recognition result is output and converted into vehicle control commands for adjusting in-vehicle functions such as volume, navigation, and air conditioning.

[0128] Figure 2 This is a flowchart of a vehicle multi-source sensor recognition and error correction gesture command method according to an embodiment of this application, such as... Figure 2 As shown, the method may include the following steps.

[0129] Step S201: Collect data from multiple sensor sources.

[0130] In this embodiment, driver gestures and environmental status information can be collected collaboratively using red, green, and blue (RGB) cameras, infrared cameras, depth cameras, ultrasonic sensors, microphone arrays, and other technologies.

[0131] Step S202: Perform fusion processing on the multi-source sensor data.

[0132] In this embodiment, feature vectors from different sensors can be input into the fusion module, and weighted fusion or time-series modeling algorithms can be used to generate comprehensive features.

[0133] Step S203: Perform preliminary gesture recognition.

[0134] In this embodiment, preliminary gesture recognition can be performed using a convolutional neural network or a skeletal point sequence model, and preliminary recognition results can be output.

[0135] Step S204: Is the confidence level lower than the confidence threshold?

[0136] In this embodiment, it can be determined whether the confidence level is lower than the confidence level threshold. If the confidence level is lower than the confidence level threshold, then step S206 is executed; otherwise, step S205 is executed.

[0137] Step S205: Output the final control command and execute it.

[0138] In this embodiment, if the confidence level is higher than the confidence level threshold, the final control command can be output and executed.

[0139] Step S206: Enter the error correction mechanism.

[0140] In this embodiment, if the confidence level is lower than the confidence level threshold, for example, 85%, then the error correction mechanism is activated.

[0141] Step S207: Output and execute the final control command after error correction.

[0142] In this embodiment, after error correction, the final control command can be output and executed.

[0143] Step S208: Feedback the execution result to the perception module.

[0144] In this embodiment, the final control command is output, and the execution result can be fed back to the sensing module to achieve closed-loop optimization.

[0145] In this embodiment, in specific application scenarios, driver gestures can be detected by region. This means that the possible functional area corresponding to the gesture is determined based on the spatial position of the driver's hand relative to the steering wheel and central control screen. For example, when the gesture is located on the central control screen, it is prioritized to determine whether it is related to navigation or multimedia; when the gesture is located on the side of the steering wheel, it is prioritized to associate it with driver assistance functions. This region detection mechanism effectively reduces the gesture recognition search space and improves real-time performance and robustness.

[0146] Figure 3 This is a flowchart of a gesture recognition and error correction method according to an embodiment of this application, such as... Figure 3 As shown, the method may include the following steps.

[0147] Step S301: Extract gesture features.

[0148] In this embodiment, gesture skeletal points, trajectories, and motion features can be extracted.

[0149] Step S302 yields preliminary classification and recognition results.

[0150] In this embodiment, by extracting gesture skeletal points, trajectories, and motion features, preliminary classification and recognition results can be obtained, namely, candidate categories.

[0151] Step S303: Is the confidence level lower than the confidence threshold?

[0152] In this embodiment, it can be determined whether the confidence level is lower than the confidence level threshold. If the confidence level is lower than the confidence level threshold, then step S305 is executed; otherwise, step S304 is executed.

[0153] Step S304, output the result.

[0154] In this embodiment, if the confidence level is higher than or equal to the confidence level threshold, the result can be output.

[0155] Step S305: Enter the error correction mechanism.

[0156] In this embodiment, if the confidence level is lower than the confidence level threshold, an error correction mechanism can be initiated, such as performing context logic comparison, historical habit matching, and multimodal assisted confirmation.

[0157] Optionally, contextual logic comparison can combine vehicle speed, driving task, and current functional scenario to determine whether the gesture is reasonable. Historical habit matching can utilize a database of frequently used user operations to correct ambiguous results. Multimodal assisted confirmation can be corrected by combining voice input and redundant sensor signals.

[0158] Step S306: Output the final recognition result after error correction.

[0159] In this embodiment, after performing context logic comparison, historical habit matching, and multimodal assisted confirmation, the results can be weighted and the final identification result can be generated and executed.

[0160] In this embodiment, an identity differentiation and access control mechanism can be introduced for multi-occupant scenarios within a vehicle. Through a driver monitoring system (DMS) and seat pressure sensors, gesture commands from the driver, front passenger, and rear passengers can be distinguished, and the driver's actions will be prioritized in case of conflict. Simultaneously, certain functional gestures from the front passenger (e.g., air conditioning adjustment, window raising / lowering) are allowed within safe limits, while gestures involving vehicle dynamic control (e.g., automatic lane changing, parking) are completely blocked to ensure driving safety.

[0161] Figure 4 This is a flowchart of a multimodal fusion determination method according to an embodiment of this application, such as... Figure 4 As shown, the method may include the following steps.

[0162] Step S401: Obtain key points of the hand.

[0163] In this embodiment, key hand points and posture data can be provided via a camera.

[0164] Step S402: Determine the gesture trajectory.

[0165] In this embodiment, three-dimensional trajectory information can be provided by a depth camera / millimeter-wave radar.

[0166] Step S403: Perform speech or sound source detection.

[0167] In this embodiment, a microphone array can be used to provide sound source localization and semantic assistance.

[0168] Step S404: Perform feature extraction and weighted judgment.

[0169] In this embodiment, feature weighting and timing determination can be performed in the fusion module.

[0170] Step S405 yields preliminary gesture recognition results.

[0171] In this embodiment, after feature weighting and timing determination are performed in the fusion module, preliminary gesture recognition results can be obtained.

[0172] Step S406: Integrate confidence calculation and error correction.

[0173] In this embodiment, if there is a conflict between multiple data sources, the confidence level is recalculated and error correction is initiated.

[0174] Step S407: Output the final control command.

[0175] In this embodiment, when the fusion results are inconsistent or have low confidence, error correction logic is entered, and finally control commands are output.

[0176] Optionally, a "soft confirmation" strategy can be designed into the error correction mechanism. When the confidence level of the current gesture recognition is detected to be lower than a set threshold, but close to a common command, a non-intrusive confirmation can be given to the driver through voice prompts or head-up display (HUD) prompts, instead of directly executing or completely ignoring the command. This strategy further reduces the probability of misoperation while maintaining smooth interaction.

[0177] In this embodiment, the above method enables a complete closed-loop process of gesture data acquisition, feature fusion, recognition, and dynamic error correction based on multi-source sensors in an in-vehicle environment. It not only maintains stable recognition accuracy under different lighting conditions, vibrations, and complex road conditions, but also effectively reduces the probability of misrecognition and false triggering through contextual constraints and multimodal information assistance. Simultaneously, the error correction mechanism can dynamically optimize based on driving status and user history habits, ensuring that gesture control results are highly consistent with driving safety. Therefore, this embodiment can improve the accuracy, robustness, and safety of in-vehicle human-machine interaction, possessing broad practical application value and potential for widespread adoption.

[0178] The embodiments of this application can be applied not only to passenger vehicles, but also to various travel scenarios such as commercial vehicles, ride-hailing services, and future autonomous taxis. In ride-hailing scenarios, gesture interaction can reduce language communication barriers between drivers and passengers; in autonomous driving scenarios, gesture interaction and error correction mechanisms can provide a more efficient and safer control method when passengers are multitasking. Furthermore, the multi-source sensor fusion framework and error correction method of the embodiments of this application can also be extended to fields such as smart homes and drone operation.

[0179] This application aims to address the problems of low accuracy, poor robustness, and high false recognition rate in existing vehicle gesture recognition technologies. Specifically, it addresses the following technical issues: Existing systems primarily rely on a single visual sensor, making them susceptible to factors such as changes in lighting, gesture occlusion, and hand angle deviations, leading to gesture recognition errors; multi-source sensor information is not effectively fused, and accurate judgments cannot be made when data from different sensors conflict or are redundant; the lack of a dynamic error correction mechanism that combines the driver's historical operation data with the vehicle's current state results in frequent misoperations and reduced driving safety; and existing gesture recognition systems lack stability at night, in strong light, or under complex road conditions, making it difficult to provide continuous and reliable interactive services to the driver.

[0180] To address the aforementioned issues, this application provides a vehicle-mounted gesture recognition method based on multi-source sensor fusion and dynamic error correction, aiming to improve recognition accuracy, system robustness, and driving safety. By employing multi-source sensor fusion and a historical operation dynamic error correction mechanism, this application achieves the following beneficial effects: It effectively reduces the impact of environmental factors such as lighting, occlusion, and gesture angle deviation on recognition accuracy, improving gesture recognition stability. Through multimodal information fusion, the system can accurately determine the gesture category even when there is conflict or redundancy in information from different sensors. The dynamic error correction mechanism, combined with historical operations and vehicle status, significantly reduces the error rate and improves driving safety. It maintains continuous and reliable gesture recognition capabilities even in complex road conditions, at night, and in strong light environments, providing drivers with an efficient and convenient human-machine interaction experience.

[0181] Compared with the prior art, the embodiments of this application have made significant improvements in the following three aspects.

[0182] Multi-source sensor hierarchical fusion. By constructing hierarchical feature extraction models for sensors such as vision, infrared, and millimeter wave, spatial trajectory features, deep structural features, and micro-motion change features are obtained respectively. A confidence-based weighted fusion mechanism is then adopted to improve adaptability to complex scenes.

[0183] Context-aware error correction mechanism. This application's embodiments not only perform real-time correction during gesture recognition, but also combine vehicle speed, driving mode, and task context information (e.g., navigation, entertainment, or call status) to dynamically correct gesture commands, avoiding erroneous operations caused by momentary recognition deviations.

[0184] Closed-loop learning and adaptive optimization. This application proposes a user feedback-driven lightweight model update method that can gradually optimize personalized gesture recognition accuracy while ensuring controllable computing power, thereby achieving the effect of becoming more accurate with long-term use.

[0185] In summary, the embodiments of this application not only improve the accuracy of in-vehicle gesture recognition, but also significantly reduce the risk of misoperation.

[0186] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0187] According to an embodiment of this application, an embodiment of a vehicle-mounted function control device is provided. It should be noted that the vehicle-mounted function control device can be used to execute the above-described vehicle-mounted function control method.

[0188] Figure 5 This is a schematic diagram of a vehicle-mounted function control device according to an embodiment of this application, such as... Figure 5 As shown, the vehicle's on-board function control device 500 may include: a fusion unit 502, an identification unit 504, a correction unit 506, a generation unit 508, and a control unit 510.

[0189] The fusion unit 502 is used to perform fusion processing on the multimodal perception data of at least one driver or passenger in the vehicle to obtain fusion features, wherein the fusion features are used to characterize the operations performed by the driver or passenger in response to the vehicle's onboard functions.

[0190] The identification unit 504 is used to identify the fused features to obtain the operation category to which the operation belongs and the confidence level of the operation category.

[0191] The correction unit 506 is used to correct the operation category in response to the confidence level being lower than the confidence level threshold.

[0192] The generation unit 508 is used to generate control commands for the vehicle based on the modified operation category.

[0193] The control unit 510 is used to control the vehicle to perform onboard functions in response to control commands.

[0194] Optionally, the correction unit 506 includes: an acquisition subunit, configured to acquire vehicle status information in response to a confidence level lower than a confidence level threshold, wherein the status information is used to represent the vehicle's operating status and / or the working status of the on-board function; a matching subunit, configured to match the operation category and status information to obtain a matching result; and a correction subunit, configured to correct the operation category based on the matching result.

[0195] Optionally, the correction subunit includes: a first acquisition subunit, configured to acquire, in response to a matching result indicating that the operation category and status information failed to match, operation information performed by the driver / passenger on the vehicle function during a historical time period, wherein the operation information includes multiple historical operation categories and execution frequencies; and a first correction subunit, configured to correct the operation category based on the operation information.

[0196] Optionally, the first correction subunit includes: a filtering subunit for filtering operation information to obtain historical operation categories with execution frequencies higher than a frequency threshold; a first extraction subunit for extracting sound source localization information and semantic features from audio data in multimodal perception data in response to the existence of multiple historical operation categories; and a second correction subunit for correcting the operation category based on the sound source localization information and semantic features.

[0197] Optionally, the first correction subunit includes: a parsing subunit, configured to parse the semantic features in the audio data of the multimodal perception data in response to detecting that the sound source localization information in the audio data of the multimodal perception data comes from the driver or passenger, and obtain the initial operation category corresponding to the semantic features; and a third correction subunit, configured to correct the operation category in response to the inconsistency between the initial operation category and the operation category.

[0198] Optionally, the fusion unit 502 includes: a second acquisition subunit, used to acquire multimodal perception data using multi-source sensors deployed in the vehicle; a first determination subunit, used to determine target multimodal perception data corresponding to the target part of the driver / passenger object from the multimodal perception data; and a fusion subunit, used to call a fusion model to perform fusion processing on the target multimodal perception data to obtain fusion features, wherein the fusion model is obtained by training a neural network using multimodal perception data samples of the driver / passenger object samples.

[0199] Optionally, the multimodal perception data includes at least one of the following: position features, attitude features, three-dimensional structural features, micro-motion change features, and kinematic features of the target part, wherein the kinematic features include acceleration features and angular velocity features. The second acquisition subunit includes: a second extraction subunit for extracting position features and attitude features from the vehicle's image data using a vision sensor deployed in the vehicle; a third extraction subunit for extracting three-dimensional structural features and micro-motion change features from the vehicle's point cloud data using a perception sensor deployed in the vehicle; and a fourth extraction subunit for extracting kinematic features from the vehicle's motion data using an inertial sensor deployed in the vehicle.

[0200] Optionally, the recognition unit 504 includes: a second determining subunit, used to determine the operating function area to which the target part belongs based on the position information of the target part of the driver / passenger object in the vehicle, wherein the operating function area is used to represent the orientation of the target part in the vehicle, and the orientation corresponds to the functional module of the vehicle function; a third determining subunit, used to determine the candidate operation category set of the driver / passenger object based on the operating function area; and a recognition subunit, used to call the recognition model to recognize the fused features and the candidate operation category set to obtain the operation category and confidence level, wherein the recognition model is obtained by training a convolutional neural network using multimodal perception data samples of the driver / passenger object sample and state information samples of the vehicle sample.

[0201] Optionally, the generation unit 508 includes: a fourth determining subunit, used to determine the identification information corresponding to the modified operation category based on the modified operation category; a fifth determining subunit, used to determine the vehicle function and the control mode of the vehicle function according to the identification information; and a generation subunit, used to generate control commands based on the current state and control mode of the vehicle function.

[0202] Optionally, the control unit 510 includes: an adjustment subunit, configured to adjust the vehicle function parameters corresponding to the current state according to the target parameters in the control command in response to the control mode being adjustment mode; a switching subunit, configured to switch the vehicle function from the current state to the target state according to the target state in the control command in response to the control mode being switching mode; and an execution subunit, configured to execute the operation corresponding to the vehicle function in the current state in response to the control mode being confirmation mode.

[0203] In the vehicle's onboard function control device of this embodiment, the fusion unit 502 performs fusion processing on the multimodal perception data of at least one driver / passenger in the vehicle to obtain fusion features, wherein the fusion features are used to characterize the operation performed by the driver / passenger on the vehicle's onboard functions; the identification unit 504 identifies the fusion features to obtain the operation category to which the operation belongs and the confidence level of the operation category; the correction unit 506 corrects the operation category in response to the confidence level being lower than the confidence level threshold; the generation unit 508 generates a vehicle control command based on the corrected operation category; and the control unit 510 controls the vehicle to execute the onboard functions in response to the control command, thereby solving the technical problem of low accuracy in vehicle onboard function control and achieving the technical effect of improving the accuracy of vehicle onboard function control.

[0204] According to another aspect of the embodiments of this application, an electronic device is also provided. Figure 6 This is a schematic diagram of an electronic device according to an embodiment of this application, such as... Figure 6As shown, the electronic device 60 may include a processor 610 and a memory 620. The memory 620 stores an executable program. The processor 610 can be used to run the program, wherein the program executes the methods described in the embodiments of this application.

[0205] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application when it runs.

[0206] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0207] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0208] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0209] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of this application.

[0210] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0211] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0212] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0213] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0214] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0215] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for controlling onboard functions of a vehicle, characterized in that, include: Multimodal perception data of at least one driver or passenger in the vehicle are fused to obtain fused features, wherein the fused features are used to characterize the operations performed by the driver or passenger in response to the vehicle's onboard functions; The fused features are identified to obtain the operation category to which the operation belongs, and the confidence level of the operation category; In response to the confidence level falling below a confidence threshold, the operation category is corrected; Based on the revised operation category, control commands for the vehicle are generated; In response to the control command, the vehicle is controlled to perform the onboard function.

2. The method according to claim 1, characterized in that, In response to the confidence level falling below a confidence threshold, the operation category is corrected, including: In response to the confidence level being lower than the confidence level threshold, the status information of the vehicle is obtained, wherein the status information is used to represent the operating status of the vehicle and / or the working status of the in-vehicle function; The operation category and the status information are matched to obtain a matching result; Based on the matching results, the operation category is modified.

3. The method according to claim 2, characterized in that, Based on the matching results, the operation category is modified, including: In response to the matching result indicating that the operation category and the status information failed to match, the operation information performed by the driver / passenger on the in-vehicle function in a historical time period is obtained, wherein the operation information includes multiple historical operation categories and execution frequencies; Based on the operation information, the operation category is modified.

4. The method according to claim 3, characterized in that, Based on the operation information, the operation category is modified, including: The operation information is filtered to obtain the historical operation categories whose execution frequency is higher than the frequency threshold; In response to the fact that there are multiple types of historical operation categories, sound source localization information and semantic features are extracted from the audio data in the multimodal perception data; Based on the sound source localization information and the semantic features, the operation category is modified.

5. The method according to claim 3, characterized in that, Based on the operation information, the operation category is modified, including: In response to detecting that the sound source localization information in the audio data of the multimodal perception data comes from the driver or passenger, the semantic features in the audio data of the multimodal perception data are parsed to obtain the initial operation category corresponding to the semantic features; In response to the inconsistency between the initial operation category and the operation category, the operation category is corrected.

6. The method according to claim 1, characterized in that, The multimodal perception data of at least one driver or passenger in the vehicle are fused to obtain fused features, including: The multi-source sensors deployed in the vehicle are used to acquire the multimodal perception data; From the multimodal perception data, determine the target multimodal perception data corresponding to the target part of the driving and riding object; The fusion model is invoked to perform fusion processing on the target multimodal perception data to obtain the fusion features. The fusion model is obtained by training a neural network using multimodal perception data samples of the driving and riding object samples.

7. The method according to claim 6, characterized in that, The multimodal sensing data includes at least one of the following: positional features, attitude features, three-dimensional structural features, micro-motion change features, and kinematic features of the target part, wherein the kinematic features include acceleration features and angular velocity features. The multimodal sensing data is acquired using multi-source sensors deployed in the vehicle, including: Using visual sensors deployed in the vehicle, the position features and the pose features are extracted from the image data of the vehicle; Using the perception sensors deployed in the vehicle, the three-dimensional structural features and the micro-motion change features are extracted from the point cloud data of the vehicle; The kinematic features are extracted from the vehicle's motion data using inertial sensors deployed in the vehicle.

8. The method according to claim 1, characterized in that, The fused features are identified to obtain the operation category to which the operation belongs, and the confidence level of the operation category, including: Based on the location information of the target part of the driver / passenger in the vehicle, the operating function area to which the target part belongs is determined, wherein the operating function area is used to indicate the orientation of the target part in the vehicle, and the orientation corresponds to the functional module of the vehicle function; Based on the aforementioned operation function area, a set of candidate operation categories for the driver and passenger is determined; The recognition model is invoked to identify the fused features and the candidate operation category set to obtain the operation category and the confidence level. The recognition model is obtained by training a convolutional neural network using multimodal perception data samples of driving and riding objects and state information samples of vehicle samples.

9. The method according to any one of claims 1 to 8, characterized in that, Based on the revised operation category, control commands for the vehicle are generated, including: Based on the revised operation category, determine the identification information corresponding to the revised operation category; The vehicle function and its control mode are determined according to the identification information. The control command is generated based on the current state of the vehicle function and the control mode.

10. The method according to claim 9, characterized in that, In response to the control command, controlling the vehicle to perform the onboard function includes: In response to the control mode being adjustment mode, the vehicle function parameters corresponding to the current state are adjusted according to the target parameters in the control command; In response to the control mode being a switching mode, the vehicle function is switched from the current state to the target state according to the target state in the control command; In response to the control mode being confirmed, the operation corresponding to the vehicle function in the current state is executed.

11. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 10.

Images