Method and device for gesture recognition control of vehicle based on cooperation of vehicle and mobile terminal

Abstract

The application provides a gesture control method and device based on cooperation of a vehicle and a mobile terminal, the method comprising: acquiring a position area of a mobile terminal in which a gesture control application is installed in a vehicle, the position area being a driving area or a non-driving area; determining control authority of the mobile terminal according to the position area; receiving a gesture control instruction sent by the mobile terminal; wherein the gesture control instruction is generated based on a user gesture operation captured by the mobile terminal; and executing the gesture control instruction if the gesture control instruction is within the control authority. Through combination of position sensing and authority determination, differentiated control of the driving area and the non-driving area is realized, direct execution of an unauthorized terminal instruction is avoided, and safety, stability and controllability of gesture control are improved.

Description

Technical Field

[0001] This application relates to the field of vehicles, and more particularly to a method and device for vehicle control based on gesture recognition in collaboration between a vehicle and a mobile terminal. Background Technology

[0002] With the rapid development of smart cockpit technology, drivers have an increasingly urgent need for vehicle control methods that require "no looking down and no contact." In real-world driving scenarios, drivers can quickly perform operations such as adjusting the air conditioning temperature, opening and closing windows, and switching blind spot monitoring via gestures, while ensuring both safety and convenience. For example, when driving at high speeds, drivers need to quickly adjust the air conditioning temperature to avoid fatigue driving, or activate blind spot monitoring before turning to improve driving safety.

[0003] In existing technologies, gesture recognition devices generally rely on dedicated in-vehicle gesture recognition hardware. A common method is to detect user gestures using cameras, but this is limited by cost and installation conditions, making it difficult to implement in many vehicles. Some solutions allow gesture recognition via a handheld mobile phone, but rear-seat passengers might accidentally trigger gesture controls, creating safety hazards while driving. For example, a child playing with a mobile phone might accidentally trigger a gesture control command, causing issues with vehicle lights and posing a safety risk.

[0004] Therefore, there is an urgent need for a vehicle control solution that requires no new hardware, is compatible with multiple vehicle models, and achieves low cost and high safety. Summary of the Invention

[0005] This application provides a method and device for vehicle control based on gesture recognition in collaboration between a vehicle and a mobile terminal. The method identifies the location of a mobile terminal with a gesture control application installed on it, and manages the vehicle control capability of the mobile terminal based on the location difference. On this basis, the method verifies the permissions and controls the execution of the gesture control commands generated by the mobile terminal, thereby realizing gesture control in collaboration between the vehicle and the mobile terminal.

[0006] In a first aspect, this application provides a method for vehicle control based on gesture recognition in collaboration between a vehicle and a mobile terminal, applicable to a vehicle, the method comprising:

[0007] Obtain the location area of ​​the mobile terminal with gesture control application installed in the vehicle, which can be either the driving area or the non-driving area;

[0008] Determine the vehicle control permissions of the mobile terminal based on the location area;

[0009] Receives gesture control commands sent by a mobile terminal; where the gesture control commands are generated based on the user's gesture operations captured by the mobile terminal.

[0010] If the gesture control command is within the vehicle control authority, then the gesture control command will be executed.

[0011] Furthermore, the method also includes:

[0012] Obtain the vehicle's driving status;

[0013] Adjust vehicle control permissions based on driving status.

[0014] Furthermore, vehicle control permissions can be adjusted based on driving status, including:

[0015] When the driving status indicates that the vehicle is parked, the preset vehicle control permissions of the mobile terminal in the non-driving area are enabled.

[0016] Furthermore, the method also includes:

[0017] If the gesture control command is not within the vehicle control permissions, a permission insufficient message will be sent to the mobile terminal.

[0018] Furthermore, the method also includes:

[0019] The number of mobile terminals was obtained;

[0020] If the number is greater than 1, determine the control priority of each mobile terminal based on the location area of ​​each mobile terminal;

[0021] When a conflict is detected between the gesture control commands of two mobile terminals, the gesture control command with higher control priority is executed.

[0022] Secondly, embodiments of this application provide a method for vehicle control based on gesture recognition in collaboration between a vehicle and a mobile terminal, applied to a mobile terminal, the method comprising:

[0023] In response to the user's gesture recognition function, the system acquires gesture data collected by the phone's sensors based on the user's gesture operation.

[0024] Based on the preset gesture-command correspondence, determine the gesture control command corresponding to the gesture data;

[0025] Send gesture control commands to the vehicle.

[0026] Thirdly, this application provides a device for vehicle control based on gesture recognition in collaboration between a vehicle and a mobile terminal, the device comprising:

[0027] The location acquisition module is used to acquire the location area of ​​the mobile terminal with gesture control application installed in the vehicle. The location area is either the driving area or the non-driving area.

[0028] The permission determination module is used to determine the vehicle control permissions of the mobile terminal based on the location area;

[0029] The instruction receiving module is used to receive gesture control instructions sent by the mobile terminal; wherein, the gesture control instructions are generated based on the user's gesture operations captured by the mobile terminal.

[0030] The execution module is used to execute the gesture control command if it is within the vehicle control permissions.

[0031] Fourthly, this application provides a device for vehicle control based on gesture recognition in collaboration between a vehicle and a mobile terminal, the device comprising:

[0032] The gesture acquisition module is used to respond to the user's operation of waking up the gesture recognition function and to acquire gesture data collected by the mobile phone sensors based on the user's gesture operation;

[0033] The instruction determination module is used to determine the gesture control instruction corresponding to the gesture data based on the preset gesture-instruction correspondence.

[0034] The sending module is used to send gesture control commands to the vehicle.

[0035] Fifthly, this application provides an electronic device, including: a memory and a processor;

[0036] The memory stores instructions that the computer executes;

[0037] The processor executes computer execution instructions stored in memory, causing the processor to perform the methods provided above, or to perform the methods provided above.

[0038] Sixthly, this application provides a vehicle, including a vehicle body, a display device, a communication device, and a controller, the controller being used to include the method provided above.

[0039] In a seventh aspect, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement or perform the methods provided above.

[0040] Eighthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided above, or the method provided above.

[0041] This application obtains the location area of ​​a mobile terminal with a gesture control application installed inside the vehicle and determines the corresponding vehicle control permissions based on the location area, enabling differentiated management of the gesture control capabilities of mobile terminals in different areas. By receiving gesture control commands generated by the mobile terminal based on captured user gesture operations and executing the gesture control commands when they are within the vehicle control permissions, the accuracy and security of gesture control can be improved, thereby reducing the risk of misoperation and improving the human-computer interaction experience of the vehicle. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] Figure 1 The flowchart of the gesture recognition vehicle control method based on vehicle and mobile terminal collaboration provided in this application is shown below. Figure 1 ;

[0044] Figure 2 This application provides a flowchart illustrating a method for vehicle control based on gesture recognition in collaboration between the vehicle and a mobile terminal. Figure 2 ;

[0045] Figure 3 Schematic diagram of the structure of the gesture recognition vehicle control device based on vehicle and mobile terminal collaboration provided in this application Figure 1 ;

[0046] Figure 4 Schematic diagram of the structure of the gesture recognition vehicle control device based on vehicle and mobile terminal collaboration provided in this application Figure 2 ;

[0047] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0048] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0050] Existing gesture control solutions for vehicles mainly fall into two categories: The first is dedicated in-vehicle gesture control, which relies on factory-installed hardware such as 3D cameras, ToF sensors, and millimeter-wave radar to control the vehicle by recognizing the driver's gestures within the cabin. This type of solution is costly, only applicable to mid-to-high-end new cars, and cannot be adapted to older or lower-spec vehicles. Furthermore, its gesture recognition rate is easily affected by cabin space and lighting conditions. The second approach involves gesture recognition via a handheld mobile phone, but passengers may accidentally trigger gesture control commands while using their phones, creating a driving safety hazard.

[0051] In summary, existing gesture control methods suffer from cumbersome operation, significant safety risks, poor adaptability, and high costs, failing to meet the demand for low-cost and widely adaptable dynamic vehicle control.

[0052] In view of this, this application establishes a correspondence between the location area of ​​the mobile terminal and vehicle control permissions, and uses this as a prerequisite for executing gesture control commands. This enables the vehicle to first complete permission determination when receiving a control request before deciding whether to respond, thereby preventing commands generated by the terminal in the non-driving area from being executed directly without constraints. This technical approach does not rely on complex physical control component extensions, but rather achieves permission-based management of gesture control behavior through collaborative judgment between the vehicle and the mobile terminal. This creates a clear boundary between the driving area and the non-driving area in the control logic, thereby improving the security, stability, and controllability of in-vehicle interaction.

[0053] The mobile terminal involved in this application can be an electronic device such as a mobile phone, tablet, or wearable smart device.

[0054] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0055] Example 1

[0056] Figure 1 The flowchart of the gesture recognition vehicle control method based on vehicle and mobile terminal collaboration provided in this application is shown below. Figure 1 ,like Figure 1 As shown, it includes:

[0057] S101: Obtain the location area of ​​the mobile terminal with the gesture control application installed in the vehicle. The location area is either the driving area or the non-driving area.

[0058] In this embodiment, the executing entity can be a vehicle controller, a vehicle main control unit, a cockpit domain controller, or an onboard gateway device that communicates with the vehicle. The mobile terminal can be a smartphone, tablet, wearable smart device, or other terminal device with gesture acquisition, processing, and wireless communication capabilities. A gesture control application is installed on the mobile terminal; this application refers to a software program deployed on the mobile terminal that collects user gestures, parses gesture semantics, and generates control commands that the vehicle can recognize. A location area refers to a functional area within the vehicle cabin divided based on spatial location, seating arrangement, operational risk level, or control safety boundaries. The driving area corresponds to the driver's seat and its pre-defined adjacent space, while the non-driving area typically corresponds to the front passenger seat, rear seats, or other spaces without high-level vehicle control permissions.

[0059] In practice, the vehicle first establishes a communication connection with a mobile terminal that has the gesture control application installed. This communication link can be established via wired connection, Bluetooth, Bluetooth Low Energy, Wi-Fi Direct, in-vehicle hotspot, or UWB short-range communication. After establishing the connection, the vehicle initiates a terminal identification request to the mobile terminal to confirm whether the terminal has completed application registration, device binding, and basic identity verification. Once the mobile terminal meets the access requirements, the vehicle begins to acquire the mobile terminal's location within the vehicle.

[0060] Location can be determined through the fusion of various in-vehicle sensing information. For example, UWB positioning modules can be deployed inside the vehicle. The spatial coordinates of the mobile terminal are calculated using distance measurements between multiple positioning modules and the mobile terminal within the cabin. Then, based on a pre-established 3D cabin area mapping table, the terminal's location is mapped to either the driving or non-driving area. Alternatively, a multi-point Bluetooth BLE array can be used to determine the mobile terminal's location. Multiple Bluetooth BLE antennas are placed in the front, rear, and trunk areas. Multi-point RSSI ranging and spatial triangulation can distinguish between the driver's seat, passenger seat, left rear seat, and right rear seat. Furthermore, in-vehicle cameras can capture cabin images, visually recognizing the mobile terminal's appearance, the user's holding posture, and the terminal's position relative to the seat. This information is then combined with image segmentation or object detection models to determine whether the terminal is located in the driver's area or another area.

[0061] For vehicles without a high-precision positioning module, data such as Bluetooth signal strength, ultrasonic ranging, infrared proximity detection, seat pressure sensors, and seatbelt usage status can be used for joint inference. For example, if only the driver's seat is occupied and the signal strength between the mobile terminal and the Bluetooth beacon on the driver's side meets a threshold condition, the terminal can be determined to be located in the driving area.

[0062] In one possible embodiment, the location region can also be obtained using the mobile terminal's own sensors. The mobile terminal can collect information from accelerometers, gyroscopes, gravity sensors, magnetometers, and screen orientation, and combine this information with the vehicle's driving direction, the vehicle's mounting position, and the relative communication delay between the terminal and the vehicle to infer the terminal's placement and holding area. After receiving the attitude feature parameters uploaded by the terminal, the vehicle end matches them with the in-vehicle space model to determine the location region. For example, when the attitude angle, distance estimate, and signal propagation characteristics of the driver's side anchor point uploaded by the mobile terminal meet the driving area template, the vehicle end can classify the terminal as belonging to the driving area; when it is closer to the passenger side or rear side template, it is classified as belonging to the non-driving area. To improve recognition stability, the vehicle end can continuously acquire terminal location information at multiple sampling times and smooth the instantaneous location results using sliding time windows, Kalman filtering, weighted averaging, or state machine de-jittering strategies. The location region is only finally determined when several consecutive recognition results are consistent or the confidence level reaches a preset threshold.

[0063] Based on the above analysis, obtaining the location area of ​​the mobile terminal is a prerequisite for the entire gesture recognition vehicle control process. This step transforms the terminal's actual spatial location in the cabin into a logical label for permission decisions—either the driving area or the non-driving area—enabling the vehicle to possess spatial semantic judgment capabilities before receiving control requests. This allows for the differentiation between high-risk driving-related controls and interactions from non-driving seats, preventing the vehicle from executing control operations solely based on the received command. This establishes a security boundary for in-vehicle interactions from the source, improving the accuracy of subsequent permission determinations and the consistency of system responses. It should be understood that the above example is merely illustrative and not limiting.

[0064] S102: Determine the vehicle control permissions of the mobile terminal based on the location area.

[0065] Vehicle control permissions refer to the range of vehicle control capabilities that a specific mobile terminal is allowed to initiate and actually execute under current conditions. This range can be defined through permission levels, sets of allowed control items, risk level mapping tables, or control domain whitelists. To facilitate system processing, the vehicle can pre-establish a correspondence between location areas and permission templates. The driving area corresponds to the first permission template, and the non-driving area corresponds to the second permission template. The first permission template allows access to higher-level body control, comfort control, and infotainment control capabilities, while the second permission template allows access to low-risk control capabilities, query capabilities, or directly corresponds to no execution permission.

[0066] In some embodiments, the non-driving area can be further divided into a front non-driving area and a rear non-driving area, and the permissions corresponding to the front non-driving area and the rear non-driving area can be different.

[0067] In practice, after completing location area identification, the vehicle terminal calls the permission management module to read the stored area permission mapping table. This table may contain area identifiers, executable command categories, and prohibited command categories. The vehicle terminal first obtains an initial permission set based on the mobile terminal's current location area. For example, when the terminal is in the driving area, the initial permission set may include vehicle function operations such as window operation, door locking and unlocking, air conditioning parameter adjustment, seat position adjustment, rearview mirror adjustment, headlight switching, and media playback control. When the terminal is in a non-driving area, the initial permission set may be limited to air conditioning on / off, fan speed adjustment, media playback pause / switching, and rear seat comfort device control, while excluding high-risk commands related to vehicle movement.

[0068] In one possible embodiment, the vehicle can further refine the initial permission set based on the user authentication result. User authentication can be completed through mobile terminal login account, vehicle key binding relationship, biometric results, vehicle owner account authorization record, or near-field authentication token. If the terminal is located in the driving area and the current user has passed vehicle owner-level authentication, the vehicle can extend the permissions to the full driver's seat control range; if the terminal is located in the driving area but has only passed ordinary occupant authentication, only limited control permissions can be granted; if the terminal is located in the non-driving area, even if the user has passed authentication, high-risk controls can be kept prohibited according to security policies.

[0069] In some embodiments, the current state of the vehicle also participates in permission determination. For example, when the vehicle is parked, the driving area terminal may be allowed to execute more static control commands; when the vehicle is in motion, some commands that may affect driving safety may be automatically downgraded to an unexecutable state, even if these commands were originally within the driving area permission template, they still need to be filtered according to real-time safety conditions.

[0070] To facilitate rapid verification, the vehicle can encapsulate the confirmed vehicle control permissions into a permission token and associate it with the device identifier of the current mobile terminal. When a gesture control command is received subsequently, the vehicle directly calls this permission context to complete the comparison, eliminating the need for recalculation and reducing system latency. The vehicle periodically checks the mobile terminal's location. If a change in location is detected during the session, such as the terminal moving from the driving area to a non-driving area, the system can trigger a dynamic permission refresh, revoking existing high-level permissions and reloading the permission template corresponding to the new area to ensure that permissions always match the actual spatial location.

[0071] S103: Responds to the user's operation of waking up the gesture recognition function, and acquires gesture data collected by the mobile phone sensors based on the user's gesture operation.

[0072] The execution entity for steps S103-S105 is the mobile terminal. A matching gesture-based vehicle control application is installed on the mobile terminal (phone). This application does not require customization or modification of the phone's system, nor does it need to obtain low-level system permissions (such as root access). Only basic application permissions need to be manually granted by the user, specifically including: sensor usage permissions (for calling the gyroscope and accelerometer to collect gesture data); communication permissions (for signal interaction and command transmission with the vehicle's infotainment system); background operation permissions (for receiving real-time vehicle status and responding to gesture operations); and notification permissions (for receiving feedback from the vehicle's infotainment system). A corresponding collaborative control plugin is installed on the vehicle's infotainment system, supporting command reception and parsing via at least three communication methods: USB, Wi-Fi, and BitTorrent.

[0073] Gesture-based vehicle control commands refer to control data that can be parsed by the vehicle, generated by the mobile terminal after collecting, recognizing, semantically mapping, and encoding specific hand movements, micro-limb movements, or interaction trajectories performed by the user. User gesture operations can include waving, clicking, swiping, pinching, rotating, holding, continuous tapping, trajectory drawing, or aerial movements based on changes in the terminal's posture. The mobile terminal can capture gesture operations using infrared sensors, depth sensors, touchscreens, accelerometers, gyroscopes, proximity sensors, millimeter-wave proximity sensing units, or external gesture devices.

[0074] In practice, the gesture-based vehicle control application on the mobile terminal continuously monitors the predetermined gesture input channels, either in the foreground or in the background under authorized operation. When a user performs a gesture, the application first acquires raw sensor data. For example, when using an inertial sensor solution, the application continuously collects data on three-axis acceleration, angular velocity, and attitude angle changes, analyzes the terminal's waving direction, rotation amplitude, duration, and rhythm of movement, and determines the semantics of the user's gesture based on a preset gesture-command correspondence or a pre-trained gesture model, further mapping it to the corresponding vehicle control command.

[0075] For the calibration of in-app gesture control commands, gesture recognition thresholds, such as angular velocity thresholds and acceleration thresholds, can be set through the app. The vehicle system status feedback method can also be configured to complete the collaborative calibration and ensure real-time and accurate signal transmission between the mobile phone and the vehicle system.

[0076] S104: Determine the gesture control command corresponding to the gesture data based on the preset gesture-command correspondence.

[0077] For example, the default gesture and vehicle control command correspondence is as follows (customizable): swaying the phone left and right (amplitude ≥ 30°) corresponds to adjusting the window height (swing left to raise the window, swing right to lower the window); swiping the phone up and down (amplitude ≥ 5cm) corresponds to adjusting the air conditioning temperature (swipe up to heat up, swipe down to cool down); rotating the phone clockwise (angle ≥ 90°) corresponds to turning on the right turn blind spot monitoring; rotating the phone counterclockwise (angle ≥ 90°) corresponds to turning off the right turn blind spot monitoring; quickly shaking the phone twice corresponds to pausing the gesture control mode.

[0078] In some embodiments, when generating a gesture control command, the mobile terminal can attach various auxiliary information to the command to facilitate parsing and verification by the vehicle. This auxiliary information may include a terminal device identifier, user account identifier, gesture type encoding, target control object encoding, generation timestamp, command validity period, gesture recognition confidence level, and current terminal posture summary. Subsequently, the application sends the gesture control command through an encrypted communication link established with the vehicle. Upon receiving the message, the vehicle's communication receiving module can unpack the data packet, perform integrity verification, source verification, and timestamp checking to confirm that the command originates from a bound mobile terminal with a valid session.

[0079] S105: Send gesture control commands to the vehicle.

[0080] In one possible embodiment, to reduce the risk of incorrect vehicle control due to misidentification, the mobile terminal can add a gesture confirmation mechanism before generating the gesture control command locally. For example, when a high-risk gesture is identified, the application first requires the user to perform a confirmation action within a short time window through interface prompts, vibration feedback, or voice broadcast. Only after successful confirmation will the vehicle control command be officially generated and sent.

[0081] S106: Receive a gesture control command sent by the mobile terminal; wherein the gesture control command is generated based on the user's gesture operation captured by the mobile terminal.

[0082] S107: If the gesture control command is within the vehicle control authority, then execute the gesture control command.

[0083] In this step, executing a gesture-based vehicle control command does not involve immediately issuing control upon receiving the command. Instead, permission comparison and security condition judgment must be completed before execution to ensure that the control action to be executed simultaneously meets both area permission restrictions and vehicle operation safety requirements. A gesture-based vehicle control command is considered to be within the vehicle control permissions, meaning that the control type, target control object, and operation level corresponding to the command are all within the permitted scope of the current mobile terminal's permission context and are not prohibited by the vehicle's current state, security policies, or conflict rules.

[0084] In one specific implementation, after receiving a gesture control command, the vehicle terminal uses a permission verification module to determine whether the gesture control command belongs to the set of commands that the current terminal can execute, based on a permission bitmap or whitelist. If the comparison result is that it is not within the permission range, the system can refuse to execute and return a failure response, insufficient permission prompt, or security restriction reason to the mobile terminal; if the comparison result is within the permission range, it further enters the pre-execution security verification. The security verification may include whether the vehicle is in a parked state, whether the vehicle speed is zero, whether the relevant doors are closed, whether child locks are present, whether the effective command conflicts with the current system state, whether the body control network is available, whether the target actuator is faulty, and whether a higher priority control request has been received, etc. Only when both the permission verification and security verification pass, does the vehicle terminal generate the corresponding execution command and send it to the target control module through the vehicle bus, such as the air conditioning controller, door controller, window controller, seat controller, lighting controller, central entertainment controller, or body domain controller.

[0085] In one possible implementation, the vehicle-side can incorporate conflict handling and rollback mechanisms. For example, when multiple mobile terminals send conflicting gesture control commands almost simultaneously, the system can arbitrate based on terminal region level, user identity level, command timestamp, controlled object priority, and current vehicle status, executing only the command with higher priority and conforming to the permission rules. For continuous adjustment commands, such as continuously increasing volume or temperature, the system can merge multiple gesture commands into a single control sequence to reduce bus load and improve response smoothness. After execution, the vehicle-side can record log information, including terminal identifier, location region, permission result, command content, execution status, and failure reason, for subsequent auditing, fault analysis, and strategy optimization.

[0086] In this embodiment, by establishing a correspondence between the location area of ​​the mobile terminal in the vehicle cabin and the vehicle control permissions, and using this permission relationship as a necessary condition for executing gesture control commands, the vehicle can first complete area identification, permission loading, command verification, and security checks when receiving control requests from different terminal units, before deciding whether to execute the corresponding operation. Without relying on additional complex physical control structures, a clear authorization boundary can be formed for multi-terminal gesture interactions within the vehicle. This solves the technical problems of difficulty in distinguishing control permissions for terminals in different areas and the potential direct execution of commands from unsafe areas during gesture control, improving the security of in-vehicle interaction, the accuracy of control behavior, and the manageability of vehicle execution logic.

[0087] Existing in-vehicle gesture control solutions all rely on dedicated automotive hardware, which is costly, complex to install, and only suitable for mid-to-high-end new cars. This system requires no additional dedicated hardware; it directly reuses the gyroscope and accelerometer sensor built into the phone. By collecting angular velocity and acceleration data from the sensors, and combining this with a companion app to complete gesture parsing, it eliminates the need for vehicle modifications and the purchase of additional equipment, significantly reducing user costs. It is also compatible with lower-end models, addressing the core pain points of existing solutions: high cost and poor compatibility. Furthermore, the phone's sensors require no additional debugging; gesture acquisition can be achieved simply through the app, making it convenient and highly compatible.

[0088] This solution uses handheld mobile phone gesture recognition, allowing drivers to make preset gestures simply by holding their phones. There is no need to touch the vehicle's infotainment system or look down to find the operating area, and drivers can keep their eyes on the road ahead at all times, greatly improving driving safety. At the same time, gesture recognition relies on the phone's sensors to collect physical data, so it is not affected by light or ambient noise, has high recognition stability, and is flexible in its operating range, adapting to different driving postures.

[0089] Example 2

[0090] Based on the above embodiment 1, on the vehicle side, the vehicle control permissions need to be further adjusted according to the vehicle's driving status. The following is a specific embodiment.

[0091] Driving status refers to state information that characterizes the current motion or stationary state of a vehicle. It can be determined by the vehicle's infotainment system based on vehicle bus data, vehicle controller status, gear position, vehicle speed signal, braking status, engine operating status, or electronic parking brake status. Alternatively, it can be determined by combining data from onboard sensors, inertial measurement units, wheel speed sensors, GPS positioning information, and status messages from the vehicle control system. In practical applications, a vehicle can be defined as parking when its speed is zero, the gear is in parking gear, and the electronic parking brake is activated. A driving status is defined when the vehicle speed is consistently greater than zero. The relevant thresholds can be set according to the vehicle model configuration and control strategy; this application does not limit this.

[0092] After acquiring the vehicle's driving status, the vehicle-mounted system dynamically adjusts the vehicle control permissions based on a pre-established mapping between status and permissions. Vehicle control permissions can include preset normal vehicle control permissions, partial vehicle control permissions, or restricted vehicle control permissions. Normal vehicle control permissions allow the execution of all preset vehicle control functions, partial vehicle control permissions allow only low-risk vehicle control functions, and restricted vehicle control permissions prohibit or restrict the execution of high-risk control functions. When the vehicle is parked, the system can adjust the vehicle control permissions to a more lenient range to support occupants performing more control operations when the vehicle is stationary. When the vehicle is in motion, the system can adjust the vehicle control permissions to a restricted range to reduce the risk of misoperation caused by control commands issued by non-driving terminals. This permission adjustment can also be completed by the vehicle-mounted system calling a strategy engine. The strategy engine updates the permission configuration based on the real-time driving status and outputs a new permission determination result. In practical applications, this strategy engine can also choose other implementation forms, which are not limited in this embodiment.

[0093] In some embodiments, the vehicle control permissions of the mobile terminal in the non-driving area are preset for both parking and driving, and the vehicle control permissions when parking are more extensive than those when driving. When the driving status indicates that the vehicle is parked, the preset vehicle control permissions of the mobile terminal in the non-driving area are activated. For example, while the vehicle is in motion, the non-driving area can only adjust the seat, air conditioning, and window height for its own position. When the vehicle is parked, the preset gesture command vehicle control permissions (vehicle system permissions, vehicle engine permissions, etc.) of the non-driving area are released.

[0094] Furthermore, permissions can be further conditional based on vehicle speed. For example, when the vehicle speed is greater than 100 km / h, the window height adjustment permission in the non-driving area can be turned off.

[0095] During operation, the vehicle first acquires its current driving status, then maps this status to the corresponding vehicle control permission level. When subsequently receiving gesture control commands from the mobile terminal, the updated permission result is used as the basis for execution, thus ensuring that permission control remains consistent with the vehicle's operating status. By introducing different vehicle states, such as stationary, low-speed movement, or high-speed driving, into permission management, the vehicle can dynamically tighten or loosen control boundaries in different operating scenarios, avoiding the security risks associated with fixed permission modes.

[0096] After adopting the above method, the vehicle control permission is no longer limited only by the terminal location area, but is updated in conjunction with the vehicle's operating status. Therefore, it can improve the usability of in-vehicle interaction when the vehicle is stationary, and enhance the strictness of control constraints when the vehicle is moving, thereby improving the safety, stability and controllability of gesture control of the vehicle.

[0097] Example 3

[0098] Based on the aforementioned embodiments, on the vehicle side, if the gesture control command is not within the vehicle control permissions, a permission insufficient message is sent to the mobile terminal.

[0099] The insufficient permissions message is used to inform the mobile terminal that the current control request has not been accepted, so that the user can know the reason why the command was blocked in a timely manner.

[0100] By adopting this implementation method, the vehicle can promptly report the reason to the mobile terminal while rejecting unauthorized commands, reducing invalid communication and duplicate requests, and improving the efficiency of human-vehicle interaction. At the same time, the insufficient permissions message can provide a clear prompt to the user, reducing potential risks caused by misoperation and enhancing the security and stability of the gesture-based vehicle control process.

[0101] Example 4

[0102] Based on the foregoing embodiments, the following embodiment describes the handling measures when commands issued by two mobile terminals conflict.

[0103] Figure 2 This application provides a flowchart illustrating a method for vehicle control based on gesture recognition in collaboration between the vehicle and a mobile terminal. Figure 2 ,like Figure 2 As shown, it includes the following steps:

[0104] S201: Obtain the number of mobile terminals.

[0105] In this embodiment, the number of mobile terminals refers to the number of terminals currently participating in vehicle gesture control interaction. In specific implementation, the vehicle controller first counts the number of mobile terminals in the current valid session and writes the result to the vehicle control management module.

[0106] S202: If the number is greater than 1, determine the control priority of each mobile terminal based on the location area of ​​each mobile terminal.

[0107] Control priority refers to the level parameters used to determine the order of instruction execution and the weight of effectiveness when multiple terminals issue control requests simultaneously. The level parameters can be pre-set to correspond to the driving area, non-driving area, or more subdivided cabin areas.

[0108] When a number greater than 1 is detected, the vehicle control management module reads the location area information corresponding to each mobile terminal and assigns control priorities to each terminal according to a preset mapping relationship. Terminals in the driving area have higher priorities, while those in non-driving areas have relatively lower priorities. The priority mapping relationship can be stored in the vehicle's local permission table and updated in real time when the terminal location changes or the occupant switches. In another implementation, priorities corresponding to different terminal device identifiers are pre-set, and priority tags are carried during the transmission of gesture control commands. Alternatively, the vehicle controller can query the corresponding priority based on the terminal device identifier.

[0109] If the quantity is equal to 1, it can be executed after the vehicle control permission verification.

[0110] S203: When a conflict is detected between the gesture control commands of two mobile terminals, the gesture control command with higher control priority will be selected as the command to be executed.

[0111] When two mobile terminals send gesture control commands targeting the same object within the same time window, the system identifies the conflict based on the consistency of the controlled object, the opposite direction of the command, or the conflict of parameters, and compares the control priorities of the corresponding terminals. The gesture control command with higher priority is sent to the vehicle's execution unit and the control output is completed, while the command with lower priority is suppressed, discarded, or recorded as an invalid request, thus avoiding execution chaos caused by concurrent control from multiple terminals. Conflict identification and priority determination can be completed by the onboard processor within a millisecond-level time window to ensure real-time command response.

[0112] In this way, when multiple mobile terminals are cooperating to control the vehicle, the number of terminals can be identified first, and then a priority relationship can be established based on the location area. In the event of a conflict of commands, only the high-priority command will be executed, making the allocation of control rights clearer and the execution results more stable. At the same time, it reduces the risk of miscontrol caused by concurrent operation of multiple terminals and improves the safety and controllability of in-vehicle gesture control.

[0113] Users inside the vehicle can terminate the gesture control mode in three ways: first, by making a preset termination gesture (such as quickly shaking the phone twice); second, by manually turning off the "gesture control" button in the app; and third, if the communication between the phone and the vehicle is interrupted for more than 10 seconds, the system will automatically terminate the gesture control mode to avoid wasting resources when there is no operation.

[0114] If the communication link between the mobile phone and the vehicle system is interrupted (such as USB disconnection or Wi-Fi signal loss), the system will immediately alert the driver through mobile phone vibration and notification, and terminate the gesture control mode. The vehicle system will remain in its current state until communication is restored, at which point the gesture control mode can be restarted and the state synchronized.

[0115] Figure 3Schematic diagram of the structure of the gesture recognition vehicle control device based on vehicle and mobile terminal collaboration provided in this application Figure 1 ,like Figure 3 As shown, the vehicle control device 30 based on gesture recognition in collaboration between the vehicle and a mobile terminal provided in this embodiment includes:

[0116] The location acquisition module 301 is used to acquire the location area of ​​a mobile terminal with a gesture control application installed in the vehicle. The location area is either the driving area or the non-driving area.

[0117] The permission determination module 302 is used to determine the vehicle control permission of the mobile terminal based on the location area;

[0118] The instruction receiving module 303 is used to receive gesture control instructions sent by the mobile terminal; wherein, the gesture control instructions are generated based on the mobile terminal capturing user gesture operations;

[0119] The execution module 304 is used to execute the gesture control command if the gesture control command is within the vehicle control authority.

[0120] Furthermore, the device also includes a permission adjustment module for:

[0121] Obtain the vehicle's driving status;

[0122] Adjust vehicle control permissions based on driving status.

[0123] Furthermore, the permission adjustment module is specifically used for:

[0124] When the driving status indicates that the vehicle is parked, the preset vehicle control permissions of the mobile terminal in the non-driving area are enabled.

[0125] Furthermore, the device also includes a sending module for sending a message indicating insufficient permissions to the mobile terminal if the gesture control command is not within the vehicle control permissions.

[0126] Furthermore, the device also includes a priority processing module for:

[0127] The number of mobile terminals was obtained;

[0128] If the number is greater than 1, determine the control priority of each mobile terminal based on the location area of ​​each mobile terminal;

[0129] When a conflict is detected between the gesture control commands of two mobile terminals, the gesture control command with higher control priority will be selected as the command to be executed.

[0130] The device provided in this embodiment can execute the method provided on the vehicle side in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0131] Figure 4 Schematic diagram of the structure of the gesture recognition vehicle control device based on vehicle and mobile terminal collaboration provided in this application Figure 2 ,like Figure 4 As shown, the vehicle control device 40 based on gesture recognition in collaboration between the vehicle and a mobile terminal provided in this embodiment includes:

[0132] The gesture acquisition module 401 is used to respond to the user's operation of waking up the gesture recognition function and to acquire gesture data collected by the mobile phone sensors based on the user's gesture operation.

[0133] The instruction determination module 402 is used to determine the gesture control instruction corresponding to the gesture data according to the preset gesture-instruction correspondence.

[0134] The sending module 403 is used to send gesture control commands to the vehicle.

[0135] The device provided in this embodiment can execute the method provided on the vehicle side in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0136] Figure 5 This is a schematic diagram of the electronic device provided in this application. The electronic device can be a vehicle controller or a terminal device. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0137] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0138] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0139] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0140] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0141] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0142] This application also provides a vehicle, including a vehicle body, a display device, a communication device, and a controller, wherein the controller is used in the method of the above method embodiments.

[0143] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method. This computer program product can be an application program built into a terminal device or an application program built into an in-vehicle infotainment system.

[0144] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0145] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0146] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0147] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0148] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0150] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0152] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for vehicle control based on gesture recognition in collaboration between a vehicle and a mobile terminal, characterized in that, Applied to vehicles, the method includes: The location area of ​​a mobile terminal with a gesture control application installed in the vehicle is obtained, wherein the location area is either the driving area or the non-driving area; Based on the location area, determine the vehicle control permissions of the mobile terminal; Receives a gesture control command sent by the mobile terminal; wherein the gesture control command is generated based on the user's gesture operation captured by the mobile terminal; If the gesture control command is within the vehicle control permissions, then the gesture control command is executed.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the vehicle's driving status; Adjust the vehicle control permissions based on the driving status.

3. The method according to claim 2, characterized in that, The step of adjusting the vehicle control permissions based on the driving status includes: When the driving status indicates that the vehicle is parked, the preset vehicle control permissions of the mobile terminal in the non-driving area are enabled.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: If the gesture control command is not within the vehicle control permissions, an insufficient permissions message is sent to the mobile terminal.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: Obtain the number of the mobile terminals; If the number is greater than 1, the control priority of each mobile terminal is determined according to the location area of ​​each mobile terminal; When a conflict is detected between the gesture control commands of two mobile terminals, the gesture control command with higher control priority will be selected as the command to be executed.

6. A method for vehicle control based on gesture recognition in collaboration between a vehicle and a mobile terminal, characterized in that, Applied to a mobile terminal, the method includes: In response to the user's gesture recognition function, the system acquires gesture data collected by the phone's sensors based on the user's gesture operation. Based on the preset gesture-command correspondence, determine the gesture control command corresponding to the gesture data; Send the gesture control command to the vehicle.

7. A device for vehicle control based on gesture recognition in collaboration between a vehicle and a mobile terminal, characterized in that, The device includes: The location acquisition module is used to acquire the location area of ​​a mobile terminal installed in the vehicle with a gesture control application, wherein the location area is either the driving area or the non-driving area. The permission determination module is used to determine the vehicle control permission of the mobile terminal based on the location area; The instruction receiving module is used to receive the gesture control instructions sent by the mobile terminal; wherein the gesture control instructions are generated based on the user's gesture operations captured by the mobile terminal. An execution module is used to execute the gesture control command if the gesture control command is within the vehicle control authority.

8. A device for vehicle control based on gesture recognition in collaboration between a vehicle and a mobile terminal, characterized in that, The device includes: The gesture acquisition module is used to respond to the user's operation of waking up the gesture recognition function and to acquire gesture data collected by the mobile phone sensors based on the user's gesture operation; The instruction determination module is used to determine the gesture control instruction corresponding to the gesture data based on a preset gesture-instruction correspondence. The sending module is used to send the gesture control command to the vehicle.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-5, or to perform the method as described in claim 6.

10. A vehicle, characterized in that, It includes a vehicle body, a display device, a communication device, and a controller, wherein the controller is used in the method as described in any one of claims 1-5.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5, or to perform the method as described in claim 6.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-5, or the method of claim 6.

Images