Gesture control method and system for vehicle-mounted screen projection operation
By connecting portable mobile devices with the vehicle's infotainment system and projecting the screen, and using cameras and convolutional neural networks to recognize gesture images, vehicle control without touching the screen is achieved. This solves the problem of distraction in traditional operations and improves the safety and convenience of vehicle control.
Patent Information
- Application Number
- CN202410976770.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-21
- Publication Date
- 2024-11-19
AI Technical Summary
Traditional in-vehicle physical buttons or touchscreens can distract drivers and pose safety hazards while driving. How can we achieve vehicle control without touching the screen to improve safety and convenience?
By connecting and projecting the screen between a portable mobile device and the vehicle's infotainment system, the system uses a camera to detect the user's gestures, combines this with a convolutional neural network to recognize the gestures, obtain the corresponding control commands, and execute the operation under a security verification mechanism.
It enables vehicle control without touching the screen, enhancing the fun, convenience, and safety of vehicle control, and improving the interactive experience and safety during driving.
Smart Images

Figure CN118981273A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of human-computer interaction technology, and more specifically, to a gesture control method and system for in-vehicle screen projection operations. Background Art
[0002] With the development of smart car technology, in-vehicle entertainment systems and navigation functions are becoming increasingly rich, but traditional physical buttons or touch screen operations may distract the driver during driving and pose a safety hazard.
[0003] Gesture control detects the user's gesture image through a camera, analyzes gesture changes through multiple frames of images, and then calls the corresponding control instructions to obtain the corresponding function of the gesture. Gesture control is simple to operate, and the operator can perform human-computer interaction without directly touching the screen, without distracting the operator too much.
[0004] Therefore, how to apply gesture control to vehicle control is a direction worthy of research. Summary of the invention
[0005] In view of the technical problems existing in the prior art, the present invention provides a gesture control method and system for in-vehicle screen projection operation, so that users can control the vehicle without touching the screen, thereby improving the fun, convenience and safety of vehicle control.
[0006] According to a first aspect of the present invention, a gesture control method for an in-vehicle screen projection operation is provided, comprising:
[0007] Turn on the screen projection mode between the mobile device and the car machine;
[0008] Acquire a gesture image, and extract corresponding gesture control operation instructions according to the gesture image;
[0009] The current state information of the vehicle is obtained, the safety of executing the gesture control operation instruction under the current state of the vehicle is determined according to the safety verification mechanism, and the gesture control operation instruction is executed according to the determination result.
[0010] Based on the above technical solution, the present invention can also make the following improvements.
[0011] Optionally, the step of enabling the screen projection mode of interconnecting the mobile device and the vehicle includes:
[0012] Establish a communication connection between the mobile device and the vehicle computer, and enable the interconnected screen projection mode on the corresponding applications on the mobile device and the vehicle computer.
[0013] Optionally, acquiring the gesture image includes:
[0014] A camera on a mobile device is used to capture multiple frames of images containing user gestures.
[0015] Optionally, extracting a corresponding gesture control operation instruction according to the gesture image includes:
[0016] Extracting features from the gesture image to obtain a gesture feature map;
[0017] Identify the user's gesture change information based on the continuous multi-frame gesture feature graph;
[0018] A gesture control instruction library is searched according to the gesture change information to obtain a gesture control operation instruction that matches the current gesture change.
[0019] Optionally, the extracting features from the gesture image to obtain a gesture feature map includes:
[0020] Inputting the gesture image into a trained convolutional neural network (CNN) model;
[0021] Extracting hand features in the gesture image through a convolutional layer;
[0022] Increase the nonlinearity of the feature map through the activation function ReLU;
[0023] Reducing the spatial dimension of the feature map through a pooling layer to extract key features of the hand features;
[0024] Integrate the key features through a fully connected layer;
[0025] The gesture feature map is output through the output layer based on the integrated key features.
[0026] Optionally, the acquiring the vehicle current state information, judging the safety of executing the gesture control operation instruction under the vehicle current state according to a safety verification mechanism, and executing the gesture control operation instruction according to the judgment result, includes:
[0027] Calculate the real-time driving speed of the vehicle according to the current wheel speed information of the vehicle, determine the current driving mode of the vehicle according to the current gear signal of the vehicle, and obtain the current fault signal of the vehicle;
[0028] According to the real-time driving speed, current driving mode and current fault signal, the safety of executing the gesture control operation instruction in the current state of the vehicle is comprehensively arbitrated in combination with a preset safety verification mechanism:
[0029] If it is determined to be safe, the corresponding gesture control operation instruction is executed;
[0030] If it is determined to be unsafe, the corresponding gesture control operation instruction will not be executed and a warning will be issued.
[0031] According to a second aspect of the present invention, a gesture control system for in-vehicle screen projection operation is provided, comprising:
[0032] A mobile device, used to enable a screen projection mode of interconnection between the mobile device and the vehicle machine on the mobile device side, and to obtain a gesture image containing a gesture signal, and to extract a corresponding gesture control operation instruction according to the gesture image;
[0033] The vehicle computer is used to enable the interconnection and projection mode between the mobile device and the vehicle computer on the vehicle computer side, and to obtain the current status information of the vehicle, and to judge the safety of executing the gesture control operation instructions under the current status of the vehicle according to the safety verification mechanism, and to execute the gesture control operation instructions according to the judgment result.
[0034] According to a third aspect of the present invention, there is provided an electronic device comprising a memory and a processor, wherein the processor is used to implement the steps of the gesture control method for the above-mentioned in-vehicle screen projection operation when executing a computer management program stored in the memory.
[0035] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer management program is stored. When the computer management program is executed by a processor, the steps of the gesture control method for the above-mentioned in-vehicle screen projection operation are implemented.
[0036] The present invention provides a gesture control method, system, electronic device and storage medium for in-vehicle screen projection operation. Through the interconnection and screen projection between a portable mobile device and a vehicle computer, the image acquisition and image processing functions of the mobile device are fully utilized without increasing the equipment cost, so that the user can control the vehicle without touching the screen, thereby improving the fun, convenience and safety of vehicle control. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 A flow chart of a gesture control method for in-vehicle screen projection operation provided by an embodiment of the present invention;
[0038] Figure 2 A schematic diagram of an implementation scenario of a gesture control method for an in-vehicle screen projection operation provided by an embodiment of the present invention;
[0039] Figure 3 A schematic diagram of a gesture control system for in-vehicle screen projection operation provided by an embodiment of the present invention;
[0040] Figure 4 A schematic diagram of gesture actions provided by an embodiment of the present invention;
[0041] Figure 5 A structural diagram of the convolutional neural network (CNN) model provided for a specific implementation scenario;
[0042] Figure 6A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;
[0043] Figure 7 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention.
[0044] In the accompanying drawings, the components represented by the reference numerals are listed as follows:
[0045] 1. Mobile phone, 1a. Mobile phone interconnection APP, 101. Camera, 102. Gesture recognition module 102, 1021. Image processing module, 1022. Convolutional neural network CNN model, 103. Mobile terminal signal control module, 104. Mobile terminal communication module, 2. Car computer, 2a. Car computer screen projection program, 201. Driving sensing module, 202. Safety control module, 203. Car-side signal control module, 204. Car-side communication module, 3. User, 301. Gesture, 3a~3l. Gesture action, 4. Communication network, 5. Vehicle speed sensor. DETAILED DESCRIPTION
[0046] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0047] Figure 1 A flow chart of a gesture control method for in-vehicle screen projection operation provided by the present invention, such as Figure 1 As shown, the method includes steps S1 to S3:
[0048] S1, turn on the screen projection mode between the mobile device and the car machine;
[0049] S2, acquiring a gesture image, and extracting a corresponding gesture control operation instruction according to the gesture image;
[0050] S3, obtaining the current state information of the vehicle, judging the safety of executing the gesture control operation instruction under the current state of the vehicle according to the safety verification mechanism, and executing the gesture control operation instruction according to the judgment result.
[0051] like Figure 2The diagram is a schematic diagram of an implementation scenario applicable to the method of this embodiment. The mobile device can be a portable device such as a mobile phone 1, a tablet computer, etc. (the mobile phone 1 is used as an example in the following text), which is equipped with a camera 101, which has the function of taking images, image processing capabilities, and wired or wireless communication functions to achieve interconnection with the vehicle machine 2. The mobile phone 1 is equipped with a mobile phone interconnection APP 1a, and the vehicle machine 2 is pre-downloaded with a vehicle machine screen projection program 2a. When the mobile device is interconnected with the vehicle machine 2, the connection between the mobile phone interconnection APP 1a and the vehicle machine screen projection program 2a can be used to open the mobile device and the vehicle machine interconnection screen projection mode, thereby realizing the method of this embodiment.
[0052] It is understandable that based on the defects in the background technology, the embodiment of the present invention proposes a gesture control method for in-vehicle screen projection operation. This method fully utilizes the image acquisition and image processing functions of the mobile device through the interconnection between the portable mobile device and the vehicle machine 2 without adding additional equipment costs, so that the user 3 can control the vehicle without touching the screen, thereby improving the fun, convenience and safety of vehicle control.
[0053] Based on the above technical solution, this embodiment can also be improved as follows.
[0054] like Figure 3 Shown is the corresponding Figure 2 A more detailed system block diagram of the implementation scenario shown. Figure 3 It can be seen that the mobile device (taking mobile phone 1 as an example) is pre-downloaded with a mobile phone interconnection APP 1a, and the mobile device includes a front camera 101, a gesture recognition module 102, a mobile phone 1 signal control module and a mobile terminal communication module, wherein the gesture recognition module 102 includes an image processing module 1021 and a convolutional neural network CNN model 1022 which are arranged in sequence.
[0055] like Figure 4 The figure shows the gesture action diagram of user 3, based on Figure 4 The gestures 3a-3l shown can express a variety of control instructions. The mobile phone 1 captures the gesture 301 issued by the user 3 through image processing, and then recognizes the corresponding control instruction to achieve the in-vehicle screen projection operation without touching the screen. Figure 4 The figure shows a variety of gestures. The gesture 301 is an artificial three-dimensional hand body movement, which combines natural gesture language and specially designed command actions to achieve contactless control of the interaction of the car screen projection program 2a through gestures, such as Figure 4The gesture 301 includes gesture 3a (pause), gesture 3b (continue), gesture 3c (confirm), gesture 3d (volume +), gesture 3e (volume -), gesture 3f (forward), gesture 3g (backward), gesture 3h (zoom out), gesture 3i (zoom in), gesture 3j (next song), gesture 3k (next song), gesture 3l (loop play), etc. It not only improves driving safety and convenient operation, but also optimizes user 3 experience and adapts to diverse needs. Gesture 301 needs to have the characteristics of high recognition, safety, versatility, friendliness, adaptability, and privacy protection.
[0056] Understandably, Figure 2 and Figure 3 The mobile phone 1 shown is not only a sensor center for gesture 301 recognition, but also a transmitter of gesture control operation instructions. Specifically, the mobile phone 1 captures the gesture action images of the user 3 (such as the driver) through its high-resolution front camera 101, and these gesture action images are transmitted in real time to the dedicated application (such as the aforementioned mobile phone interconnection APP 1a) inside the mobile phone 1 for processing. Therefore, the mobile phone 1 not only needs to have an application that uses advanced artificial intelligence algorithms to recognize and analyze the gesture 301, and then converts the recognition results into specific control instructions, but also needs to have a high-performance camera 101, a powerful processor, stable wireless communication capabilities, good battery life, and a compatible operating system. Android mobile phones 1 are recommended first, such as Samsung Galaxy S23Ultra series or Huawei Mate60-Pro, Xiaomi 14Ultra, etc., and Apple mobile phones 1 are second, such as iPhone 15Pro Max, etc.
[0057] The mobile phone interconnection APP 1a is mainly responsible for wireless or wired interconnection with the car screen projection program 2a of the car machine 2 to realize multiple functions such as infotainment, navigation, and communication. Therefore, it needs to have functions such as navigation, media playback, phone and message management, vehicle information display, voice control, and third-party application expansion, and has the characteristics of high security, easy operation, personalized customization, real-time, and wide compatibility. Apple CarPlay, Baidu CarLife, and Android Auto are recommended first, and CarbitLink and MirrorLink are second.
[0058] The car screen projection program 2a is a smart interconnection program in the current smart cockpit car 2, which mainly provides users 3 with car interconnection, Internet access, entertainment, multimedia, navigation screen projection and other purposes. As a preferred option, it not only needs to have high compatibility, wireless connection, high-definition mirroring, audio synchronization, two-way control, touch mapping, smart adaptation, low-latency transmission and other functions, but also needs to meet the characteristics of safe driving, high entertainment experience, convenient operation, strong compatibility, real-time interaction, etc. The car screen projection program 2a recommends Apple's AriPlay and Android's Miracast first, and Kabit's code scanning connection (Wlan direct connection) and Baidu CarLife screen projection are second.
[0059] Since the car computer 2 is pre-downloaded with a car computer screen projection program 2a, under the premise that the mobile device and the car computer 2 are connected to the network, the car computer screen projection program 2a can be linked with the mobile phone interconnection APP 1a on the mobile phone 1.
[0060] based on Figure 3 The functional modules shown in the figure are composed of: in step S1, the screen projection mode of interconnecting the mobile device and the vehicle is turned on, including:
[0061] Establish a communication connection between the mobile device and the vehicle computer 2, and enable the interconnected screen projection mode on the corresponding applications on the mobile device and the vehicle computer 2.
[0062] In step S2, obtaining the gesture image includes:
[0063] The camera 101 mounted on the mobile device captures multiple frames of images including the gesture 301 of the user 3 .
[0064] It is understandable that the main function of the front camera 101 equipped in the mobile phone 1 is to capture images containing the hand movements of the user 3. Therefore, it needs to have high-definition shooting capabilities, wide-angle lenses, autofocus and HDR functions, night scene mode, AI beauty and special effects, high-speed shooting capabilities, depth perception technology, etc. Mobile phones 1 in the high-end market are recommended, such as the Samsung Galaxy S23Ultra series or Huawei Mate60-Pro, Xiaomi 14Ultra, iPhone 15Pro Max, etc., and mid-range or affordable Xiaomi 12, OPPO Reno8 Pro+, Redmi Note 11TPro, etc. are secondly recommended.
[0065] In one possible implementation manner, in step S2, extracting a corresponding gesture control operation instruction according to the gesture image includes:
[0066] Extracting features from the gesture image to obtain a gesture feature map;
[0067] Identifying gesture change information of user 3 according to the continuous multi-frame gesture feature graph;
[0068] A gesture control instruction library is searched according to the gesture change information to obtain a gesture control operation instruction that matches the current gesture change.
[0069] like Figure 3 As shown, the gesture image obtained by the camera 101 is input into the gesture recognition module 102 for gesture 301 recognition. The gesture recognition module 102 is mainly used to capture the gesture action of the user 3 from the gesture image, and use computer vision and machine learning algorithms to recognize the type and intention of the gesture 301. It mainly includes an image processing module 1021 and a convolutional neural network CNN model 1022 which are arranged in sequence. It needs to have the characteristics of high-precision recognition, low power consumption, high compatibility, high friendliness, high security, etc.
[0070] The image processing module 1021 is one of the core components of the method of the present invention, and is mainly used for real-time preprocessing of gesture images to provide high-quality image data for gesture 301 recognition of the subsequent convolutional neural network CNN model 1022. The image processing module 1021 needs to have the characteristics of low latency, high precision, low power consumption, high robustness, and high compatibility. The OpenCV library is recommended first, and MediaPipe or TensorFlow Lite for Mobile are second.
[0071] Furthermore, based on Figure 5 The convolutional neural network CNN model 1022 shown in the figure, extracting features from the gesture image to obtain a gesture feature map, includes:
[0072] Input the gesture image into the trained convolutional neural network CNN model 1022;
[0073] Extracting hand features in the gesture image through a convolutional layer;
[0074] Increase the nonlinearity of the feature map through the activation function ReLU;
[0075] Reducing the spatial dimension of the feature map through a pooling layer to extract key features of the hand features;
[0076] After n (n≥1) rounds of convolution layer-activation function-pooling layer operations, the key features are integrated through a fully connected layer;
[0077] The gesture feature map is output through the output layer based on the integrated key features.
[0078] It is understandable that the convolutional neural network CNN model 1022 is a deep learning algorithm module designed for image and video recognition tasks, and is particularly suitable for detecting and recognizing gestures 301 in the gesture recognition module 102. By connecting to the mobile terminal communication module 104, the model is trained by using techniques such as feature extraction, spatial invariance, multi-level abstraction, classification and recognition. Figure 5 As shown, after the gesture image is input into the convolutional neural network CNN model 1022, it passes through the convolution layer C1, applies multiple filters to the hand features in the gesture image, passes through the activation function ReLU to increase nonlinearity, and the pooling layer Pooling reduces the spatial dimension to extract key features. According to actual needs, the process of convolution layer, activation function, and pooling layer is repeated to build a deeper network structure. In order to simplify the description, Figure 5 Only the second convolution layer C2, the second activation function ReLU and the second pooling layer Pooling are shown. The actual network model may contain more such layers. Finally, all the extracted key features are integrated through the fully connected layer F1 to perform the final classification or regression prediction. The output layer gives the prediction results of the network model, such as the classification label of the image. As a preference, the convolutional neural network CNN model 1022 needs to have the characteristics of efficient learning, high accuracy, strong generalization ability, and real-time performance. TensorFlow or PyTorch is recommended first, and TensorFlow Lite / PyTorch Mobile is second.
[0079] After obtaining the extracted gesture feature map, the corresponding gesture control operation instructions can be extracted according to the gesture action category in the gesture feature map. This part of the work is completed in the mobile terminal control signal module.
[0080] The mobile terminal control signal module is mainly responsible for calling the corresponding gesture control operation instructions according to the gesture action, and converting the gesture control operation instructions into control signals that can be understood and executed by the vehicle system. It has the functions of command reception and parsing, protocol adaptation, command management, security response, status feedback, etc. As a preferred option, it needs to have the characteristics of high efficiency, high security, high compatibility, high flexibility, high stability, etc. An embedded microcontroller unit (MCU) is recommended, and a secure microprocessor or a standardized communication protocol stack is recommended as a second choice.
[0081] The mobile phone 1 communicates and interconnects with the vehicle computer 2 through the mobile terminal communication module 104 , thereby transmitting the control signal converted according to the gesture control operation instruction to the vehicle computer 2 .
[0082] The mobile terminal communication module 104 is mainly responsible for the wireless communication between the mobile phone 1 and the vehicle computer 2. It is a wireless communication module of the mobile phone 1 and wirelessly communicates with the vehicle computer 2 through the communication network 4. As a preferred choice, it needs to meet the characteristics of high compatibility, high efficiency, easy integration, stability, and friendliness. Qualcomm QCA4020 / 4024 or BroadcomBCM43907 is recommended first, and Nordic nRF52840 is second.
[0083] The communication network 4 is responsible for providing network support for the mobile phone 1, and has the characteristics of high-speed transmission, low latency, stability and redundancy, flexibility and scalability, standardization and compatibility, fine management, and security. Ethernet is preferred, and CAN FD or time-sensitive network (TSN) is secondarily preferred.
[0084] Figure 3 The functional module composition of the vehicle computer 2 is also shown. The vehicle computer 2 includes a driving sensing module 201, a safety control module 202, a vehicle-side signal control module 203 and a vehicle-side communication module 204. The vehicle-side communication module 204 of the vehicle computer 2 is interconnected with the mobile terminal communication module 104 of the mobile phone 1 to realize signal transmission between the mobile phone 1 and the vehicle computer 2.
[0085] Specifically, the vehicle-side communication module 204 is mainly responsible for the wireless communication between the mobile phone 1 and the vehicle computer 2. It is the wireless communication module of the vehicle computer 2, and has multi-mode wireless connection, network security, remote update, integrated vehicle network interface, location service and navigation and other functions. As a preferred choice, it needs to have the characteristics of low power consumption, automotive-grade certification, low power consumption, long-term support and upgrade, easy integration, high-speed data transmission, etc. NXP Semiconductors S32G Vehicle Network Processor is recommended first, and Qualcomm Snapdragon Automotive 4G / 5G Platforms or Texas Instruments Jacinto Automotive Processors are second.
[0086] The driving sensing module 201 is one of the core components used to obtain the vehicle status in the method of this embodiment. It is mainly responsible for real-time monitoring of the vehicle's driving speed and driving mode. It captures wheel speed information through integrated sensors (such as vehicle speed sensor 5) and converts it into driving speed data. By obtaining signals such as the current transmission gear position of the vehicle, it is determined whether the vehicle is currently in economy mode, sports mode or other specific driving mode. As a preference, the driving sensing module 201 needs to have the characteristics of high precision, fast response, durability, stability, compatibility, standardization, low power consumption, safety, etc. Bosch Automotive Speed Sensor Modules are recommended first, and Continental MK C1 One-Box Brake System or Delphi Technologies DS150 Wheel Speed Sensor are second.
[0087] The vehicle speed sensor 5 mainly monitors the vehicle speed by real-time monitoring of the wheel speed, and converts this information into an electrical signal and transmits it to the driving sensing module 201, so as to provide data support for real-time acquisition of the vehicle speed. Preferably, the vehicle speed sensor 5 needs to have the characteristics of high precision, durability, fast response, high compatibility, easy installation and maintenance, etc. Bosch ABS wheel speed sensor is preferred, followed by Continental VDO wheel speed sensor or Denso speed sensor.
[0088] The safety control module 202 is mainly responsible for processing and managing key signals for communication inside the vehicle and with the outside world. It ensures the security, reliability and compliance of all signal transmissions according to the preset security verification mechanism and its corresponding algorithm. It has functions such as signal conversion and routing, security policy execution, encryption and decryption, fault diagnosis and reporting, security isolation and protection, and OTA update management. As a preferred option, the safety control module 202 needs to have the advantages of high reliability, low latency, high throughput, network security protection, configurability, scalability, high integration, and compliance with vehicle standards. Infineon AURIXTC3xx / TC4xx series microcontrollers are recommended first, and NXP S32G Vehicle Network Processor is the second choice.
[0089] The vehicle-side signal control module 203 plays the role of a bridge in the embodiment of the present invention. It converts the safety-related signals (such as vehicle speed, driving mode, emergency brake signal, etc.) generated by the safety control module 202 into instructions that can be recognized and executed by the vehicle screen projection program 2a. It has functions such as signal translation and adaptation, interactive instruction generation, priority management, interface development, and safety policy execution. As a preference, the vehicle-side signal control module 203 needs to have the characteristics of high reliability, low latency response, high compatibility, programmable configurability, and safety. Vector Informatik customized development module is recommended first, followed by Elektrobit (EB) or TTTech Auto.
[0090] based on Figure 3 The functional module composition structure of the vehicle computer 2 shown in step S3, the current state information of the vehicle is obtained, the safety of executing the gesture control operation instruction in the current state of the vehicle is judged according to the safety verification mechanism, and the gesture control operation instruction is executed according to the judgment result, including:
[0091] Calculate the real-time driving speed of the vehicle according to the current wheel speed information collected by the vehicle speed sensor 5, determine the current driving mode of the vehicle according to the current gear signal of the vehicle, and obtain the current fault signal of the vehicle (for example, whether emergency braking occurs);
[0092] According to the real-time driving speed, current driving mode and current fault signal, the safety of executing the gesture control operation instruction in the current state of the vehicle is comprehensively arbitrated in combination with a preset safety verification mechanism:
[0093] If it is determined to be safe, the corresponding gesture control operation instruction is executed, and the corresponding function of the vehicle is controlled according to the control signal parsed from the gesture control operation instruction;
[0094] If it is determined to be unsafe, the corresponding gesture control operation instruction will not be executed and a warning will be issued.
[0095] The present invention is now described with reference to several specific implementation scenarios.
[0096] Embodiment 1:
[0097] First, user 3 needs to interconnect mobile phone 1 and vehicle computer 2, that is, mobile phone 1 and vehicle computer 2 perform wired or wireless screen projection, such as the mobile phone interconnection APP 1a and vehicle computer screen projection program 2a mentioned in the previous embodiment to open the mobile device and vehicle computer interconnection screen projection mode, and project the screen through wireless interconnection;
[0098] User 3 uses the Internet screen projection application to open the music player in the application and play Bluetooth music / local music / music software in the application to play multimedia;
[0099] User 3 needs to switch to the previous song / next song / pause / mute / volume+ / volume-, etc. At this time, he only needs to make a corresponding gesture 301 to the mobile phone 1. For example, if he needs to pause, he can make gesture 3a (pause), gesture 3d (volume+), gesture 3e (volume-), gesture 3f (forward), gesture 3g (backward), etc.
[0100] At the same time, the front camera 101 in the mobile phone 1 captures the gesture action 3a (pause) of the user 3, transmits the acquired gesture image to the image processing module 1021 for preprocessing, and then inputs it into the convolutional neural network CNN model 1022. After the classification and recognition functions of the convolutional neural network CNN model 1022, the corresponding gesture feature map is obtained;
[0101] The image processing module 1021 processes the identified continuous frame gesture feature graph to obtain gesture action category data, transmits the gesture control operation instruction corresponding to the gesture action category data to the mobile phone interconnection APP 1a, and sends the gesture control operation instruction to the mobile terminal signal control module 103 through the specific algorithm and definition of the mobile phone interconnection APP 1a;
[0102] The mobile terminal signal control module 103 converts the gesture control operation instruction into a corresponding control signal and sends it to the mobile terminal communication module 104. The mobile terminal communication module 104 transmits the data to the vehicle terminal communication module 204 through wireless technology. The vehicle terminal communication module 204 sends the control signal corresponding to the gesture control operation instruction to the safety control module 202 of the vehicle computer 2.
[0103] The safety control module 202 obtains the vehicle status information (vehicle speed, driving mode, emergency braking, etc.) transmitted by the driving sensing module 201 in real time, and comprehensively arbitrates the safety of executing the gesture control operation instruction under the current state of the vehicle through a safety verification mechanism and a specific algorithm: if it is determined to be safe, the control signal corresponding to the gesture control operation instruction is transmitted to the vehicle-side signal control module 203; if it is determined to be unsafe, the corresponding gesture control operation instruction is not executed, and a warning is issued;
[0104] The vehicle-side signal control module 203 analyzes the received control signal through a specific algorithm, and translates the received control signal into a gesture signal that can be recognized by the vehicle-side screen projection program 2a. After the translation is completed, the execution gesture signal is sent to the vehicle-side screen projection program 2a;
[0105] The car screen projection program 2a correctly recognizes the gesture signal and completes the multimedia playback status, such as pause (gesture action 3a), or volume (gesture action 3d), volume-(gesture action 3e), forward (gesture action 3f), and backward (gesture action 3g).
[0106] Embodiment 2:
[0107] First, user 3 needs to interconnect mobile phone 1 and vehicle computer 2, that is, mobile phone 1 and vehicle computer 2 perform wired or wireless screen projection, such as the mobile phone interconnection APP 1a and vehicle computer screen projection program 2a mentioned in the previous embodiment to open the mobile device and vehicle computer interconnection screen projection mode, and project the screen through wireless interconnection;
[0108] User 3 uses the Internet screen projection application to open the navigation icon in the application and use the navigation application in the application for real-time navigation;
[0109] When the user 3 needs to zoom in or out the map display, he only needs to make a corresponding gesture 301 towards the mobile phone 1. For example, if he needs to zoom out the map, he can make a gesture 3h (zoom out); if he needs to zoom in, he can make a gesture 3i (zoom in); if he needs to confirm the destination, he can directly make a gesture 3c (confirm);
[0110] At the same time, the front camera 101 in the mobile phone 1 captures the gesture action 3h (zooming out) of the user 3, transmits the acquired gesture image to the image processing module 1021 for preprocessing, and then inputs it into the convolutional neural network CNN model 1022, and obtains the corresponding gesture feature map through the classification and recognition functions of the convolutional neural network CNN model 1022;
[0111] The image processing module 1021 processes the identified continuous frame gesture feature graph to obtain gesture action category data, transmits the gesture control operation instruction corresponding to the gesture action category data to the mobile phone interconnection APP 1a, and sends the gesture control operation instruction to the mobile terminal signal control module 103 through the specific algorithm and definition of the mobile phone interconnection APP 1a;
[0112] The mobile terminal signal control module 103 converts the gesture control operation instruction into a corresponding control signal and sends it to the mobile terminal communication module 104. The mobile terminal communication module 104 transmits the data to the vehicle terminal communication module 204 through wireless technology. The vehicle terminal communication module 204 sends the control signal corresponding to the gesture control operation instruction to the safety control module 202 of the vehicle computer 2.
[0113] The safety control module 202 obtains the vehicle status information (vehicle speed, driving mode, emergency braking, etc.) transmitted by the driving sensing module 201 in real time, and comprehensively arbitrates the safety of executing the gesture control operation instruction under the current state of the vehicle through a safety verification mechanism and a specific algorithm: if it is determined to be safe, the control signal corresponding to the gesture control operation instruction is transmitted to the vehicle-side signal control module 203; if it is determined to be unsafe, the corresponding gesture control operation instruction is not executed, and a warning is issued;
[0114] The vehicle-side signal control module 203 analyzes the received control signal through a specific algorithm, and translates the received control signal into a gesture signal that can be recognized by the vehicle-side screen projection program 2a. After the translation is completed, the execution gesture signal is sent to the vehicle-side screen projection program 2a;
[0115] The car screen projection program 2a correctly recognizes the gesture signal and completes the map zooming out (gesture action 3h), or the map zooming in (gesture action 3i), destination confirmation (gesture action 3c), etc.
[0116] Embodiment three:
[0117] First, user 3 needs to interconnect mobile phone 1 and vehicle computer 2, that is, mobile phone 1 and vehicle computer 2 perform wired or wireless screen projection, such as the mobile phone interconnection APP 1a and vehicle computer screen projection program 2a mentioned in the previous embodiment to open the mobile device and vehicle computer interconnection screen projection mode, and project the screen through wireless interconnection;
[0118] User 3 uses the Internet screen projection application to open the gallery management (such as the photo album) in the application and opens the photo on mobile phone 1;
[0119] When user 3 needs to zoom in or out on a photo, he only needs to make a corresponding gesture 301 towards mobile phone 1. For example, if he needs to zoom out on a photo, he can make gesture 3h (zoom out); if he needs to zoom in, he can make gesture 3i (zoom in); if he needs to delete a photo, he can make gesture 3c (confirm);
[0120] At the same time, the front camera 101 in the mobile phone 1 captures the gesture action 3h (zooming out) of the user 3, transmits the acquired gesture image to the image processing module 1021 for preprocessing, and then inputs it into the convolutional neural network CNN model 1022, and obtains the corresponding gesture feature map through the classification and recognition functions of the convolutional neural network CNN model 1022;
[0121] The image processing module 1021 processes the identified continuous frame gesture feature graph to obtain gesture action category data, transmits the gesture control operation instruction corresponding to the gesture action category data to the mobile phone interconnection APP 1a, and sends the gesture control operation instruction to the mobile terminal signal control module 103 through the specific algorithm and definition of the mobile phone interconnection APP 1a;
[0122] The mobile terminal signal control module 103 converts the gesture control operation instruction into a corresponding control signal and sends it to the mobile terminal communication module 104. The mobile terminal communication module 104 transmits the data to the vehicle terminal communication module 204 through wireless technology. The vehicle terminal communication module 204 sends the control signal corresponding to the gesture control operation instruction to the safety control module 202 of the vehicle computer 2.
[0123] The safety control module 202 obtains the vehicle status information (vehicle speed, driving mode, emergency braking, etc.) transmitted by the driving sensing module 201 in real time, and comprehensively arbitrates the safety of executing the gesture control operation instruction under the current state of the vehicle through a safety verification mechanism and a specific algorithm: if it is determined to be safe, the control signal corresponding to the gesture control operation instruction is transmitted to the vehicle-side signal control module 203; if it is determined to be unsafe, the corresponding gesture control operation instruction is not executed, and a warning is issued;
[0124] The vehicle-side signal control module 203 analyzes the received control signal through a specific algorithm, and translates the received control signal into a gesture signal that can be recognized by the vehicle-side screen projection program 2a. After the translation is completed, the execution gesture signal is sent to the vehicle-side screen projection program 2a;
[0125] The car screen projection program 2a correctly recognizes the gesture signal and completes the reduction of the photo (gesture action 3h), or the enlargement of the map (gesture action 3i), the confirmation of photo deletion (gesture action 3c), etc.
[0126] Embodiment 4:
[0127] First, user 3 needs to interconnect mobile phone 1 and vehicle computer 2, that is, mobile phone 1 and vehicle computer 2 perform wired or wireless screen projection, such as the mobile phone interconnection APP 1a and vehicle computer screen projection program 2a mentioned in the previous embodiment to open the mobile device and vehicle computer interconnection screen projection mode, and project the screen through wireless interconnection;
[0128] User 3 uses the Internet screen projection application to open the video player in the application and play the video files / short videos stored locally or in mobile phone 1;
[0129] User 3 needs to switch to previous / next / pause / mute / volume+ / volume-, etc. At this time, he only needs to make a corresponding gesture 301 to the mobile phone 1. For example, if he needs to pause, he can make gesture 3a (pause), gesture 3d (volume+), gesture 3e (volume-), gesture 3f (forward), gesture 3g (backward), etc.
[0130] At the same time, the front camera 101 in the mobile phone 1 captures the gesture action 3h (zooming out) of the user 3, transmits the acquired gesture image to the image processing module 1021 for preprocessing, and then inputs it into the convolutional neural network CNN model 1022, and obtains the corresponding gesture feature map through the classification and recognition functions of the convolutional neural network CNN model 1022;
[0131] The image processing module 1021 processes the identified continuous frame gesture feature graph to obtain gesture action category data, transmits the gesture control operation instruction corresponding to the gesture action category data to the mobile phone interconnection APP 1a, and sends the gesture control operation instruction to the mobile terminal signal control module 103 through the specific algorithm and definition of the mobile phone interconnection APP 1a;
[0132] The mobile terminal signal control module 103 converts the gesture control operation instruction into a corresponding control signal and sends it to the mobile terminal communication module 104. The mobile terminal communication module 104 transmits the data to the vehicle terminal communication module 204 through wireless technology. The vehicle terminal communication module 204 sends the control signal corresponding to the gesture control operation instruction to the safety control module 202 of the vehicle computer 2.
[0133] The safety control module 202 obtains the vehicle status information (vehicle speed, driving mode, emergency braking, etc.) transmitted by the driving sensing module 201 in real time, and comprehensively arbitrates the safety of executing the gesture control operation instruction under the current state of the vehicle through a safety verification mechanism and a specific algorithm: if it is determined to be safe, the control signal corresponding to the gesture control operation instruction is transmitted to the vehicle-side signal control module 203; if it is determined to be unsafe, the corresponding gesture control operation instruction is not executed, and a warning is issued;
[0134] The vehicle-side signal control module 203 analyzes the received control signal through a specific algorithm, and translates the received control signal into a gesture signal that can be recognized by the vehicle-side screen projection program 2a. After the translation is completed, the execution gesture signal is sent to the vehicle-side screen projection program 2a;
[0135] The car screen projection program 2a correctly recognizes the gesture signal and completes the switching of the video playback status, such as pause (gesture action 3a), or volume + (gesture action 3d), volume - (gesture action 3e), forward (gesture action 3f), backward (gesture action 3g), etc.
[0136] Figure 2 A structural diagram of a gesture control system for a vehicle-mounted screen projection operation provided by an embodiment of the present invention, such as Figure 2 As shown, a gesture control system for in-vehicle screen projection operation includes a mobile device and a vehicle computer 2, wherein:
[0137] A mobile device, used to enable a screen projection mode of interconnection between the mobile device and the vehicle machine on the mobile device side, and to obtain a gesture image containing a gesture signal, and to extract a corresponding gesture control operation instruction according to the gesture image;
[0138] The vehicle computer 2 is used to start the screen projection mode of interconnection between the mobile device and the vehicle computer on the vehicle computer side, and to obtain the current status information of the vehicle, and to judge the safety of executing the gesture control operation instructions under the current status of the vehicle according to the safety verification mechanism, and to execute the gesture control operation instructions according to the judgment result.
[0139] It can be understood that the gesture control system for in-vehicle screen projection operation provided by the present invention corresponds to the gesture control method for in-vehicle screen projection operation provided by the aforementioned embodiments. The relevant technical features of the gesture control system for in-vehicle screen projection operation can refer to the relevant technical features of the gesture control method for in-vehicle screen projection operation, which will not be repeated here.
[0140] See also Figure 6 , Figure 6 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 6 As shown, an embodiment of the present invention provides an electronic device 600, including a memory 610, a processor 620, and a computer program 611 stored in the memory 610 and executable on the processor 620. When the processor 620 executes the computer program 611, the following steps are implemented:
[0141] Turn on the screen projection mode between the mobile device and the car machine;
[0142] Acquire a gesture image, and extract corresponding gesture control operation instructions according to the gesture image;
[0143] The current state information of the vehicle is obtained, the safety of executing the gesture control operation instruction under the current state of the vehicle is determined according to the safety verification mechanism, and the gesture control operation instruction is executed according to the determination result.
[0144] See also Figure 7 , Figure 7 Schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. Figure 7 As shown, this embodiment provides a computer-readable storage medium 700, on which a computer program 711 is stored. When the computer program 711 is executed by a processor, the following steps are implemented:
[0145] Turn on the screen projection mode between the mobile device and the car machine;
[0146] Acquire a gesture image, and extract corresponding gesture control operation instructions according to the gesture image;
[0147] The current state information of the vehicle is obtained, the safety of executing the gesture control operation instruction under the current state of the vehicle is determined according to the safety verification mechanism, and the gesture control operation instruction is executed according to the determination result.
[0148] The embodiment of the present invention provides a gesture control method, system and storage medium for in-vehicle screen projection operation. Through the interconnection between the portable mobile device and the vehicle computer 2, without adding extra equipment costs, it fully utilizes the image acquisition and image processing functions of the mobile device, so that the user 3 can control the vehicle without touching the screen, thereby improving the fun, convenience and safety of vehicle control. The present invention controls the in-vehicle screen projection operation by capturing hand movements, which not only greatly improves the interactive experience and safety during driving, but also shows the humanized development direction of future intelligent vehicle systems. By integrating advanced image recognition technology and intelligent algorithms, a more natural and intuitive human-computer interaction mode is achieved, which is an important technological innovation in the field of smart travel.
[0149] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0150] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0151] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0152] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0154] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0155] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A gesture control method for in-vehicle screen projection operation, It is characterized in that include: Turn on the screen projection mode between the mobile device and the car machine; Acquire a gesture image, and extract corresponding gesture control operation instructions according to the gesture image; The current state information of the vehicle is obtained, the safety of executing the gesture control operation instruction under the current state of the vehicle is determined according to the safety verification mechanism, and the gesture control operation instruction is executed according to the determination result.
2. According to claim 1, a gesture control method for in-vehicle screen projection operation, It is characterized in that The method of starting the screen projection mode of interconnection between the mobile device and the vehicle includes: Establish a communication connection between the mobile device and the vehicle computer, and enable the interconnected screen projection mode on the corresponding applications on the mobile device and the vehicle computer.
3. According to claim 1, a gesture control method for in-vehicle screen projection operation, It is characterized in that The step of acquiring the gesture image comprises: A camera on a mobile device is used to capture multiple frames of images containing user gestures.
4. The gesture control method for in-vehicle screen projection operation according to claim 1, It is characterized in that The step of extracting a corresponding gesture control operation instruction according to the gesture image includes: Extracting features from the gesture image to obtain a gesture feature map; Identify the user's gesture change information based on the continuous multi-frame gesture feature graph; A gesture control instruction library is searched according to the gesture change information to obtain a gesture control operation instruction that matches the current gesture change.
5. According to claim 4, a gesture control method for in-vehicle screen projection operation, It is characterized in that The step of extracting features from the gesture image to obtain a gesture feature map includes: Inputting the gesture image into a trained convolutional neural network (CNN) model; Extracting hand features in the gesture image through a convolutional layer; Increase the nonlinearity of the feature map through the activation function ReLU; Reducing the spatial dimension of the feature map through a pooling layer to extract key features of the hand features; Integrate the key features through a fully connected layer; The gesture feature map is output through the output layer based on the integrated key features.
6. The gesture control method for in-vehicle screen projection operation according to claim 1, It is characterized in that The obtaining of the vehicle current state information, judging the safety of executing the gesture control operation instruction in the vehicle current state according to the safety verification mechanism, and executing the gesture control operation instruction according to the judgment result, includes: Calculate the real-time driving speed of the vehicle according to the current wheel speed information of the vehicle, determine the current driving mode of the vehicle according to the current gear signal of the vehicle, and obtain the current fault signal of the vehicle; According to the real-time driving speed, current driving mode and current fault signal, the safety of executing the gesture control operation instruction in the current state of the vehicle is comprehensively arbitrated in combination with a preset safety verification mechanism: If it is determined to be safe, the corresponding gesture control operation instruction is executed; If it is determined to be unsafe, the corresponding gesture control operation instruction will not be executed and a warning will be issued.
7. A gesture control system for in-vehicle screen projection operation, It is characterized in that include: A mobile device, used to enable a screen projection mode of interconnection between the mobile device and the vehicle machine on the mobile device side, and to obtain a gesture image containing a gesture signal, and to extract a corresponding gesture control operation instruction according to the gesture image; The vehicle computer is used to enable the interconnection and projection mode between the mobile device and the vehicle computer on the vehicle computer side, and to obtain the current status information of the vehicle, and to judge the safety of executing the gesture control operation instructions under the current status of the vehicle according to the safety verification mechanism, and to execute the gesture control operation instructions according to the judgment result.
8. An electronic device, It is characterized in that It includes a memory and a processor, and the processor is used to implement the steps of a gesture control method for in-vehicle screen projection operation as described in any one of claims 1 to 6 when executing a computer management program stored in the memory.
9. A computer-readable storage medium, It is characterized in that A computer management program is stored thereon, and when the computer management program is executed by the processor, the steps of a gesture control method for in-vehicle screen projection operation as described in any one of claims 1-6 are implemented.
