Vehicle control method and device based on gesture recognition, electronic equipment and medium

Abstract

The invention relates to the technical field of vehicle control, and discloses a vehicle control method and device based on gesture recognition, electronic equipment and a medium, and the method comprises the steps: obtaining gesture information in a preset recognition region, carrying out the gesture recognition of the gesture information through a preset gesture recognition model, and obtaining a target gesture, the gesture recognition model is obtained by carrying out model training on a gesture information sample with a gesture label, a target control instruction is determined from preset control instructions according to a matching result of a target gesture and a preset gesture, the preset gesture and the preset control instruction have a one-to-one correspondence relationship, and the target control instruction and the preset control instruction have a one-to-one correspondence relationship; controlling a target vehicle part on the vehicle according to the target control instruction to respond to the gesture information; the vehicle control method based on gesture recognition is more practical and more accurate.

Description

Technical Field

[0001] The present application relates to the field of vehicle control technology, and in particular to a vehicle control method, device, electronic device and medium based on gesture recognition. Background Art

[0002] In traditional dedicated vehicle control technology, physical buttons or lever controllers are usually used to implement user operations. Although this method is simple and direct, it has some obvious limitations and shortcomings. First, the use of buttons and lever controllers requires users to divert their attention to find and operate specific buttons or levers when moving or operating dedicated vehicles, especially buttons or lever controllers that are far away from the driver. This operation method not only increases the complexity of operation, but also limits the user's freedom and flexibility, which is especially inconvenient for users with limited mobility.

[0003] In order to solve these problems, a new control method based on gesture recognition has emerged. Gesture recognition technology uses advanced sensor technology, such as cameras and inertial sensors, to capture and analyze user gestures in real time. These gestures can include arm movements, finger posture changes, etc., which are recognized and analyzed by the processor module. Once the user's intention and operation instructions are recognized, the control module can realize the corresponding actions of the special vehicle, such as forward, backward, and steering. This system is not only more intuitive and natural to operate, and users can quickly get started without learning complex control methods, but also greatly improves the user's convenience and comfort.

[0004] Compared with the traditional button control method, the control method based on gesture recognition has many advantages. First, it significantly improves the user experience, making the operation of special vehicles simpler, smarter and more personalized. Users can control the vehicle through simple gestures, which is not only more intuitive, but also avoids complicated button layouts and operating steps. Secondly, the gesture recognition control method is more flexible and can recognize a variety of gestures corresponding to different operations, which provides users with more choices and freedom, making the control method more in line with individual differences and preferences. In addition, the gesture recognition control method can also adapt to different environments and usage scenarios, such as indoors, outdoors, and different terrains, providing users with a consistent and efficient operating experience.

[0005] Gesture recognition technology is particularly important and beneficial for users with limited mobility. These users often face additional challenges when operating button-controlled systems, such as insufficient hand strength or inconvenient movements. The introduction of gesture recognition control provides them with a more convenient and autonomous mobile solution. They can control special vehicles with simple gestures, no longer relying on complex buttons or lever controllers, thereby improving their quality of life and independence.

[0006] In short, the special vehicle control method based on gesture recognition represents an important development direction of future intelligent special vehicle technology. It not only significantly improves the user's operating experience and ease of use, but also has important application potential in technological innovation and intelligent development. With the further maturity of technology and the promotion of application, this system is expected to become the mainstream control method in the field of special vehicles, bringing more convenient and intelligent travel experience to the majority of users.

[0007] It should be noted that the above content only provides background technical information related to the present application and does not necessarily constitute prior art. Summary of the invention

[0008] In view of the shortcomings of the prior art mentioned above, the present application provides a vehicle control method, device, electronic device and medium based on gesture recognition, so that users can achieve precise control of special vehicles through simple gesture movements, without the need for additional learning and cumbersome operating steps, thereby improving the user's operating experience and ease of use, which is particularly suitable for users with limited mobility, and brings higher autonomy and quality of life.

[0009] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0010] According to one aspect of an embodiment of the present application, a vehicle control method based on gesture recognition is provided, including: acquiring gesture information within a preset recognition area; performing gesture recognition on the gesture information through a preset gesture recognition model to obtain a target gesture, wherein the gesture recognition model is obtained by model training using gesture information samples with gesture labels; determining a target control instruction from preset control instructions based on a matching result between the target gesture and a preset gesture, wherein there is a one-to-one correspondence between the preset gesture and the preset control instruction; and controlling a target vehicle component on the vehicle according to the target control instruction to respond to the gesture information.

[0011] In one embodiment of the present application, based on the aforementioned scheme, the gesture recognition model is obtained by model training with gesture information samples with gesture labels, including: obtaining multiple historical gesture information and gesture results corresponding to each of the historical gesture information; taking each of the historical gesture information as a gesture information sample, and labeling the gesture information sample according to the gesture result to obtain the gesture label corresponding to each of the gesture information samples; performing model training on the gesture information samples with gesture labels to obtain the gesture recognition model.

[0012] In one embodiment of the present application, based on the aforementioned scheme, gesture recognition is performed on the gesture information through a preset gesture recognition model to obtain a target gesture, including: performing gesture recognition on the gesture information through the gesture recognition model to obtain gesture key points and gesture motion trajectories; comparing the gesture key points with preset key points to obtain a first comparison result; comparing the gesture motion trajectory with the preset motion trajectory to obtain a second comparison result; and performing fusion processing based on the first comparison result and the second comparison result to determine the target gesture.

[0013] In one embodiment of the present application, based on the aforementioned scheme, gesture recognition is performed on the gesture information through a preset gesture recognition model to obtain a target gesture, and the method further includes: extracting features from the gesture information to obtain gesture key point features and motion trajectory features, wherein the motion trajectory features include a trend of gesture changes in space over time; inputting the gesture key point features and the motion trajectory features into the gesture recognition model to obtain the target gesture, and the gesture recognition model is a convolutional deep neural network; wherein, feature extraction is performed on gesture information samples in a training set to obtain sample gesture key point features and sample motion trajectory features; a target gesture key point category probability is determined based on the sample gesture key point features, and a target motion trajectory category probability is determined based on the sample motion trajectory features; a target training loss function is constructed based on the target motion trajectory category probability and the target gesture key point category probability; and a convolutional deep neural network is trained through the target training loss function to obtain the gesture recognition model.

[0014] In one embodiment of the present application, based on the aforementioned scheme, a target control instruction is determined from preset control instructions according to the matching result between the target gesture and the preset gesture, including: performing a similarity comparison between the target gesture and the preset gesture to obtain a similarity comparison result; if the similarity comparison result is greater than or equal to a preset comparison threshold, it is determined that the target gesture matches the preset gesture successfully; and the preset control instruction corresponding to the preset gesture is determined as the target control instruction.

[0015] In one embodiment of the present application, based on the aforementioned scheme, the target gesture is compared with the preset gesture for similarity, and after obtaining the similarity comparison result, the method further includes: if the similarity comparison result is less than the preset comparison threshold, it is determined that the target gesture fails to match the preset gesture; feedback information is generated based on the determination result to notify the user that the current gesture fails to match, and to suggest that the user re-execute the gesture or perform a preset alternative operation.

[0016] In one embodiment of the present application, based on the aforementioned scheme, the method also includes at least one of the following: generating a gesture model of the target gesture according to the target gesture, displaying the gesture model on a vehicle display screen of the target vehicle, and confirming the target gesture according to a first input instruction from the user; generating current control information according to the target control instruction and the target vehicle component, displaying the current control information on the vehicle display screen, and confirming the target control instruction according to a second input instruction from the user.

[0017] According to one aspect of an embodiment of the present application, a vehicle control device based on gesture recognition is provided, including: an acquisition module, used to acquire gesture information within a preset recognition area; a recognition module, used to perform gesture recognition on the gesture information through a preset gesture recognition model to obtain a target gesture, wherein the gesture recognition model is obtained by model training using gesture information samples with gesture labels; a determination module, used to determine a target control instruction from preset control instructions based on a matching result between the target gesture and a preset gesture, wherein there is a one-to-one correspondence between the preset gesture and the preset control instruction; and a control module, used to control a target vehicle component on the vehicle according to the target control instruction to respond to the gesture information.

[0018] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements a vehicle control method based on gesture recognition as described in any one of the above embodiments.

[0019] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of a computer, the computer is enabled to execute a vehicle control method based on gesture recognition as described in any one of the above embodiments.

[0020] The beneficial effects of the present application are as follows: the present application obtains gesture information within a preset recognition area, performs gesture recognition on the gesture information through a preset gesture recognition model, and obtains a target gesture, wherein the gesture recognition model is obtained by model training using gesture information samples with gesture labels, and determines a target control instruction from a preset control instruction according to a matching result between the target gesture and the preset gesture, wherein there is a one-to-one correspondence between the preset gesture and the preset control instruction, and controls a target vehicle component on the vehicle according to the target control instruction to respond to the gesture information; solves the problems of inconvenience in use and cumbersome operation existing in traditional vehicle control methods, and avoids the inconvenience caused to users with limited mobility by traditional physical buttons or lever-type controllers. Users only need to use simple gesture movements, such as arm movements or finger posture changes, to achieve precise control of the vehicle without the need for additional learning and cumbersome operating steps, thereby improving the user's operating experience and ease of use, and is particularly suitable for users with limited mobility, bringing higher autonomy and quality of life.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0023] Figure 1 is a schematic diagram of an exemplary system architecture shown in an exemplary embodiment of the present application;

[0024] Figure 2 is a flow chart of a vehicle control method based on gesture recognition shown in an exemplary embodiment of the present application;

[0025] Figure 3 is a schematic diagram of gesture instructions of a vehicle control method based on gesture recognition shown in an exemplary embodiment of the present application;

[0026] Figure 4 is a schematic diagram of the overall framework of a vehicle control method based on gesture recognition shown in an exemplary embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a flow chart of determining a target control instruction of a vehicle control method based on gesture recognition according to an exemplary embodiment of the present application;

[0028] Figure 6is a schematic diagram of a flow chart of executing a gesture instruction of a vehicle control method based on gesture recognition according to an exemplary embodiment of the present application;

[0029] Figure 7 is a block diagram of a vehicle control device based on gesture recognition according to an exemplary embodiment of the present application;

[0030] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0032] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present application in a schematic manner, and therefore the illustrations only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0033] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0034] First of all, it should be noted that Convolutional Deep Neural Networks (CNNs) are a type of deep learning model that is particularly suitable for image and video recognition. They can automatically extract high-level, abstract feature representations from raw images by simulating the visual processing mechanism in the human brain, thereby performing tasks such as image classification and object detection.

[0035] Pattern recognition algorithm is an important technology, which is widely used in many fields such as image recognition, speech recognition, bioinformatics, etc. This kind of algorithm analyzes and processes the characteristics of data and matches these characteristics with known patterns to achieve classification and prediction of new data.

[0036] The DMS (Driver Monitor System) camera is a camera specifically used to monitor the driver's behavior and status. It mainly captures the driver's facial parameters and compares them with the preset database to determine whether the driver is distracted or fatigued, and triggers the warning system in time to ensure driving safety. The DMS camera is an important part of ADAS (Advanced Driver Assistance System), which can realize functions such as driver identity recognition, fatigue driving detection, and dangerous driving behavior monitoring.

[0037] The OMS (Occupant Monitor System) camera is a derivative application of the DMS system that extends to the entire interior space of the car. It monitors the status of the driver and passengers based on visual technology, and provides corresponding intelligent services by identifying the behavior and needs of passengers. For example, the OMS system can identify whether the passenger is wearing a seat belt, whether he has left it in the car, and whether the seat or air conditioning needs to be adjusted.

[0038] Figure 1 It is a schematic diagram of an exemplary system architecture shown in an exemplary embodiment of the present application.

[0039] Reference Figure 1 As shown, the system architecture may include a data acquisition device 101 and a computer device 102. The computer device 102 may be at least one of a desktop graphics processing unit (GPU) computer, a GPU computing cluster, a neural network computer, etc. The data acquisition device 101 is used to collect gesture information within a preset recognition area. In this embodiment, the data acquisition device 101 obtains the above data and provides it to the computer device 102 for processing. Relevant technicians can use the computer device 102 to perform gesture recognition on the gesture information through a preset gesture recognition model to obtain a target gesture, wherein the gesture recognition model is obtained by model training with gesture information samples with gesture labels, and the target control instruction is determined from the preset control instruction according to the matching result between the target gesture and the preset gesture, wherein there is a one-to-one correspondence between the preset gesture and the preset control instruction, and the target vehicle component on the vehicle is controlled according to the target control instruction to respond to the gesture information to complete the vehicle control based on gesture recognition. It should be noted that the data acquisition device 101 and the computer device 102 provided in this embodiment are only an example and should not bring any limitation to the function and scope of use of the embodiment of the present application.

[0040] It should be noted that the vehicle control method based on gesture recognition provided in the embodiment of the present application is generally executed by the computer device 102 , and accordingly, the vehicle control device based on gesture recognition is generally arranged in the computer device 102 .

[0041] Figure 2 is a flow chart of a vehicle control method based on gesture recognition according to an exemplary embodiment of the present application. The vehicle control method based on gesture recognition can be executed by a computing and processing device, which can be Figure 1 The computer device 102 shown in FIG. Figure 2 As shown, the vehicle control method based on gesture recognition includes at least steps S210 to S240, which are described in detail as follows:

[0042] In step S210, gesture information within a preset recognition area is obtained.

[0043] In one embodiment of the present application, gesture information within a preset recognition area is obtained through a multi-sensor system, and the multi-sensor system may include but is not limited to an image acquisition device, a depth sensor, and a radar sensor, such as a short-range radar, a color camera, and a depth camera for dynamic gesture recognition. Also for example, an in-car camera (DMS camera, OMS camera) and a posture sensor. The multi-sensor system detects dynamic gestures with the help of a short-range radar system. In addition, a convolutional deep neural network classifier is used to fuse the collected data from the three sensors and classify ten different dynamic gestures. There are many advantages to combining an image sensor, i.e., an image acquisition device, a depth sensor, and a radar sensor. First, the robustness of the overall solution to different lighting conditions can be improved, ensuring that the data collected by at least one sensor is reliable under all lighting conditions. Secondly, since the three sensors provide complementary information about the shape, color, and instantaneous angular velocity of the hand, they can be combined to improve the classification accuracy of the system. Finally, the use of a radar sensor can help easily detect and segment dynamic gestures and reduce the power consumption of the system.

[0044] In this embodiment, the user's gesture is captured by an image sensor, that is, a camera, to obtain an image or video frame containing gesture information, to extract feature points in the image through deep learning and computer vision technology, to match the extracted feature points with a preset gesture template, and to determine the type of gesture through a pattern recognition algorithm. The depth information of the gesture is captured by a depth sensor to generate a depth image containing the three-dimensional shape of the gesture, to match the extracted three-dimensional features of the gesture with a preset gesture template, and to determine the type of gesture through a pattern recognition algorithm. Depth information can provide more accurate gesture position and shape information, which helps to identify complex gestures and subtle movements. The radar sensor uses the emitted radio frequency waves and receives the reflected signals, and estimates the position, speed and other parameters of the gesture by measuring the delay, phase and other information of the signal. To extract the dynamic features of the gesture, such as the motion trajectory of the gesture, speed changes, etc. The extracted features are classified and identified using a machine learning algorithm, a convolutional neural network, to determine the type of gesture.

[0045] In step S220, gesture recognition is performed on the gesture information using a preset gesture recognition model to obtain a target gesture.

[0046] In one embodiment of the present application, a process of performing gesture recognition on gesture information through a preset gesture recognition model to obtain a target gesture includes the following steps: performing gesture recognition on gesture information through a gesture recognition model to obtain gesture key points and gesture motion trajectories; comparing the gesture key points with the preset key points to obtain a first comparison result; comparing the gesture motion trajectory with the preset motion trajectory to obtain a second comparison result; and performing fusion processing based on the first comparison result and the second comparison result to determine the target gesture.

[0047] In this embodiment, the user's gesture information is identified through the camera and the sensor, the gesture information obtained by the camera and the sensor is analyzed to identify the gesture key points and the gesture motion trajectory, the gesture key points are compared with the preset key points to obtain a first comparison result; the gesture motion trajectory is compared with the preset motion trajectory to obtain a second comparison result; and the user gesture recognition result, i.e., the target gesture, is determined based on the intersection of the first comparison result and the second comparison result.

[0048] In one embodiment of the present application, the steps of training a gesture recognition model using gesture information samples with gesture labels include: obtaining multiple historical gesture information and gesture results corresponding to each historical gesture information; using each historical gesture information as a gesture information sample, and labeling the gesture information sample according to the gesture result to obtain a gesture label corresponding to each gesture information sample; training the gesture information samples with gesture labels to obtain a gesture recognition model.

[0049] In one embodiment of the present application, gesture information is subjected to gesture recognition through a preset gesture recognition model, and the process of obtaining a target gesture further includes the following steps: performing feature extraction on the gesture information to obtain gesture key point features and motion trajectory features, wherein the motion trajectory features include a trend of gesture changes in space over time; inputting the gesture key point features and the motion trajectory features into the gesture recognition model to obtain the target gesture, and the gesture recognition model is a convolutional deep neural network; wherein, feature extraction is performed on gesture information samples in a training set to obtain sample gesture key point features and sample motion trajectory features; the target gesture key point category probability is determined based on the sample gesture key point features, and the target motion trajectory category probability is determined based on the sample motion trajectory features; a target training loss function is constructed based on the target motion trajectory category probability and the target gesture key point category probability; and the convolutional deep neural network is trained through the target training loss function to obtain a gesture recognition model.

[0050] In this embodiment, it can be understood that the target training loss function can be a loss function required for training the initial gesture recognition model, i.e., the convolutional deep neural network. A first cross entropy loss function is constructed by the target gesture key point category probability, and a second cross entropy loss function is constructed by the main target motion trajectory category probability; a target training loss function is constructed based on the first cross entropy loss function and the second cross entropy loss function.

[0051] In this embodiment, the losses of multiple feature dimensions are fused into a total loss function by weighted summing the first cross entropy loss function and the second cross entropy loss function. Specifically, a target training loss function is constructed based on the first cross entropy loss function and the first preset weight, the second cross entropy loss function and the second preset weight, wherein the first preset weight is the weight corresponding to the gesture key point feature, and the second preset weight is the weight corresponding to the motion trajectory feature. It can be understood that the first preset weight and the second preset weight can be set according to actual needs.

[0052] In step S230, a target control instruction is determined from preset control instructions according to a matching result between the target gesture and the preset gesture.

[0053] In one embodiment of the present application, the target gesture is compared with the preset gesture for similarity to obtain a similarity comparison result; if the similarity comparison result is greater than or equal to a preset comparison threshold, it is determined that the target gesture matches the preset gesture successfully; and the preset control instruction corresponding to the preset gesture is determined as the target control instruction. There is a one-to-one correspondence between the preset gesture and the preset control instruction.

[0054] In this embodiment, there is a one-to-one correspondence between the preset gesture and the target vehicle component, i.e., the control command of the target vehicle component. The one-to-one correspondence includes that a gesture representing a user's intention has only a unique control instruction corresponding to it, i.e., one preset gesture corresponds to one preset control instruction. The gesture recognition model is trained using a convolutional deep neural network classifier to integrate and analyze data collected by various sensors. The preset gesture can be stored in the cloud, i.e., the target gesture is compared with the preset gesture in the cloud for recognition. In the present application, by storing the preset gesture in the cloud in advance, it is ensured that the user can match the gesture action when using it, i.e., when performing gesture actions, to ensure fast and efficient recognition. At the same time, a simple gesture action combination can also achieve complex instructions. For special vehicles, such as explosion-proof, transportation, industry, etc., more work instructions can be realized and the work effect can be improved.

[0055] In one embodiment of the present application, a target gesture is compared with a preset gesture for similarity. After obtaining a similarity comparison result, if the similarity comparison result is less than a preset comparison threshold, it is determined that the target gesture fails to match the preset gesture; feedback information is generated based on the determination result to notify the user that the current gesture fails to match, and to suggest that the user re-execute the gesture or perform a preset alternative operation.

[0056] In this embodiment, when the gesture recognition similarity comparison process fails to successfully match the expected result, that is, when a comparison failure occurs, clear feedback information is immediately sent to the vehicle to inform the user or the vehicle control system that the current gesture recognition has not been successfully performed. The feedback information may include a brief description or a visual prompt, which is intended to clearly convey the status of the comparison failure to the user and suggest that the user re-execute the gesture or take other preset alternative operations. The preset alternative operations include but are not limited to voice commands or directly using the corresponding physical buttons or stick controllers.

[0057] In one embodiment of the present application, the vehicle control method based on gesture recognition also includes at least one of the following: generating a gesture model of the target gesture according to the target gesture, displaying the gesture model on a vehicle display screen of the target vehicle, and confirming the target gesture according to a first input instruction from the user; generating current control information according to the target control instruction and the target vehicle component, displaying the current control information on the vehicle display screen, and confirming the target control instruction according to a second input instruction from the user.

[0058] In this embodiment, before querying and determining the target control instruction corresponding to the target gesture, the system will display the user's gesture recognition result, that is, the target gesture, in real time on the vehicle display screen in the form of a three-dimensional model. After querying the corresponding target control instruction based on the user gesture recognition result, the system will generate current control information based on the target control instruction and the target vehicle component, and display the current control information on the vehicle display screen.

[0059] In this embodiment, the vehicle control instruction at least includes the user's operation intention on the vehicle's front hood, sunroof, lights, navigation, and answering calls and audio, such as opening, closing, etc. Figure 3 , Figure 3 is a schematic diagram of gesture instructions of a vehicle control method based on gesture recognition, shown in an exemplary embodiment of the present application. Figure 3 The preset gestures and their corresponding preset control instructions are shown. Figure 3 The gestures and corresponding instructions are only examples. In actual applications, they can be set according to actual needs or user preferences. This application does not limit this.

[0060] In step S240 , a target vehicle component on the vehicle is controlled according to the target control instruction to respond to the gesture information.

[0061] In one embodiment of the present application, historical control instructions and historical control results corresponding to the historical control instructions are obtained; the user's component control habits are determined according to the historical control instructions and the historical control results; and the target vehicle components are controlled according to the target control instructions and the component control habits. The vehicle system can automatically adapt to the driver's habits through continuous learning and optimization of the control algorithm, and even predict and meet the driving intention that is not clearly expressed, thereby improving the user's experience and the accuracy of vehicle component control.

[0062] In one embodiment of the present application, the target gesture is used to trigger corresponding controls of the vehicle, such as windows, audio, navigation, etc. These controls should directly reflect the user's intentions. After receiving the target control instruction corresponding to the target gesture, the vehicle executes the corresponding action and feeds back the execution result (i.e., the control result) to the system.

[0063] Figure 4 It is a schematic diagram of the overall framework of a vehicle control method based on gesture recognition shown in an exemplary embodiment of the present application. In an exemplary embodiment, the user performs a gesture action, and the gesture information is collected in a preset recognition area through the DMS camera, the posture sensor, and the OMS camera. The gesture recognition model is obtained through the deep neural network algorithm, and the gesture recognition result, i.e., the target gesture, is obtained through the gesture recognition model. The target control instruction is determined from the preset control instruction according to the comparison result between the target gesture and the preset gesture, the target control instruction is transmitted to the vehicle, and the target vehicle component corresponding to the target control instruction is controlled to execute the target control instruction. The specific implementation method has been described in detail in the above embodiments and will not be repeated here.

[0064] Figure 5It is a flow chart of determining target control instructions of a vehicle control method based on gesture recognition, shown in an exemplary embodiment of the present application. In an exemplary embodiment, first, visual sensors and cameras installed inside the vehicle are used to obtain user gesture information. These sensors are located at different locations in the vehicle, such as the roof, dashboard or steering wheel, to capture the driver's gestures. The visual sensor can accurately identify and track the position, movement and posture changes of the driver's hands, and collect original signals. Secondly, the data processing unit integrates the collected signals, especially for the presence of interference signals in complex environments, so signal screening is required. In the data processing stage, the algorithm analyzes the characteristics and intentions of the driver's gestures through deep learning and computer vision technology. These algorithms can recognize different gestures, such as clenching fists, pointing, waving, etc., and can distinguish between normal driving operations and unexpected actions to ensure the accuracy and reliability of the system. After that, the infrared sensor is used to locate the human body position, determine the hand signal, and extract the hand movement features. When the driver makes a specific gesture, the system can analyze the intention of the gesture, determine the target gesture based on the result of the comparison with the preset gesture, and then confirm the target control command, such as adjusting the volume, switching music, answering the phone, etc., which not only improves the driver's user experience, but also enhances the sense of connection and safety between the driver and the vehicle. At the same time, the feedback mechanism of the vehicle ensures that the driver can get instant feedback. For example, after recognizing the driver's gesture, the system can confirm the driver's instructions through a display screen or voice prompts, as well as the status feedback of the system performing the corresponding operation. This instant feedback helps to reduce the driver's distraction and operation errors and improve driving safety. In this embodiment, the extracted hand movement features can be uploaded to the cloud, so that the cloud can determine and send the corresponding preset gesture to the target vehicle according to the received hand movement features, so that the target vehicle can compare the target gesture with the preset gesture; the target gesture can also be sent to the cloud after determining the target gesture, so that the target gesture can be compared with the preset gesture in the cloud, and the target control command can be determined according to the comparison result.

[0065] Figure 6This is a flow chart of the execution of gesture instructions of a vehicle control method based on gesture recognition, shown in an exemplary embodiment of the present application. In an exemplary embodiment, before the formal gesture recognition is performed, when the vehicle is started, various sensors (including visual sensors, radars, and lidars, etc.) begin to collect data of the surrounding environment in real time to obtain gesture information. When the user performs a gesture action, the action information is captured by the gesture information, and gesture recognition is performed. If the gesture recognition fails, the user needs to be reminded to make the gesture again, and the display reminds that the gesture recognition failed; if the gesture recognition is successful, the recognition data can be sent to the vehicle data processing system to guide the vehicle to perform the correct work, such as opening windows, playing music, wipers, etc. In particular, for special vehicles, special gestures can be set to perform professional work instructions, so that the work can be completed more quickly and efficiently, while reducing the risk of drivers of special vehicles.

[0066] The present application achieves control of vehicle components, such as the front hood, sunroof, navigation, lighting, and music playing of the vehicle, by recognizing the gestures of the vehicle driver. The vehicle control gesture recognition system is implemented by comparing the gestures recognized in the gesture recognition area by the in-vehicle camera, attitude sensor, and convolutional deep neural network classifier with the gestures set by the user in the cloud to achieve the control of vehicle components; the activation of the vehicle control gesture recognition system corresponds to a separate activation gesture, which improves the user experience, the safety and stability of vehicle driving, solves the inconvenience and limitations of traditional vehicle control methods, and improves the intuitiveness and naturalness of user operations. Through simple gestures, users can achieve precise control of the vehicle without complex button operations, significantly reducing cognitive burden and physical exertion, not only improving the convenience and comfort of operation, but also enhancing the safety and accuracy of operation, especially suitable for users with limited mobility, providing them with a more independent and autonomous vehicle control solution, and bringing higher autonomy and quality of life.

[0067] Figure 7 is a block diagram of a vehicle control device based on gesture recognition shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The implementation environment shown in the figure is specifically configured in the computer device 102. The device can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applied.

[0068] like Figure 7 As shown, the exemplary vehicle control device based on gesture recognition includes: an acquisition module 710 , a recognition module 720 , a determination module 730 and a control module 740 .

[0069] Among them, the acquisition module 710 is used to obtain gesture information within a preset recognition area; the recognition module 720 is used to perform gesture recognition on the gesture information through a preset gesture recognition model to obtain a target gesture, wherein the gesture recognition model is obtained by model training using gesture information samples with gesture labels; the determination module 730 is used to determine a target control instruction from preset control instructions based on a matching result between the target gesture and the preset gesture, wherein there is a one-to-one correspondence between the preset gesture and the preset control instruction; the control module 740 is used to control a target vehicle component on the vehicle according to the target control instruction to respond to the gesture information.

[0070] It should be noted that the vehicle control device based on gesture recognition provided in the above embodiment and the vehicle control method based on gesture recognition provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In actual application, the vehicle control device based on gesture recognition provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0071] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the electronic device implements the vehicle control method based on gesture recognition provided in the above-mentioned embodiments.

[0072] Figure 8 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0073] like Figure 8As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage part 808 to the random access memory (RAM) 803, such as executing the methods provided in the above-mentioned various embodiments. In the RAM 803, various programs and data required for system operation are also stored. The CPU 801, the ROM 802 and the RAM 803 are connected to each other through the bus 804. The input / output (I / O) interface 805 is also connected to the bus 804.

[0074] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0075] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 809, and / or installed from a removable medium 811. When the computer program is executed by a central processing unit (CPU) 801, various functions defined in the system of the present application are executed.

[0076] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0077] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0078] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0079] Another aspect of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes the vehicle control method based on gesture recognition provided in the above embodiments. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently without being assembled into the electronic device.

[0080] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0081] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the vehicle control method based on gesture recognition provided in each of the above embodiments.

[0082] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0083] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0084] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A vehicle control method based on gesture recognition, characterized in that: include: Get gesture information within the preset recognition area; Performing gesture recognition on the gesture information through a preset gesture recognition model to obtain a target gesture, wherein the gesture recognition model is obtained by model training using gesture information samples with gesture labels; Determining a target control instruction from preset control instructions according to a matching result between the target gesture and the preset gesture, wherein there is a one-to-one correspondence between the preset gesture and the preset control instruction; A target vehicle component on the vehicle is controlled according to the target control instruction to respond to the gesture information.

2. The vehicle control method based on gesture recognition according to claim 1, characterized in that: The gesture recognition model is obtained by training the model using gesture information samples with gesture labels, including: Acquire multiple historical gesture information and gesture results corresponding to each of the historical gesture information; Taking each of the historical gesture information as a gesture information sample, and marking the gesture information sample according to the gesture result, to obtain a gesture label corresponding to each of the gesture information samples; The gesture information samples with gesture labels are subjected to model training to obtain the gesture recognition model.

3. The vehicle control method based on gesture recognition according to claim 1, characterized in that: Performing gesture recognition on the gesture information through a preset gesture recognition model to obtain a target gesture includes: Performing gesture recognition on the gesture information through the gesture recognition model to obtain gesture key points and gesture motion trajectory; Comparing the gesture key point with the preset key point to obtain a first comparison result; Comparing the gesture motion trajectory with a preset motion trajectory to obtain a second comparison result; The target gesture is determined by performing a fusion process based on the first comparison result and the second comparison result.

4. The vehicle control method based on gesture recognition according to claim 1, characterized in that: Performing gesture recognition on the gesture information through a preset gesture recognition model to obtain a target gesture, further comprising: Extracting features of the gesture information to obtain gesture key point features and motion trajectory features, wherein the motion trajectory features include a change trend of the gesture in space over time; Inputting the gesture key point features and the motion trajectory features into the gesture recognition model to obtain the target gesture, wherein the gesture recognition model is a convolutional deep neural network; Among them, feature extraction is performed on gesture information samples in the training set to obtain sample gesture key point features and sample motion trajectory features; the target gesture key point category probability is determined based on the sample gesture key point features, and the target motion trajectory category probability is determined based on the sample motion trajectory features; a target training loss function is constructed according to the target motion trajectory category probability and the target gesture key point category probability; a convolutional deep neural network is trained by the target training loss function to obtain the gesture recognition model.

5. The vehicle control method based on gesture recognition according to claim 1, characterized in that: Determining a target control instruction from the preset control instructions according to a matching result between the target gesture and the preset gesture includes: Comparing the target gesture with the preset gesture to obtain a similarity comparison result; If the similarity comparison result is greater than or equal to a preset comparison threshold, it is determined that the target gesture matches the preset gesture successfully; The preset control instruction corresponding to the preset gesture is determined as the target control instruction.

6. The vehicle control method based on gesture recognition according to claim 5, characterized in that: After comparing the target gesture with the preset gesture for similarity and obtaining a similarity comparison result, the method further includes: If the similarity comparison result is less than the preset comparison threshold, it is determined that the target gesture fails to match the preset gesture; Feedback information is generated based on the determination result to inform the user that the current gesture matching fails, and to suggest the user to re-execute the gesture or perform a preset alternative operation.

7. The vehicle control method based on gesture recognition according to any one of claims 1 to 6, characterized in that: The method further comprises at least one of the following: generating a gesture model of the target gesture according to the target gesture, displaying the gesture model on a vehicle display screen of the target vehicle, and confirming the target gesture according to a first input instruction of the user; Current control information is generated according to the target control instruction and the target vehicle component, the current control information is displayed on the vehicle display screen, and the target control instruction is confirmed according to a second input instruction of the user.

8. A vehicle control device based on gesture recognition, characterized in that: include: An acquisition module is used to acquire gesture information within a preset recognition area; A recognition module, used to perform gesture recognition on the gesture information through a preset gesture recognition model to obtain a target gesture, wherein the gesture recognition model is obtained by model training using gesture information samples with gesture labels; A determination module, configured to determine a target control instruction from preset control instructions according to a matching result between the target gesture and a preset gesture, wherein there is a one-to-one correspondence between the preset gesture and the preset control instruction; A control module is used to control a target vehicle component on the vehicle according to the target control instruction to respond to the gesture information.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the vehicle control method based on gesture recognition as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the vehicle control method based on gesture recognition as described in any one of claims 1 to 7.

Images