Vehicle-mounted air conditioner control method and device, electronic equipment and storage medium

Abstract

The invention provides a vehicle-mounted air conditioner control method and device, electronic equipment and a storage medium, and the method comprises the steps that an image frame is obtained, the image frame is generated by a monocular camera, and the monocular camera is used for capturing a user gesture action appearing in front of a vehicle-mounted screen; the quality index value of the image frame is determined, and the quality index value of the image frame represents the illumination quality of the image frame; and when the quality index value of the image frame is smaller than a first preset threshold value, performing quality enhancement processing on the image frame so that the quality index value of the image frame is greater than the first preset threshold value. According to the method, the image frames generated by the monocular camera are adopted for recognition, multiple cameras are not needed for imaging, the technical defect of high cost in the prior art can be overcome at least, and therefore the method has the advantage of being low in cost. And on the other hand, through image quality enhancement processing, the influence of an illumination environment of monocular shooting on gesture action recognition can be reduced.

Description

Technical Field

[0001] The present application relates to the field of vehicle air conditioning control, and more specifically, to a vehicle air conditioning control method, device, electronic device and storage medium. Background Art

[0002] In the era of rapid development of intelligence, there are more and more auxiliary operations in the car, but the related operations are also becoming more and more complicated. When changing the auxiliary operations in the car, the driver's line of sight will inevitably deviate from the driving route, which is prone to traffic accidents. In order to simplify the auxiliary operations of related systems in the car and prevent the driver's line of sight from deviating from the driving route. At present, the technical solutions for vehicle air conditioning control include the following methods:

[0003] 1. Traditional human-computer interaction. The driver manually adjusts the knobs or physical buttons on the air conditioning control panel on the central console, and adjusts and sets the parameters of the vehicle air conditioning according to the parameters on the vehicle display, including temperature, wind speed, wind direction, etc. However, this technical solution is not conducive to the driver's high concentration during driving. The driver needs to check the operation interface to operate related functions, and his sight will inevitably deviate from the driving route, which is not conducive to the driver's safe driving.

[0004] 2. Voice human-computer interaction. The driver recognizes voice commands through the built-in microphone of the vehicle system, and the voice processing unit parses and understands the voice commands and converts them into corresponding control commands to adjust the vehicle air conditioning system. However, this technical solution is limited by the noise level of the driving environment and the differences in the driver's accent and dialect, and the recognition effect is not good.

[0005] 3. Gesture human-computer interaction. The 3D depth-sensing camera installed in the car is used to obtain the three-dimensional depth data in the car, and then transmitted to the vehicle system for calculation, processing and recognition. The three-dimensional depth data in the car obtained by the 3D depth-sensing camera can simplify the calculation processing of gesture recognition, thereby reducing the computing power required for the vehicle system to process images and recognize gestures. However, multi-module cameras are expensive and difficult to popularize. Summary of the invention

[0006] The purpose of the embodiments of the present application is to provide a vehicle air-conditioning control method, device, electronic device and storage medium, which are used to recognize user gestures based on image frames generated by a monocular camera, and control the vehicle air-conditioning based on the user gestures. Compared with the prior art, the present application uses image frames generated by a monocular camera for recognition while realizing the control of the vehicle air-conditioning based on gestures, without the need for multiple cameras to image, thereby at least overcoming the technical defect of high cost in the prior art, thereby having the advantage of low cost. On the other hand, the present application can reduce the influence of the lighting environment of the monocular camera on gesture recognition through image quality enhancement processing.

[0007] In a first aspect, the present invention provides a vehicle air conditioning control method, the method comprising:

[0008] Acquire an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture a user's gesture action appearing in front of the vehicle-mounted screen;

[0009] Determining a quality index value of the image frame, wherein the quality index value of the image frame represents a lighting quality of the image frame;

[0010] When the quality index value of the image frame is less than a first preset threshold, performing quality enhancement processing on the image frame so that the quality index value of the image frame is greater than the first preset threshold;

[0011] When the number of image frames that meet the condition is greater than or equal to a second preset threshold, identifying the user gesture action based on the image frames that meet the condition and the internal and external parameters of the monocular camera, wherein the image frames that meet the condition are image frames whose quality index values ​​are greater than the first preset threshold;

[0012] The vehicle air conditioner is controlled based on the user gesture action.

[0013] The method of the present application obtains an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture the user's gesture action that appears in front of the vehicle screen, and then the quality index value of the image frame can be determined, wherein the quality index value of the image frame represents the illumination quality of the image frame, and then when the quality index value of the image frame is less than a first preset threshold, the image frame can be quality enhanced so that the quality index value of the image frame is greater than the first preset threshold, and then when the number of image frames that meet the conditions is greater than or equal to a second preset threshold, the user gesture action can be identified based on the image frame that meets the conditions and the internal and external parameters of the monocular camera, wherein the image frame that meets the conditions is an image frame whose quality index value is greater than the first preset threshold, so that the vehicle air conditioner can be controlled based on the user gesture action, and finally, the user gesture action is identified based on the image frame generated by the monocular camera, and the vehicle air conditioner is controlled based on the user gesture action. Compared with the prior art, the present application uses the image frame generated by the monocular camera for identification while realizing the control of the vehicle air conditioner based on gesture action, and does not require multiple cameras for imaging, thereby overcoming the technical defect of high cost in the prior art, and thus has the advantage of low cost. On the other hand, the present application can reduce the impact of the lighting environment of monocular camera on gesture action recognition through image quality enhancement processing.

[0014] In an optional implementation, after the image frame is enhanced and before the user gesture is recognized based on the image frame that meets the conditions and the internal and external parameters of the monocular camera, the method further includes:

[0015] It is determined whether the quality index value of the image frame is still less than the first preset threshold value. If the quality index value of the image frame is still less than the first preset threshold value, the image frame is discarded.

[0016] This optional implementation manner determines whether the quality index value of the image frame is still less than the first preset threshold, and can discard the image frame when the quality index value of the image frame is still less than the first preset threshold, thereby reducing the difficulty of data processing for complex scenes.

[0017] In an optional implementation, the identifying the user gesture based on the image frame that meets the condition and the internal and external parameters of the monocular camera includes:

[0018] Using the image frame that meets the conditions as an input of a hand region detection model to identify a pre-selected hand region based on the hand region detection model;

[0019] determining a target hand area based on the pre-selected hand area;

[0020] Using the target hand area as an input of a gesture key point detection model to identify target gesture key points based on the gesture key point detection model;

[0021] The user gesture action is recognized based on the target gesture key points.

[0022] This optional implementation can use the image frame that meets the conditions as the input of the hand area detection model, so as to identify the pre-selected hand area based on the hand area detection model, and then determine the target hand area based on the pre-selected hand area, and then use the target hand area as the input of the gesture key point detection model, so as to identify the target gesture key points based on the gesture key point detection model, and then recognize the user gesture action based on the target gesture key points.

[0023] In an optional embodiment, the hand area detection model is a YOLOv7 algorithm model, wherein the YOLOv7 algorithm model is used to extract medium target features and large target features in the image frame based on an attention mechanism.

[0024] This optional implementation can identify the hand area based on the YOLOv7 algorithm model, and can introduce the attention mechanism into the YOLOv7 algorithm model, so that the YOLOv7 algorithm model pays more attention to the medium target features and large target features in the image frame, thereby improving its own recognition accuracy of the hand area.

[0025] In an optional implementation, determining the target hand area based on the preliminary hand area includes:

[0026] Determine the size information of the preliminary human head area and the distance from the preliminary human head area to the center of the monocular camera;

[0027] Whether the preliminary candidate hand area is the target hand area is determined based on the size information of the preliminary candidate hand area and the distance from the preliminary candidate hand area to the center of the monocular camera.

[0028] This optional implementation manner determines the size information of the preliminary contestant hand area and the distance from the preliminary contestant hand area to the center of the monocular camera, and then can determine whether the preliminary contestant hand area is the target hand area based on the size information of the preliminary contestant hand area and the distance from the preliminary contestant hand area to the center of the monocular camera.

[0029] In an optional implementation, the gesture key point detection model is a HandUNet network model, wherein the HandUNet network model is used to identify 21 gesture key points from the target hand area.

[0030] This optional implementation may use the HandUNet network model as a gesture key point detection model, and then identify 21 gesture keys from the target hand area based on the HandUNet network model.

[0031] In an optional implementation, the identifying the user gesture action based on the target gesture key point includes:

[0032] Determine the coordinates of each of the 21 gesture key points and the angle information of the fingers;

[0033] The user gesture action is identified based on the coordinates of each of the gesture key points and the angle information of the finger.

[0034] This optional implementation determines the coordinates of each of the 21 gesture key points and the angle information of the finger, and can then identify the user gesture action based on the coordinates of each of the gesture key points and the angle information of the finger.

[0035] In a second aspect, the present invention provides a vehicle air conditioning control device, the device comprising:

[0036] An acquisition module, used to acquire an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture a user's gesture action appearing in front of the vehicle-mounted screen;

[0037] A determination module, configured to determine a quality index value of the image frame, wherein the quality index value of the image frame represents a lighting quality of the image frame;

[0038] An image enhancement processing module, configured to perform quality enhancement processing on the image frame when the quality index value of the image frame is less than a first preset threshold value, so that the quality index value of the image frame is greater than the first preset threshold value;

[0039] a gesture recognition module, configured to recognize user gestures based on the image frames meeting the conditions and the internal and external parameters of the monocular camera when the number of image frames meeting the conditions is greater than or equal to a second preset threshold, wherein the image frames meeting the conditions are image frames whose quality index values ​​are greater than the first preset threshold;

[0040] A control module is used to control the vehicle air conditioner based on the user gesture action.

[0041] The device of the present application executes a vehicle air conditioning control method, and can obtain an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture user gestures that appear in front of the vehicle screen, and then the quality index value of the image frame can be determined, wherein the quality index value of the image frame represents the lighting quality of the image frame, and then when the quality index value of the image frame is less than a first preset threshold, the image frame can be quality enhanced so that the quality index value of the image frame is greater than the first preset threshold, and then when the number of image frames that meet the conditions is greater than or equal to a second preset threshold, the user gesture can be identified based on the image frames that meet the conditions and the internal and external parameters of the monocular camera, wherein the image frames that meet the conditions are image frames whose quality index values ​​are greater than the first preset threshold, so that the vehicle air conditioner can be controlled based on the user gesture, and finally, the user gesture is identified based on the image frame generated by the monocular camera, and the vehicle air conditioner is controlled based on the user gesture. Compared with the prior art, the present application uses the image frame generated by the monocular camera for identification while realizing the control of the vehicle air conditioner based on gestures, and does not require multiple cameras for imaging, thereby overcoming the technical defect of high cost in the prior art, and thus has the advantage of low cost. On the other hand, the present application can reduce the impact of the lighting environment of monocular camera on gesture action recognition through image quality enhancement processing.

[0042] In a third aspect, the present invention provides an electronic device, comprising:

[0043] Processor; and

[0044] The memory is configured to store machine-readable instructions, and when the instructions are executed by the processor, the vehicle air conditioning control method as described in any one of the aforementioned implementations is executed.

[0045] The electronic device of the present application can obtain image frames by executing a vehicle air-conditioning control method, wherein the image frames are generated by a monocular camera, and the monocular camera is used to capture user gestures that appear in front of a vehicle screen, and then the quality index value of the image frames can be determined, wherein the quality index value of the image frames characterizes the lighting quality of the image frames, and then when the quality index value of the image frames is less than a first preset threshold, the image frames can be quality enhanced so that the quality index value of the image frames is greater than the first preset threshold, and then when the number of image frames that meet the conditions is greater than or equal to a second preset threshold, based on the image frames that meet the conditions and the internal and external parameters of the monocular camera, the user gestures can be identified, wherein the image frames that meet the conditions are image frames whose quality index values ​​are greater than the first preset threshold, so that the vehicle air-conditioning can be controlled based on the user gestures, and finally, the user gestures are identified based on the image frames generated by the monocular camera, and the vehicle air-conditioning is controlled based on the user gestures. Compared with the prior art, the present application uses the image frames generated by a monocular camera for recognition while realizing the control of the vehicle air conditioner based on gesture actions, without the need for multiple cameras for imaging, thereby overcoming the technical defect of the prior art of high cost, thus having the advantage of low cost. On the other hand, the present application can reduce the influence of the lighting environment of the monocular camera on the recognition of gesture actions through image quality enhancement processing.

[0046] In a fourth aspect, the present invention provides a storage medium storing a computer program, wherein the computer program is executed by a processor as the vehicle air conditioning control method as described in any one of the aforementioned embodiments.

[0047] The storage medium of the present application can obtain image frames by executing a vehicle air-conditioning control method, wherein the image frames are generated by a monocular camera, and the monocular camera is used to capture user gestures that appear in front of a vehicle screen, so as to determine a quality index value of the image frames, wherein the quality index value of the image frames characterizes the lighting quality of the image frames, and then when the quality index value of the image frames is less than a first preset threshold, the image frames can be quality enhanced so that the quality index value of the image frames is greater than the first preset threshold, and then when the number of image frames that meet the conditions is greater than or equal to a second preset threshold, based on the image frames that meet the conditions and the internal and external parameters of the monocular camera, the user gestures can be identified, wherein the image frames that meet the conditions are image frames whose quality index values ​​are greater than the first preset threshold, so that the vehicle air-conditioning can be controlled based on the user gestures, and finally, the user gestures are identified based on the image frames generated by the monocular camera, and the vehicle air-conditioning is controlled based on the user gestures. Compared with the prior art, the present application uses the image frames generated by a monocular camera for recognition while realizing the control of the vehicle air conditioner based on gesture actions, without the need for multiple cameras for imaging, thereby overcoming the technical defect of the prior art of high cost, thus having the advantage of low cost. On the other hand, the present application can reduce the influence of the lighting environment of the monocular camera on the recognition of gesture actions through image quality enhancement processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 It is a flow chart of a vehicle air conditioning control method disclosed in an embodiment of the present application;

[0050] Figure 2 is a structural schematic diagram of a vehicle air conditioning control device disclosed in an embodiment of the present application;

[0051] Figure 3 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0053] Embodiment 1

[0054] See also Figure 1 , Figure 1 is a flow chart of a vehicle air conditioning control method disclosed in an embodiment of the present application, such as Figure 1 As shown, the method of the embodiment of the present application includes the following steps:

[0055] 101. Acquire an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture a user's gesture action appearing in front of the vehicle screen;

[0056] 102. Determine a quality index value of the image frame, wherein the quality index value of the image frame represents the illumination quality of the image frame;

[0057] 103. When the quality index value of the image frame is less than a first preset threshold, perform quality enhancement processing on the image frame so that the quality index value of the image frame is greater than the first preset threshold;

[0058] 104. When the number of image frames that meet the condition is greater than or equal to the second preset threshold, identifying the user gesture based on the image frames that meet the condition and the internal and external parameters of the monocular camera, wherein the image frames that meet the condition are image frames whose quality index values ​​are greater than the first preset threshold;

[0059] 105. Control the vehicle air conditioner based on user gestures.

[0060] The method of the embodiment of the present application obtains an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture the user's gesture action that appears in front of the vehicle screen, and then the quality index value of the image frame can be determined, wherein the quality index value of the image frame represents the lighting quality of the image frame, and then when the quality index value of the image frame is less than the first preset threshold, the image frame can be quality enhanced so that the quality index value of the image frame is greater than the first preset threshold, and then when the number of image frames that meet the conditions is greater than or equal to the second preset threshold, the user gesture action can be identified based on the image frames that meet the conditions and the internal and external parameters of the monocular camera, wherein the image frames that meet the conditions are image frames whose quality index value is greater than the first preset threshold, so that the vehicle air conditioner can be controlled based on the user gesture action, and finally, the user gesture action is identified based on the image frame generated by the monocular camera, and the vehicle air conditioner is controlled based on the user gesture action. Compared with the prior art, the present application uses the image frame generated by the monocular camera for recognition while realizing the control of the vehicle air conditioner based on the gesture action, and does not require multiple cameras for imaging, thereby overcoming the technical defect of the high cost in the prior art, and thus has the advantage of low cost. On the other hand, the present application can reduce the impact of the lighting environment of monocular camera on gesture action recognition through image quality enhancement processing.

[0061] In the embodiment of the present application, a monocular camera refers to a camera device with only one lens, which is lower in cost than the multi-camera used in the prior art. Furthermore, the prior art does not use a monocular camera because the prior art obtains more images, including multi-angle imaging, to ensure the accuracy of gesture recognition, wherein multi-angle imaging is conducive to improving the accuracy of gesture recognition, but its cost is higher.

[0062] In the embodiment of the present application, an image frame refers to a static image formed by a monocular camera at a certain moment, wherein the monocular camera is aimed at the front and back of the vehicle-mounted screen, and will capture the user's gestures appearing in front of the vehicle-mounted screen in real time to form multiple image frames.

[0063] In the embodiment of the present application, the quality index values ​​of the image frame include image correlation loss, brightness distortion, and contrast distortion.

[0064] In an embodiment of the present application, the first preset threshold refers to the comparison value of the quality index value of the image frame, wherein, if the quality index value of the image frame is less than the first preset threshold, it means that the image quality of the image frame is not very good, which may affect subsequent steps such as hand area recognition and gesture key point recognition. Based on this, by enhancing such image frames, the impact of such image frames on hand area recognition and gesture key point recognition can be reduced, thereby improving the accuracy of hand area recognition and gesture key point recognition.

[0065] In the embodiment of the present application, since the user gesture is continuous and the continuous user gesture is reflected by multiple image frames, the recognition of the user gesture must be based on multiple image frames. Furthermore, in general, the user gesture can be completely recorded by 30 image frames, so the second preset threshold can refer to the value of 30, that is, when the number of image frames that meet the conditions is greater than or equal to 30, the user gesture is recognized based on the image frames that meet the conditions and the internal and external parameters of the monocular camera.

[0066] In an embodiment of the present application, the internal parameters of a monocular camera refer to parameters related to the characteristics of the camera itself, including focal length, principal point offset, and pixel scale factor, etc. These parameters are usually obtained through a camera calibration process and are used to convert points in an image coordinate system into a camera coordinate system. On the other hand, the external parameters of a monocular camera refer to parameters that describe the position and posture of the camera relative to a world coordinate system, mainly including a rotation matrix and a translation vector. The external parameters reflect the specific position and direction of the camera in space, which is very important for multi-view geometry calculations. In an embodiment of the present application, as an optional implementation, after the image frame is enhanced, based on the internal and external parameters of the image frame that meets the conditions and the monocular camera, before recognizing the user's gesture action, the method of the embodiment of the present application also includes the following steps:

[0067] It is determined whether the quality index value of the image frame is still less than the first preset threshold value. If the quality index value of the image frame is still less than the first preset threshold value, the image frame is discarded.

[0068] This optional implementation method can discard the image frame by judging whether the quality index value of the image frame is still less than the first preset threshold value, thereby reducing the difficulty of data processing for complex scenes.

[0069] For this optional implementation, after the image frame is quality enhanced, its quality index value may still be less than the first preset threshold. In the case where the quality index value is still less than the first preset threshold, the image frame is discarded to avoid using complex processing algorithms to further process the image frame, thereby reducing the difficulty of data processing in complex scenes.

[0070] In the embodiment of the present application, as an optional implementation, based on the image frames that meet the conditions and the internal and external parameters of the monocular camera, the user gesture action is recognized, including the following sub-steps:

[0071] Using the image frame that meets the conditions as the input of the hand region detection model to identify the pre-selected hand region based on the hand region detection model;

[0072] Determining a target hand area based on the pre-selected hand area;

[0073] Taking the target hand region as an input of the gesture key point detection model to identify the target gesture key points based on the gesture key point detection model;

[0074] Recognize user gestures based on target gesture key points.

[0075] This optional implementation can use image frames that meet the conditions as the input of the hand area detection model, so as to identify the pre-selected hand area based on the hand area detection model, and then determine the target hand area based on the pre-selected hand area, and then use the target hand area as the input of the gesture key point detection model, so as to identify the target gesture key points based on the gesture key point detection model, and thus recognize the user gesture action based on the target gesture key points.

[0076] In the above optional implementation, the hand area detection model usually outputs several image frames, wherein the image frames mark the areas in the image where gestures may exist. These areas are temporarily determined to have gestures, and are therefore called preliminary hand areas, while the areas where gestures have been determined to exist are called target gesture areas.

[0077] In the above optional implementation manner, the user gesture action can be determined by the geometric relationship formed by the gesture key points.

[0078] In an embodiment of the present application, in an optional implementation, the hand area detection model is a YOLOv7 algorithm model, wherein the YOLOv7 algorithm model is used to extract medium target features and large target features in the image frame based on the attention mechanism.

[0079] This optional implementation can identify the hand area based on the YOLOv7 algorithm model, and can introduce the attention mechanism into the YOLOv7 algorithm model, so that the YOLOv7 algorithm model pays more attention to the medium target features and large target features in the image frame, thereby improving its own recognition accuracy of the hand area.

[0080] In this optional embodiment, "YOLOv7 algorithm model" refers to an advanced real-time target detection algorithm. The YOLO (You Only Look Once) series of algorithms are a very famous class of algorithms for object detection in the field of computer vision. YOLOv7 is the seventh version of the series. It maintains the fast detection speed of the YOLO series of algorithms while further improving the detection accuracy and efficiency. YOLOv7 is particularly suitable for target detection application scenarios that require high performance and low latency, such as autonomous driving, video surveillance, etc.

[0081] In the embodiment of the present application, as an optional implementation, determining the target hand area based on the pre-selected hand area includes the following sub-steps:

[0082] Determine the size information of the preliminary facial area and the distance from the preliminary facial area to the center of the monocular camera;

[0083] Based on the size information of the preliminary hand area and the distance from the preliminary hand area to the center of the monocular camera, it is determined whether the preliminary hand area is the target hand area.

[0084] This optional implementation method determines the size information of the pre-selected hand area and the distance from the pre-selected hand area to the center of the monocular camera, and then can determine whether the pre-selected hand area is the target hand area based on the size information of the pre-selected hand area and the distance from the pre-selected hand area to the center of the monocular camera.

[0085] For the above optional implementation, generally, the distance from the pre-selected hand area to the center of the monocular camera is associated with the size information of the pre-selected hand area, wherein the larger the distance from the hand area to the center of the monocular camera, the larger the size of the hand area in theory. In this way, the target gesture area can be screened out from multiple pre-selected hand areas through the size information of the pre-selected hand area and the distance from the pre-selected hand area to the center of the monocular camera. For example, if the distance from the pre-selected hand area to the center of the monocular camera is A, and the size of the pre-selected hand area is B, theoretically, when the distance from the hand area to the center of the monocular camera is A, the size of the hand area should be C, then it can be determined that the pre-selected hand area is not the target gesture area, and if the distance from the hand area to the center of the monocular camera is A, the size of the hand area should be B, then the pre-selected hand area can be determined as the target gesture area.

[0086] In an embodiment of the present application, as an optional implementation, the gesture key point detection model is a HandUNet network model, wherein the HandUNet network model is used to identify 21 gesture key points from the target hand area.

[0087] This optional implementation may use the HandUNet network model as a gesture key point detection model, and then identify 21 gesture keys from the target hand area based on the HandUNet network model.

[0088] For this optional implementation, the "HandUNet network model" is a deep learning model specifically designed for hand recognition or hand posture estimation. This model is usually based on the UNet architecture, which is a convolutional neural network (CNN) commonly used for image segmentation tasks. UNet is characterized by an encoder-decoder structure, in which the encoder is responsible for extracting features from the input image, and the decoder maps these features back to an output of the same size as the input image, thereby achieving pixel-level prediction. HandUNet makes specific adjustments and optimizations based on UNet to better adapt to the processing needs of hand images, such as improving the ability to capture hand details and enhancing the robustness of the model.

[0089] In an embodiment of the present application, as an optional implementation, recognizing a user gesture action based on a target gesture key point includes the following steps:

[0090] Determine the coordinates of each of the 21 gesture key points, as well as the angle information of the fingers;

[0091] Identify user gestures based on the coordinates of each gesture key point and the angle information of the fingers.

[0092] This optional implementation method determines the coordinates of each of the 21 gesture key points and the angle information of the fingers, and can then identify the user's gesture action based on the coordinates of each gesture key point and the angle information of the fingers.

[0093] For the embodiment of the present application, as an example:

[0094] The visual image is obtained through a monocular camera, and each frame of hand data collected is subjected to a low-light test to determine whether low-light image enhancement is required. The evaluation index used for the low-light test is the image universal quality index UQI. (Universal Quality Image Index (UQI): It combines three factors: image correlation loss, brightness distortion, and contrast distortion to reflect the quality of the image.) If the UQI is greater than 0.8, the frame image is directly transmitted to the hand area detection module. If the UQI is less than 0.8, low-light image enhancement is required, and then the enhanced image is judged again. If the UQI is greater than 0.8, it is transmitted to the hand area detection module. If it is still less than 0.8, the frame image is directly discarded. When the number of image frames that pass the low-light test reaches 30 frames, the image and the internal and external parameter data of the monocular camera are input into the hand area detection module.

[0095] Furthermore, after the 30 frames of low-light testing are transmitted to the hand area detection module, the Backbone and Neck in the YOLOv7 model are used to obtain image features. The overall process of the Backbone layer is to pass through 4 CBS layers with a Maxpool layer and then be transmitted to the ELAN module. After that, the output of the three MP+ELAN is connected to increase the grid level, improve the model accuracy, and obtain image features.

[0096] Furthermore, in the Neck layer, the input feature map is encoded along the horizontal and vertical coordinates, and the feature map embedded with the width and height coordinate information is obtained by average pooling. The formulas for the two feature maps are as follows:

[0097]

[0098] Where: x c The feature vector representing the input; Represents the output at height h;

[0099] H represents the height of the initial input feature vector;

[0100] W represents the width of the initial input feature vector.

[0101] Furthermore, the two average pooled feature maps are concatenated and batch normalized to obtain the feature map F1, and then the feature map f is obtained through a nonlinear activation function. The formula of the feature map f is as follows:

[0102] f=δ(F([z h ,z w ]));

[0103] Then the feature map f is split vertically and horizontally into feature maps F with the same number of channels as the original h and F w , and then use the sigmoid activation function to obtain the attention weight information g in the height and width directions respectively h and g w , where the attention weight information g h and g w The formula is as follows:

[0104]

[0105] Finally, the attention weight information g in the height and width directions is h and g w , perform weighted multiplication and output as a feature map y with attention weights c (i,j). The calculation formula is as follows:

[0106]

[0107] After the above steps, the CA attention mechanism module is added to MP-2 in the original Neck layer, so that the network model can pay more attention to medium and large targets, enhance the feature extraction of medium and large targets in the feature map by the YOLOv7 algorithm, and improve the accuracy of hand area detection. Subsequently, all detected hand areas are judged whether they are the driver's hand areas by the size of the target box and the distance to the center of the camera.

[0108] Furthermore, the image of hand area detection by the improved YOLOv7 network model is input into the HandUNet network model for hand key point detection, and 21 hand key points of the operator are detected. Each key point data contains four information: x-axis coordinate, y-axis coordinate, z-axis coordinate and confidence, so the 21 hand key points obtained have a total of 84 information. Then, each finger has 4 key points plus one key point at the base of the palm, and the two adjacent points of each finger are connected, and the points at the base of the finger are connected with the points at the base of the palm. It can be seen that one finger can get 3 angle information, and 5 fingers can collect 15 angle information in total. Gesture label classification is performed based on the angle information and the information of the hand key points. Then, the BiGRU network is fused with the Capsule network to extract the driver's hand key points and angle information to achieve high-precision gesture command recognition.

[0109] Embodiment 2

[0110] See also Figure 2 , Figure 2 is a structural schematic diagram of a vehicle air conditioning control device disclosed in an embodiment of the present application, such as Figure 2 As shown, the device of the embodiment of the present application includes the following functional modules:

[0111] An acquisition module 201 is used to acquire an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture the user's gesture action appearing in front of the vehicle-mounted screen;

[0112] A determination module 202 is used to determine a quality index value of an image frame, wherein the quality index value of the image frame represents a lighting quality of the image frame;

[0113] The image enhancement processing module 203 is used to perform quality enhancement processing on the image frame when the quality index value of the image frame is less than the first preset threshold value, so that the quality index value of the image frame is greater than the first preset threshold value;

[0114] The gesture recognition module 204 is used to recognize the user's gesture action based on the image frames that meet the conditions and the internal and external parameters of the monocular camera when the number of image frames that meet the conditions is greater than or equal to the second preset threshold, wherein the image frames that meet the conditions are image frames whose quality index values ​​are greater than the first preset threshold;

[0115] The control module 205 is used to control the vehicle air conditioner based on user gestures.

[0116] The device of the embodiment of the present application executes the vehicle air conditioning control method, and can obtain an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture the user's gesture action that appears in front of the vehicle screen, and then the quality index value of the image frame can be determined, wherein the quality index value of the image frame represents the lighting quality of the image frame, and then when the quality index value of the image frame is less than the first preset threshold, the image frame can be quality enhanced so that the quality index value of the image frame is greater than the first preset threshold, and then when the number of image frames that meet the conditions is greater than or equal to the second preset threshold, the user gesture action can be identified based on the image frames that meet the conditions and the internal and external parameters of the monocular camera, wherein the image frames that meet the conditions are image frames whose quality index value is greater than the first preset threshold, so that the vehicle air conditioner can be controlled based on the user gesture action, and finally, the user gesture action is identified based on the image frame generated by the monocular camera, and the vehicle air conditioner is controlled based on the user gesture action. Compared with the prior art, the present application uses the image frame generated by the monocular camera for recognition while realizing the control of the vehicle air conditioner based on the gesture action, and does not require multiple cameras for imaging, thereby overcoming the technical defect of high cost in the prior art, and thus has the advantage of low cost. On the other hand, the present application can reduce the impact of the lighting environment of monocular camera on gesture action recognition through image quality enhancement processing.

[0117] It should be noted that for other detailed descriptions of the device in the embodiment of the present application, please refer to the relevant description of the embodiment of the present application, and the embodiment of the present application will not go into details therein.

[0118] Embodiment 3

[0119] See also Figure 3 , Figure 3 is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application, such as Figure 3 As shown, the electronic device of the embodiment of the present application includes:

[0120] Processor 301; and

[0121] The memory 302 is configured to store machine-readable instructions, and when the instructions are executed by the processor 301, the vehicle air conditioning control method as described in any of the aforementioned implementations is executed.

[0122] The electronic device of the embodiment of the present application can obtain an image frame by executing the vehicle air conditioning control method, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture the user's gesture action that appears in front of the vehicle screen, and then the quality index value of the image frame can be determined, wherein the quality index value of the image frame represents the lighting quality of the image frame, and then when the quality index value of the image frame is less than the first preset threshold, the image frame can be quality enhanced so that the quality index value of the image frame is greater than the first preset threshold, and then when the number of image frames that meet the conditions is greater than or equal to the second preset threshold, the user gesture action can be identified based on the image frames that meet the conditions and the internal and external parameters of the monocular camera, wherein the image frames that meet the conditions are image frames whose quality index value is greater than the first preset threshold, so that the vehicle air conditioning can be controlled based on the user gesture action, and finally, the user gesture action is identified based on the image frame generated by the monocular camera, and the vehicle air conditioning is controlled based on the user gesture action. Compared with the prior art, the present application uses the image frame generated by the monocular camera for recognition while realizing the control of the vehicle air conditioning based on the gesture action, and does not require multiple cameras for imaging, thereby overcoming the technical defect of high cost in the prior art, and thus has the advantage of low cost. On the other hand, the present application can reduce the impact of the lighting environment of monocular camera on gesture action recognition through image quality enhancement processing.

[0123] Embodiment 4

[0124] An embodiment of the present application provides a storage medium, which stores a computer program. The computer program is executed by a processor as a vehicle air conditioning control method as described in any of the aforementioned implementations.

[0125] The storage medium of the embodiment of the present application can obtain an image frame by executing the vehicle air conditioning control method, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture the user's gesture action that appears in front of the vehicle screen, and then the quality index value of the image frame can be determined, wherein the quality index value of the image frame represents the lighting quality of the image frame, and then when the quality index value of the image frame is less than the first preset threshold, the image frame can be quality enhanced so that the quality index value of the image frame is greater than the first preset threshold, and then when the number of image frames that meet the conditions is greater than or equal to the second preset threshold, the user gesture action can be identified based on the image frames that meet the conditions and the internal and external parameters of the monocular camera, wherein the image frames that meet the conditions are image frames whose quality index value is greater than the first preset threshold, so that the vehicle air conditioning can be controlled based on the user gesture action, and finally, the user gesture action is identified based on the image frame generated by the monocular camera, and the vehicle air conditioning is controlled based on the user gesture action. Compared with the prior art, the present application uses the image frame generated by the monocular camera for recognition while realizing the control of the vehicle air conditioning based on the gesture action, and does not require multiple cameras for imaging, thereby overcoming the technical defect of the high cost in the prior art, and thus has the advantage of low cost. On the other hand, the present application can reduce the impact of the lighting environment of monocular camera on gesture action recognition through image quality enhancement processing.

[0126] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0127] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0129] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM) random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0130] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0131] The above are only embodiments of the present application and are not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A vehicle air conditioning control method, characterized in that: The method comprises: Acquire an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture a user's gesture action appearing in front of the vehicle-mounted screen; Determining a quality index value of the image frame, wherein the quality index value of the image frame represents a lighting quality of the image frame; When the quality index value of the image frame is less than a first preset threshold, performing quality enhancement processing on the image frame so that the quality index value of the image frame is greater than the first preset threshold; When the number of image frames that meet the condition is greater than or equal to a second preset threshold, identifying the user gesture action based on the image frames that meet the condition and the internal and external parameters of the monocular camera, wherein the image frames that meet the condition are image frames whose quality index values ​​are greater than the first preset threshold; The vehicle air conditioner is controlled based on the user gesture action.

2. The method according to claim 1, characterized in that After the image frame is enhanced, and before the user gesture is recognized based on the image frame that meets the conditions and the internal and external parameters of the monocular camera, the method further includes: It is determined whether the quality index value of the image frame is still less than the first preset threshold value. If the quality index value of the image frame is still less than the first preset threshold value, the image frame is discarded.

3. The method according to claim 1, characterized in that The identifying the user gesture action based on the image frame that meets the conditions and the internal and external parameters of the monocular camera includes: Using the image frame that meets the conditions as an input of a hand region detection model to identify a pre-selected hand region based on the hand region detection model; determining a target hand area based on the pre-selected hand area; Using the target hand area as an input of a gesture key point detection model to identify target gesture key points based on the gesture key point detection model; The user gesture action is recognized based on the target gesture key points.

4. The method according to claim 3, characterized in that The hand area detection model is a YOLOv7 algorithm model, wherein the YOLOv7 algorithm model is used to extract medium target features and large target features in the image frame based on an attention mechanism.

5. The method according to claim 4, characterized in that The determining of the target hand area based on the pre-selected hand area includes: Determine the size information of the preliminary human head area and the distance from the preliminary human head area to the center of the monocular camera; Whether the preliminary candidate hand area is the target hand area is determined based on the size information of the preliminary candidate hand area and the distance from the preliminary candidate hand area to the center of the monocular camera.

6. The method according to claim 3, characterized in that The gesture key point detection model is a HandUNet network model, wherein the HandUNet network model is used to identify 21 gesture key points from the target hand area.

7. The method according to claim 6, characterized in that The identifying the user gesture action based on the target gesture key point includes: Determine the coordinates of each of the 21 gesture key points and the angle information of the fingers; The user gesture action is identified based on the coordinates of each of the gesture key points and the angle information of the finger.

8. A vehicle air conditioning control device, characterized in that: The device comprises: An acquisition module, used to acquire an image frame, wherein the image frame is generated by a monocular camera, and the monocular camera is used to capture a user's gesture action appearing in front of the vehicle-mounted screen; A determination module, configured to determine a quality index value of the image frame, wherein the quality index value of the image frame represents a lighting quality of the image frame; An image enhancement processing module, configured to perform quality enhancement processing on the image frame when the quality index value of the image frame is less than a first preset threshold value, so that the quality index value of the image frame is greater than the first preset threshold value; a gesture recognition module, configured to recognize user gestures based on the image frames meeting the conditions and the internal and external parameters of the monocular camera when the number of image frames meeting the conditions is greater than or equal to a second preset threshold, wherein the image frames meeting the conditions are image frames whose quality index values ​​are greater than the first preset threshold; A control module is used to control the vehicle air conditioner based on the user gesture action.

9. An electronic device, characterized in that: include: processor; as well as A memory configured to store machine-readable instructions, which, when executed by the processor, execute the vehicle air conditioning control method as described in any one of claims 1-7.

10. A storage medium, characterized in that: The storage medium stores a computer program, and the computer program is executed by a processor to implement the vehicle air conditioning control method according to any one of claims 1 to 7.

Images