Control method and device of vehicle-mounted air conditioner, vehicle and computer readable storage medium
Through the fusion of facial and voice data, the operating parameters of the on-board air conditioner are automatically adjusted, which solves the problem of manual adjustment of the on-board air conditioner and improves the user experience and temperature perception accuracy.
Patent Information
- Application Number
- CN202310601100.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2024-11-26
AI Technical Summary
After the on-board air conditioner is running, the user needs to manually adjust the cooling/heating/air supply effect is inconvenient, and manual adjustment in the driving state may affect driving safety.
By obtaining facial data and voice data, the user's temperature perception results of the operation effect of the on-board air conditioner are identified, and the on-board air conditioner operation parameters are automatically adjusted after integration.
It realizes automatic adjustment of the operating parameters of the on-board air conditioner, improves the user experience, improves the credibility and accuracy of the temperature perception results, and avoids the inconvenience of manual adjustment by users.
Smart Images

Figure CN119017891A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of vehicle technology, and more specifically, to a method for controlling a vehicle air conditioner, a control device for a vehicle air conditioner, a vehicle, and a computer-readable storage medium. Background Art
[0002] In the related art, after the car air conditioner is started and running, if the user wants to change the cooling / heating / air supply effect, the user needs to manually adjust the operating parameters of the car air conditioner (such as wind speed, cooling temperature and heating temperature, etc.), which is very inconvenient. Summary of the invention
[0003] Embodiments of the present application provide a method for controlling a vehicle air conditioner, a control device for the vehicle air conditioner, a vehicle, and a computer-readable storage medium.
[0004] The present application provides a method for controlling a vehicle air conditioner, including:
[0005] Obtain facial data and voice data of the target object;
[0006] Identifying, based on the facial data, a first temperature perception result of the target object regarding the current operating effect of the vehicle air conditioner;
[0007] identifying, according to the voice data, a second temperature perception result of the target object regarding the current operating effect of the vehicle air conditioner;
[0008] Determine, according to the first temperature sensing result and the second temperature sensing result, a temperature sensing recognition result of the target object corresponding to the current operating effect of the vehicle air conditioner;
[0009] The vehicle air conditioner is controlled to change operating parameters according to the temperature sensing and recognition result.
[0010] In the control method of the vehicle air conditioner of the present application, the facial data and voice data of the target object are first obtained; then, based on the facial data, the first temperature perception result of the target object for the current operating effect of the vehicle air conditioner is identified, that is, the user's feelings about the in-vehicle environment, such as comfort, fear of cold, heat, etc., are judged through the facial data; at the same time, the second temperature perception result of the target object for the current operating effect of the vehicle air conditioner is identified based on the voice data, that is, the user's feelings about the in-vehicle environment are judged through the user's voice; then, the two temperature perception results are merged to obtain the final detection result, that is, the temperature perception recognition result of the target object corresponding to the current operating effect of the vehicle air conditioner is determined based on the first temperature perception result and the second temperature perception result; finally, the vehicle air conditioner is controlled to change the operating parameters according to the temperature perception recognition result.
[0011] In this way, the implementation mode of the present application enables the operating parameters of the vehicle air conditioner to be automatically adjusted based on the detected temperature perception recognition results, avoiding the situation where the user needs to make manual adjustments, thereby improving the user experience; at the same time, because the user's facial data and voice data are simultaneously used to determine the user's temperature perception results of the current operating effect of the vehicle air conditioner, the credibility of the temperature perception results is relatively high, and thus the vehicle air conditioner operating parameters can be accurately and effectively controlled to change.
[0012] In some embodiments, identifying, based on the facial data, a first temperature perception result of the target object for the current operating effect of the vehicle air conditioner includes:
[0013] The facial data is input into a preset graph network model to obtain the first temperature perception result.
[0014] In this way, the implementation manner of the present application predicts the first temperature perception result based on the graph network model, so that the credibility and accuracy of the first temperature perception result are relatively high.
[0015] In some embodiments, inputting the facial data into a preset graph network model to obtain the first temperature perception result includes:
[0016] Inputting partial facial data obtained based on different shooting angles in the facial data into corresponding branch networks in the graph network model respectively, and obtaining reference temperature perception results corresponding to each partial facial data, wherein each branch network is used to receive the corresponding partial facial data and generate the reference temperature perception results corresponding to the partial facial data;
[0017] The first temperature sensing result is obtained according to the reference temperature sensing result corresponding to each of the partial facial data.
[0018] In this way, the implementation mode of the present application enables the acquisition of the first temperature perception result based on partial facial data from different shooting angles, thereby avoiding the situation where the facial data obtained from a single shooting angle cannot truly reflect the user's expression / facial details, thereby ensuring the credibility of the first temperature perception result; at the same time, because the partial facial data from each shooting angle will output the corresponding reference temperature perception result based on the corresponding branch network, the accuracy of each reference temperature perception result is relatively high, thereby further improving the credibility of the first temperature perception result composed of multiple reference temperature perception results.
[0019] In some embodiments, before inputting the partial facial data obtained based on different shooting angles in the facial data into the corresponding branch networks in the graph network model to obtain the reference temperature perception results corresponding to the partial facial data, the method further includes:
[0020] Using the partial facial data corresponding to the branch network as a training sample, and obtaining a data enhancement sample according to at least one of the other partial facial data;
[0021] The branch network is trained according to the training samples and data enhancement samples of the branch network.
[0022] In this way, the implementation method of the present application is based on different partial facial data, so that the number of samples used in the training of each branch network is guaranteed, thereby avoiding the occurrence of underfitting during the training of each branch network, and therefore, the robustness of each branch network is guaranteed.
[0023] In some embodiments, inputting the facial data into a preset graph network model to obtain the first temperature perception result includes:
[0024] Performing frame processing on the facial data to obtain multiple frames of facial images;
[0025] Identify key points of the face in the facial image to obtain key point data corresponding to the facial image;
[0026] The key point data corresponding to each frame of the facial image is input into the graph network model to obtain the first temperature perception result.
[0027] In this way, the implementation mode of the present application predicts the first temperature perception result based on the key points of the face, so that the prediction of the first temperature perception result can avoid the interference of background factors in the facial image and reduce the computing resources required for the prediction. Thus, the efficiency of obtaining the first temperature perception result is improved.
[0028] In some embodiments, the step of inputting the facial data into a preset graph network model to obtain the first temperature perception result further includes:
[0029] Normalizing the key point data to obtain processed key point data;
[0030] The step of inputting key point data corresponding to each frame of the facial image into the graph network model to obtain the first temperature perception result includes:
[0031] The processed key point data corresponding to each frame of the facial image is input into the graph network model to obtain the first temperature perception result.
[0032] In this way, the implementation mode of the present application is based on the normalization processing of the key point data, so that the data volume of the key point data can be reduced, and thus fewer computing resources are required to predict the first temperature perception result, thereby further improving the efficiency of obtaining the first temperature perception result.
[0033] In some embodiments, the step of inputting the facial data into a preset graph network model to obtain the first temperature perception result further includes:
[0034] Generate a relationship graph according to the connection relationship between the same two face key points in the key point data corresponding to the adjacent facial images and the connection relationship between the adjacent face key points in the key point data corresponding to each frame of the facial image;
[0035] The step of inputting key point data corresponding to each frame of the facial image into the graph network model to obtain the first temperature perception result includes:
[0036] The relationship graph is input into multiple graph convolution layers of the graph network model to obtain the first temperature perception result.
[0037] In this way, the present application is based on the relationship graph and the graph convolution layer, so that the feature information corresponding to the connection relationship of each key point in multiple frames of facial images can be used to predict the first temperature perception result, so the credibility of the first temperature perception result can be further improved.
[0038] In some embodiments, inputting the relationship graph into a plurality of graph convolution layers of the graph network model to obtain the first temperature sensing result includes:
[0039] Inputting the relationship graph into a plurality of graph convolutional layers sequentially connected in the graph network model;
[0040] For each of the graph convolution layers, processing the received input data according to the graph convolution layer to obtain output data, and sending the input data and the output data to a subsequent graph convolution layer of the graph convolution layer;
[0041] The first temperature sensing result is obtained according to the output data of the last graph convolution layer.
[0042] In this way, the implementation method of the present application enables the latter graph convolution layer of the two preceding and succeeding graph convolution layers to complete corresponding operations based on the input and output of the previous graph convolution layer, so that the gradient vanishing phenomenon in the graph network model can be improved, thereby ensuring the robust operation of the graph network model.
[0043] In some embodiments, identifying, based on the voice data, a second temperature perception result of the target object for the current operating effect of the vehicle air conditioner includes:
[0044] Inputting the voice data into a preset recognition network model to obtain the second temperature perception result;
[0045] The determining, according to the first temperature sensing result and the second temperature sensing result, a temperature sensing recognition result of the target object corresponding to the current operating effect of the vehicle air conditioner includes:
[0046] The second temperature sensing result and the first temperature sensing result are input into a fully connected layer network to obtain the temperature sensing recognition result.
[0047] In this way, the implementation mode of the present application enables both the first temperature perception result and the second temperature perception result to be predicted based on the pre-trained neural network model, so that the accuracy of the first temperature perception result and the second temperature perception result can be guaranteed; at the same time, because the first temperature perception result and the second temperature perception result will be fused based on the fully connected network, the credibility of the fused temperature perception recognition result can also be guaranteed.
[0048] In some embodiments, the step of inputting the second temperature perception result and the first temperature perception result into a fully connected layer network to obtain the temperature perception recognition result includes:
[0049] Inputting the second temperature sensing result and the first temperature sensing result into the fully connected layer network;
[0050] For each fully connected unit of the fully connected layer network, processing the received data according to the fully connected unit to generate a probability value, and inputting the probability value to the next fully connected unit, wherein the fully connected layer network includes a plurality of the fully connected units connected in sequence, and the fully connected unit includes at least one fully connected layer;
[0051] The temperature perception recognition result is obtained according to the probability value of the last fully connected unit.
[0052] In this way, the implementation mode of the present application is based on the setting of a fully connected layer network, so that the first temperature perception result and the second temperature perception result can make full use of the prediction results corresponding to the facial data and the prediction results corresponding to the sound data when they are fused, thereby ensuring the credibility of the temperature perception recognition results.
[0053] In some embodiments, identifying, based on the voice data, a second temperature perception result of the target object for the current operating effect of the vehicle air conditioner includes:
[0054] Extracting features from the spectral data in the speech data to obtain speech feature data of the speech data at different times;
[0055] The second temperature perception result is obtained according to the voice feature data at different times.
[0056] In this way, the implementation mode of the present application enables the second temperature perception result to be obtained based on the spectrum data and the speech feature data obtained by feature extraction of the spectrum data, so the second temperature perception result can reflect the changes in the voice line and / or tone of the target object / user.
[0057] In some implementations, obtaining the second temperature perception result according to the voice feature data at different times includes:
[0058] For the two speech feature data at the previous and next moments, the weight of the speech feature data at the next moment is obtained according to the state vector of the speech feature data at the previous moment and the speech feature data at the next moment, wherein the state vector of the speech feature data at the first moment is a zero vector;
[0059] For the two speech feature data at previous and next moments, according to the state vector of the speech feature data at the previous moment and the weight of the speech feature data at the next moment, obtain the state vector of the speech feature data at the next moment;
[0060] The second temperature perception result is obtained according to the weight of the voice feature data at each moment.
[0061] In this way, the implementation mode of the present application enables the second temperature perception result to be based on the continuous fusion of the state vector and weight of the voice feature data, so that the second temperature perception result can carry the information of the voice feature data at each moment.
[0062] In some implementations, the two speech feature data at the previous and next moments, obtaining the weight of the speech feature data at the next moment according to the state vector of the speech feature data at the previous moment and the speech feature data at the next moment, comprises:
[0063] For the two speech feature data at the previous and next moments, determine the state vector of the speech feature data at the previous moment and the pixel difference of the speech feature data at the next moment, and input the pixel difference into a preset fully connected network to obtain the weight of the speech feature data at the next moment.
[0064] In some implementations, obtaining the second temperature perception result according to the voice feature data at different times further includes:
[0065] Accumulate the weights of each of the previous speech feature data before the time to which the speech feature data belongs, to obtain a fusion feature corresponding to the speech feature data;
[0066] The step of obtaining the state vector of the speech feature data at the next moment according to the state vector of the speech feature data at the previous moment and the weight of the speech feature data at the next moment for the two speech feature data at the previous moment comprises:
[0067] For the two speech feature data at previous and next moments, the state vector of the speech feature data at the previous moment is obtained based on the fusion features and the state vector corresponding to the speech feature data at the previous moment, and the weight of the speech feature data at the next moment.
[0068] The present application provides a control device for a vehicle air conditioner, including:
[0069] An acquisition module, used to acquire facial data and voice data of a target object;
[0070] A first recognition module, configured to recognize, based on the facial data, a first temperature perception result of the target object regarding the current operating effect of the vehicle air conditioner;
[0071] A second recognition module is used to recognize a second temperature perception result of the target object for the current operating effect of the vehicle air conditioner according to the voice data;
[0072] a determination module, configured to determine a temperature perception recognition result of the target object corresponding to a current operating effect of the vehicle air conditioner according to the first temperature perception result and the second temperature perception result;
[0073] A control module is used to control the vehicle air conditioner to change operating parameters according to the temperature sensing and recognition result.
[0074] An embodiment of the present application provides a vehicle, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned vehicle air conditioner control method is implemented.
[0075] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by one or more processors, the above-mentioned vehicle air conditioner control method is implemented.
[0076] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0077] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0078] Figure 1 A flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application;
[0079] Figure 2 A schematic diagram of a control device for a vehicle air conditioner in certain embodiments of the present application;
[0080] Figure 3 A flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application;
[0081] Figure 4 A flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application;
[0082] Figure 5 A flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application;
[0083] Figure 6 A schematic diagram of application scenarios of certain embodiments of the present application;
[0084] Figure 7 A schematic diagram of the operation of a model of certain embodiments of the present application;
[0085] Figure 8 A flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application;
[0086] Fig. 9 A flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application;
[0087] Fig.10 A flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application;
[0088] Fig.11 A flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application;
[0089] Fig.12 A relationship diagram of certain embodiments of the present application;
[0090] Fig.13 A flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application;
[0091] Fig.14 A schematic diagram of the operation of a graph convolution layer in certain embodiments of the present application;
[0092] Fig.15 A schematic diagram of reasoning of a graph network model in certain embodiments of the present application;
[0093] Fig.16A flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application;
[0094] Fig.17 A flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application;
[0095] Fig.18 This is a schematic diagram of reasoning of a fully connected layer network in certain implementation modes of the present application;
[0096] Fig.19 A flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application;
[0097] Fig. 20 A flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application;
[0098] Fig.21 A schematic diagram of the operation of a model of certain embodiments of the present application;
[0099] Fig. 22 This is a flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application. DETAILED DESCRIPTION
[0100] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.
[0101] When the vehicle air conditioner performs cooling / heating / air supply functions based on the currently set operating parameters (i.e., the set cooling temperature / heating temperature / wind speed), causing the in-vehicle environment to change accordingly due to the operation of the vehicle air conditioner (such as the in-vehicle temperature rising or falling), if the user cannot tolerate the current in-vehicle environment, that is, thinks that the in-vehicle environment is not suitable, and then wants to adjust the operating parameters of the vehicle air conditioner, the user can manually adjust the operating parameters of the vehicle air conditioner through the vehicle controller, user terminal, in-vehicle touch panel and other devices. However, when the vehicle is in motion, it is very inconvenient for the user to manually adjust the operating parameters of the vehicle air conditioner, and it may cause driving safety issues.
[0102] Based on the above problems you may encounter, please refer to Figure 1 The present application provides a method for controlling a vehicle air conditioner, comprising:
[0103] 01: Obtain the facial data and voice data of the target object;
[0104] 02: Based on the facial data, identify the target object’s first temperature perception result of the current operating effect of the vehicle air conditioner;
[0105] 03: Based on the voice data, identify the second temperature perception result of the target object regarding the current operating effect of the vehicle air conditioner;
[0106] 04: Determine a temperature perception recognition result of the target object corresponding to the current operating effect of the vehicle air conditioner according to the first temperature perception result and the second temperature perception result;
[0107] 05: Control the vehicle air conditioner to change operating parameters based on the temperature sensing and recognition results.
[0108] Please refer to Figure 2 , the embodiment of the present application provides a control device 100 for a vehicle air conditioner, and the control method for the vehicle air conditioner of the embodiment of the present application can be implemented by the control device 100 for the vehicle air conditioner of the embodiment of the present application. Specifically, the control device 100 for the vehicle air conditioner includes an acquisition module 110, a first recognition module 120, a second recognition module 130, a determination module 140 and a control module 150. Among them. The acquisition module 110 is used to acquire facial data and voice data of the target object; the first recognition module 120 is used to identify the first temperature perception result of the target object for the current operating effect of the vehicle air conditioner based on the facial data; the second recognition module 130 is used to identify the second temperature perception result of the target object for the current operating effect of the vehicle air conditioner based on the voice data; the determination module 140 is used to determine the temperature perception recognition result of the target object corresponding to the current operating effect of the vehicle air conditioner based on the first temperature perception result and the second temperature perception result; the control module 150 is used to control the vehicle air conditioner to change the operating parameters according to the temperature perception recognition result.
[0109] The embodiment of the present application provides a vehicle, including a memory and a processor. The control method of the vehicle air conditioner of the embodiment of the present application can be implemented by the vehicle of the embodiment of the present application. Specifically, a computer program is stored in the memory, and the processor is used to obtain facial data and voice data of the target object; based on the facial data, identify the first temperature perception result of the target object for the current operating effect of the vehicle air conditioner; based on the voice data, identify the second temperature perception result of the target object for the current operating effect of the vehicle air conditioner; based on the first temperature perception result and the second temperature perception result, determine the temperature perception recognition result of the target object corresponding to the current operating effect of the vehicle air conditioner; and control the vehicle air conditioner to change the operating parameters according to the temperature perception recognition result.
[0110] Specifically, in the vehicle air conditioning control method provided in the embodiment of the present application, the vehicle first controls the camera module (such as a camera) and the sound collection module (such as a microphone) in the vehicle to monitor the user's facial image and sound / voice, that is, to collect the facial data and voice data of the target object.
[0111] In some embodiments, the facial data and language data are obtained by collecting the face and voice of the user in the main cab.
[0112] In some implementations, the facial data is a video that includes the user's full face.
[0113] In some embodiments, the voice data is spectral data that can represent the user's voice / speech.
[0114] Then, after collecting the facial data and voice data, the vehicle of the implementation mode of the present application performs corresponding recognition on the facial data and voice data respectively to identify the target object's human body temperature perception expressed through facial expressions / face (i.e., the first temperature perception result) and the human body temperature perception expressed through voice / language (i.e., the second temperature perception result) in the in-car environment created by the car air-conditioning with the current operating parameters.
[0115] It should be understood that the current operating effect of the vehicle air conditioner in the implementation of the present application refers to the in-vehicle environment created by the vehicle air conditioner with the current operating parameters, including the in-vehicle temperature, the air supply speed of the vehicle air conditioner, the air supply angle of the vehicle air conditioner and other information. Furthermore, the first temperature perception result and the second temperature perception result in the implementation of the present application represent the user's feelings about the in-vehicle temperature, the air supply speed of the vehicle air conditioner, the air supply angle of the vehicle air conditioner and other information under the current operating effect of the vehicle air conditioner.
[0116] In some embodiments, the vehicle will recognize the user's expression based on the facial data, and determine the first temperature perception result through the preset correspondence between the expression and different temperature perception results. For example, when the vehicle air conditioner performs the cooling function, if the user's expression is recognized as painful / uncomfortable, it means that the user feels that the temperature in the car is too low, so the second temperature perception result is "fear of cold".
[0117] In some embodiments, the vehicle performs natural speech processing on the speech data to detect whether the user's speech includes specific keywords / keywords, such as "very cold", "very hot" and other words, and then generates the above-mentioned second temperature perception result based on the detected keywords / keywords. Exemplarily, when the vehicle air conditioner performs the cooling function, if the keyword "very cold" is detected, it means that the user feels that the temperature in the car is too low, so the second temperature perception result is "fear of cold".
[0118] In some embodiments, the first temperature perception result and the second temperature perception result both include “uncomfortable heat”, “fear of cold”, “comfortable” and other classification results.
[0119] Then, the vehicle of the embodiment of the present application will further fuse the first temperature perception result corresponding to the face and the second temperature perception result corresponding to the voice to determine a more credible final result, i.e., the temperature perception recognition result, through the temperature perception results in two dimensions of face / human face and voice.
[0120] In some embodiments, the temperature perception recognition result is similar to the first temperature perception result and the second temperature perception result described above, and also includes "uncomfortable heat", "fear of cold", "comfortable" and other classification results.
[0121] In some embodiments, the vehicle will perform a weighted sum operation based on a first weight corresponding to the first temperature sensing result and a second weight corresponding to the second temperature sensing result to obtain a temperature sensing recognition result.
[0122] Finally, the vehicle of the embodiment of the present application will adjust the operating parameters of the vehicle air conditioner according to the temperature perception recognition result. For example, when the temperature perception recognition result is "uncomfortable heat" and the operating parameter of the vehicle air conditioner is "20 degrees cooling", the vehicle of the embodiment of the present application will control the operating parameter to change to "18 degrees cooling".
[0123] In this way, the implementation mode of the present application enables the operating parameters of the vehicle air conditioner to be automatically adjusted based on the detected temperature perception recognition results, avoiding the situation where the user needs to make manual adjustments, thereby improving the user experience; at the same time, because the user's facial data and voice data are simultaneously used to determine the user's temperature perception results of the current operating effect of the vehicle air conditioner, the credibility of the temperature perception results is relatively high, and thus the vehicle air conditioner operating parameters can be accurately and effectively controlled to change.
[0124] To more clearly illustrate the implementation of this application, please refer to Figure 3 , Figure 3 This is a flow chart of a method for controlling a vehicle air conditioner in certain embodiments of the present application.
[0125] based on Figure 3 Before the user enters the vehicle, the vehicle receives the instruction sent by the user through the user terminal or other control device, and after knowing the target vehicle temperature and the user's vehicle use time, the vehicle will control the vehicle air conditioner to change the vehicle temperature to the target vehicle temperature and maintain it when the user's vehicle use time arrives; or, when the vehicle determines that the user is close to the vehicle based on the preset sensor or detection module, the vehicle will use the vehicle air conditioner to quickly increase or decrease the vehicle temperature. In this way, the vehicle environment is more suitable, so that the user can get a better vehicle experience when entering the vehicle.
[0126] Then, the vehicle will execute steps 01 to 05 above to obtain the final human body temperature detection result, that is, the above temperature perception recognition result, and then know whether the user is satisfied with the in-car environment / the current operating effect of the vehicle air conditioner.
[0127] Finally, based on the temperature sensing and recognition results, the vehicle controls the operating parameters of the vehicle air conditioner, such as gear position, set temperature and other parameters.
[0128] In certain embodiments, see Figure 4 , step 02 includes:
[0129] 020, input the facial data into a preset graph network model to obtain a first temperature perception result.
[0130] The first recognition module of the implementation mode of the present application is also used to input facial data into a preset graph network model to obtain a first temperature perception result.
[0131] The processor of the embodiment of the present application is also used to input facial data into a preset graph network model to obtain a first temperature perception result.
[0132] That is, the vehicle of the implementation mode of the present application will complete the prediction / identification of the first temperature perception result based on the graph network model (Graph Network).
[0133] It is understandable that compared to convolutional neural networks (CNN) or recurrent neural networks (RNN), graph network models can complete corresponding prediction / recognition tasks based on the connection between data at different positions in the image (such as the edges formed by pixels) and the relationship between previous and subsequent images (such as the state output of the previous frame is used to predict the state of the next frame).
[0134] At the same time, because the user's expression / face will undergo a series of continuous changes when the user perceives the temperature inside the car, the implementation method of the present application uses a graph network model to perform corresponding recognition of facial data, and then complete the prediction of the first temperature perception result.
[0135] It can be understood that, in the case of predicting the first temperature perception result using a graph network model, the facial data in the embodiment of the present application may be in the form of video data including multiple image frames.
[0136] In this way, the implementation manner of the present application predicts the first temperature perception result based on the graph network model, so that the credibility and accuracy of the first temperature perception result are relatively high.
[0137] In certain embodiments, see Figure 5, step 020 comprises:
[0138] 0200, inputting partial facial data obtained based on different shooting angles in the facial data into corresponding branch networks in the graph network model respectively, and obtaining reference temperature perception results corresponding to each partial facial data, wherein each branch network is used to receive corresponding partial facial data and generate reference temperature perception results corresponding to the partial facial data;
[0139] 0201. Obtain a first temperature sensing result according to the reference temperature sensing results corresponding to each partial facial data.
[0140] The first recognition module of the embodiment of the present application is also used to input partial facial data obtained based on different shooting angles in the facial data into corresponding branch networks in the graph network model respectively, to obtain reference temperature perception results corresponding to each partial facial data, wherein each branch network is used to receive corresponding partial facial data, and generate reference temperature perception results corresponding to the partial facial data; and obtain the first temperature perception result according to the reference temperature perception results corresponding to each partial facial data.
[0141] The processor of the embodiment of the present application is also used to input partial facial data in the facial data obtained based on different shooting angles into the corresponding branch networks in the graph network model respectively, to obtain reference temperature perception results corresponding to each partial facial data, wherein each branch network is used to receive the corresponding partial facial data, and generate a reference temperature perception result corresponding to the partial facial data; and obtain a first temperature perception result based on the reference temperature perception results corresponding to each partial facial data.
[0142] It should be understood that the expressions of a face may be inconsistent when observed from different angles. Therefore, in order to avoid the facial data obtained by a single shooting angle being unable to truly reflect the user's expression / facial details, the implementation method of the present application is equipped with multiple cameras, and each camera has a different shooting angle, and then the first temperature perception result is predicted through partial facial data from different shooting angles.
[0143] It should also be understood that since there may be corresponding data differences in partial facial data from different shooting angles, there may be misjudgments when the same model is used to predict different data. Therefore, in order to avoid the misjudgment problem that may be caused by a single model, the implementation method of the present application is provided with multiple branch networks, so that each branch network is used to predict the partial facial data corresponding to the shooting angle, that is, to output the reference temperature perception result.
[0144] To more clearly illustrate the implementation of this application, please refer to Figure 6 , Figure 6 A schematic diagram of application scenarios of certain embodiments of the present application.
[0145] Specifically, Figure 6 There are three cameras in the system, each of which captures the user's facial data at a different angle. Figure 6 When facial data is acquired in the scenario shown, the facial data will be composed of three partial facial data; and, the graph network model in the implementation manner of the present application will include three branch networks, one branch network being used to predict partial facial data corresponding to a shooting angle to generate a corresponding reference temperature perception result.
[0146] In some embodiments, the facial data / partial facial data is obtained by capturing the complete face of the user using an infrared vision sensor.
[0147] To more clearly illustrate the implementation of this application, please refer to Figure 7 , Figure 7 This is a schematic diagram of the model operation of certain embodiments of the present application.
[0148] Specifically, after each partial facial data is input into the corresponding branch network, each branch network will output a corresponding reference temperature perception result. The reference temperature perception results of all branch networks will be fused to obtain the first temperature perception result.
[0149] It is understandable that the reference temperature perception result may be similar to the first temperature perception result, and also include “uncomfortable heat”, “fear of cold”, “comfortable” and other classification results.
[0150] It can also be understood that the process of fusing different reference temperature perception results into a first temperature perception result is a content that can be set according to actual conditions. For example, in some embodiments, different reference temperature perception results are input into a pre-trained fully connected network, so that the fully connected network generates a first temperature perception result based on all reference temperature perception results according to the trained bias parameters and weight parameters.
[0151] In this way, the implementation mode of the present application enables the acquisition of the first temperature perception result based on partial facial data from different shooting angles, thereby avoiding the situation where the facial data obtained from a single shooting angle cannot truly reflect the user's expression / facial details, thereby ensuring the credibility of the first temperature perception result; at the same time, because the partial facial data from each shooting angle will output the corresponding reference temperature perception result based on the corresponding branch network, the accuracy of each reference temperature perception result is relatively high, thereby further improving the credibility of the first temperature perception result composed of multiple reference temperature perception results.
[0152] In certain embodiments, see Figure 8 Before step 0200, the vehicle air conditioner control method further includes:
[0153] 06: Using the partial facial data corresponding to the branch network as a training sample, and obtaining a data enhancement sample based on at least one of the other partial facial data;
[0154] 07: Train the branch network based on its training samples and data augmentation samples.
[0155] The vehicle air conditioner control device of the embodiment of the present application further includes a sample acquisition module and a training module. The sample acquisition module is used to use the partial facial data corresponding to the branch network as a training sample, and obtain a data enhancement sample based on at least one of the other partial facial data. The training module is used to train the branch network based on the training sample and the data enhancement sample of the branch network.
[0156] The processor of the embodiment of the present application is also used to use the partial facial data corresponding to the branch network as a training sample, and obtain a data enhancement sample based on at least one of the other partial facial data; and train the branch network based on the training sample and the data enhancement sample of the branch network.
[0157] That is, in order to ensure the robustness and accuracy of each branch network, the implementation method of the present application will, in the process of training each branch network, not only use the partial facial data of the corresponding shooting angle as training samples, but also use partial facial data from other shooting angles for data enhancement, thereby increasing the sample size of the branch network.
[0158] Demonstration, Figure 6 Taking the scene shown as an example, suppose the three cameras from top to bottom are named C1, C2 and C3 respectively, the three partial facial data captured by C1, C2 and C3 are named F1, F2 and F3 respectively, F1, F2 and F3 are the training samples of branch network N1, branch network N2 and branch network N3 respectively, then F1 can be used for training N2 and / or N3 in addition to training N1.
[0159] It is understandable that the specific process of using other partial facial data to perform data enhancement is a content that can be set according to actual conditions. Taking the above C1 to C3, F1 to F3, and N1 to N3 as examples, in some embodiments, N1 trains F1 and F2 without any adjustment. In other embodiments, the vehicle will also perform affine transformation processing on F2 according to preset array parameters to obtain F2', and then use F2' and F1 to complete the training of N1.
[0160] In some embodiments, the video sampling frame rates and zoom ratios of different infrared vision sensors are inconsistent.
[0161] In this way, the implementation method of the present application is based on different partial facial data, so that the number of samples used in the training of each branch network is guaranteed, thereby avoiding the occurrence of underfitting during the training of each branch network, and therefore, the robustness of each branch network is guaranteed.
[0162] In certain embodiments, see Fig. 9 , step 020 comprises:
[0163] 0202, performing frame processing on the facial data to obtain multiple frames of facial images;
[0164] 0203, identifying key points of the face in the facial image, and obtaining key point data corresponding to the facial image;
[0165] 0204, input the key point data corresponding to each frame of the facial image into the graph network model to obtain the first temperature perception result.
[0166] The first recognition module of the implementation mode of the present application is also used to perform frame processing on facial data to obtain multiple frames of facial images; identify facial key points in the facial images to obtain key point data corresponding to the facial images; input the key point data corresponding to each frame of the facial image into the graph network model to obtain a first temperature perception result.
[0167] The processor of the embodiment of the present application is also used to perform frame processing on facial data to obtain multiple frames of facial images; identify facial key points in the facial images to obtain key point data corresponding to the facial images; input the key point data corresponding to each frame of the facial image into the graph network model to obtain a first temperature perception result.
[0168] That is, the implementation mode of the present application will perform frame processing on the video data / facial data to obtain multiple frames of facial images arranged in time in the video data / facial data; at the same time, the graph network model will capture the connection between the facial images at different times in the multiple frames of facial images, thereby completing the prediction of the first temperature perception result.
[0169] Furthermore, in order to avoid the interference of background factors in facial images and to reduce the computing resources required for prediction, the implementation mode of the present application will detect the key points of the face of each facial image, and then complete the prediction of the first temperature perception result through the key points obtained by the detection, that is, the key point data.
[0170] It is understandable that the specific method of identifying the key points of the face in the facial image is a content that can be set according to the actual situation. In some embodiments, the acquisition of the key points of the face / key point data is based on the ERT (ensemble of regression trees, ERT) cascade regression tree.
[0171] In this way, the implementation mode of the present application predicts the first temperature perception result based on the key points of the face, so that the prediction of the first temperature perception result can avoid the interference of background factors in the facial image and reduce the computing resources required for the prediction. Thus, the efficiency of obtaining the first temperature perception result is improved.
[0172] In certain embodiments, see Fig.10 , step 020 further includes:
[0173] 0205: normalize the key point data to obtain processed key point data;
[0174] Step 0204 includes:
[0175] 02040, input the processed key point data corresponding to each frame of the facial image into the graph network model to obtain a first temperature perception result.
[0176] The first recognition module of the implementation mode of the present application is also used to normalize the key point data to obtain processed key point data; the processed key point data corresponding to each frame of the facial image is input into the graph network model to obtain a first temperature perception result.
[0177] The processor of the implementation mode of the present application is also used to normalize the key point data to obtain processed key point data; the processed key point data corresponding to each frame of the facial image is input into the graph network model to obtain a first temperature perception result.
[0178] That is, in order to further reduce the computing resources required to predict the first temperature perception result, the implementation mode of the present application also normalizes the key point data of each facial image before using the facial key points / key point data of each facial image for prediction, so that the data amount of the key point data after normalization is reduced.
[0179] In this way, the implementation mode of the present application is based on the normalization processing of the key point data, so that the data volume of the key point data can be reduced, and thus fewer computing resources are required to predict the first temperature perception result, thereby further improving the efficiency of obtaining the first temperature perception result.
[0180] In certain embodiments, see Fig.11 , step 020 further includes:
[0181] 0206: Generate a relationship graph according to the connection relationship between the same two face key points in the key point data corresponding to the adjacent facial images and the connection relationship between the adjacent face key points in the key point data corresponding to each frame of the facial image;
[0182] Step 0204 includes:
[0183] 02041: Input the relationship graph into multiple graph convolution layers of the graph network model to obtain a first temperature perception result.
[0184] The first recognition module of the implementation mode of the present application is also used to generate a relationship graph based on the connection relationship between the same two facial key points in the key point data corresponding to adjacent facial images, and the connection relationship between adjacent facial key points in the key point data corresponding to each frame of facial image; the relationship graph is input into multiple graph convolution layers of the graph network model to obtain a first temperature perception result.
[0185] The processor of the embodiment of the present application is also used to generate a relationship graph based on the connection relationship between the same two facial key points in the key point data corresponding to adjacent facial images, and the connection relationship between adjacent facial key points in the key point data corresponding to each frame of facial image; the relationship graph is input into multiple graph convolution layers of the graph network model to obtain a first temperature perception result.
[0186] That is, in order to further improve the credibility of the first temperature sensing result, the vehicle of the embodiment of the present application adjusts the input data of the graph network model. Fig.12 , Fig.12 This is a relationship diagram of some embodiments of the present application.
[0187] It should be noted that Fig.12 The figure shows a human face composed of multiple dots / key points, and two faces represent facial images of different frames. Furthermore, there are connecting edges between the same key points of the two faces, such as Fig.12 The two key points of the eyebrow part of the left face are connected to the two key points of the eyebrow part of the right face (i.e. Fig.12 ).
[0188] It should also be noted that Fig.12 Only the connection relationship between some key points of two faces is shown, but in actual situations, the same key points in adjacent frame images have connection relationships / connection edges. Fig.12 Just an example.
[0189] It can also be understood that in a single frame image, there are also connection relationships / connection edges between adjacent key points, such as Fig.12 There are connecting edges between the six key points of the left eye in the left face.
[0190] Thus, based on the connection relationship between the key points of adjacent frames of facial images and the connection relationship between adjacent key points in a single frame of facial images, the embodiment of the present application will generate a similar Fig.12 The relationship diagram shown in FIG. 1 is stored based on the corresponding data structure. In some embodiments, the relationship diagram is stored based on the following data structure, namely:
[0191] G=(V,E)
[0192] V={v ti |t=1,2,...,T;i=1,2,...,N}
[0193] E F = {v ti v (t+1)i}
[0194] E S = {v ti v tj |i,j∈N}
[0195] Among them, G represents the relationship graph, T represents the number of facial images, N represents the number of key points, V represents the key points, and E represents the connection relationship. E is composed of E F and E S Composition, v ti represents the i-th key point in the t-th frame facial image, E F Indicates the connection relationship between the same key point in the image points of adjacent frames, E S Represents the connection relationship between adjacent key points in a single frame facial image.
[0196] Furthermore, after obtaining the relationship graph, the implementation mode of the present application will capture the connection relationship features of adjacent key points in a single-frame facial image and the connection relationship features of the same shutdown points in adjacent frames of facial images based on the graph convolution in the graph network model, thereby completing the prediction of the first temperature perception result.
[0197] In some embodiments, the graph convolutional layer converts v ti The nearby K nodes are v ti The neighborhood set of , and calculate v ti The average value of the feature vector of each key point in the neighborhood set is calculated, and then the corresponding convolution operation is completed based on the average value.
[0198] In some embodiments, the operation of the graph convolution layer is shown as follows:
[0199]
[0200] In the formula, f out represents the output of the graph convolutional layer, f in Represents the input of the graph convolution layer, x represents the key point, h and w represent the height and width respectively, p represents the average calculation, and ω represents the weight.
[0201] In this way, the present application is based on the relationship graph and the graph convolution layer, so that the feature information corresponding to the connection relationship of each key point in multiple frames of facial images can be used to predict the first temperature perception result, so the credibility of the first temperature perception result can be further improved.
[0202] In certain embodiments, see Fig.13 , step 02041 includes:
[0203] 020410: Input the relationship graph into multiple graph convolutional layers connected sequentially in the graph network model;
[0204] 020411: For each graph convolution layer, the received input data is processed according to the graph convolution layer to obtain output data, and the input data and the output data are sent to the next graph convolution layer of the graph convolution layer;
[0205] 020412: Obtain the first temperature perception result based on the output data of the last graph convolution layer.
[0206] The first recognition module of the implementation mode of the present application is also used to input the relationship graph into multiple graph convolution layers connected in sequence in the graph network model; for each graph convolution layer, the received input data is processed according to the graph convolution layer to obtain output data, and the input data and the output data are sent to the next graph convolution layer of the graph convolution layer; the first temperature perception result is obtained according to the output data of the last graph convolution layer.
[0207] The processor of the embodiment of the present application is also used to input the relationship graph into multiple graph convolution layers connected in sequence in the graph network model; for each graph convolution layer, the received input data is processed according to the graph convolution layer to obtain output data, and the input data and the output data are sent to the next graph convolution layer of the graph convolution layer; and the first temperature perception result is obtained according to the output data of the last graph convolution layer.
[0208] That is, the implementation mode of the present application also sets the data transmission mode of each graph convolution layer based on the residual connection method, so that the input and output of any graph convolution layer will serve as the input of the subsequent graph convolution layer.
[0209] In addition, to more clearly illustrate the implementation of this application, please refer to Fig.14 , Fig.14 A schematic diagram of the operation of a graph convolutional layer in some embodiments of the present application.
[0210] Specifically, after receiving the input data x, a graph convolution layer in an embodiment of the present application will perform two graph convolution operations to generate F(x), and x and F(x) will be used together as the input data H(x) of the next graph convolution layer, that is, H(x)=x+F(x).
[0211] In this way, the implementation method of the present application enables the latter graph convolution layer of the two preceding and succeeding graph convolution layers to complete corresponding operations based on the input and output of the previous graph convolution layer, so that the gradient vanishing phenomenon in the graph network model can be improved, thereby ensuring the robust operation of the graph network model.
[0212] To more clearly illustrate the implementation of this application, please refer to Fig.15 , Fig.15 This is a schematic diagram of reasoning for a graph network model in certain embodiments of the present application.
[0213] Specifically, the implementation method of the present application first pre-processes the facial data, that is, performs frame processing, facial key point recognition and key point data normalization. Then, the processed key point data is input into the graph convolution layer in the graph network model so that the graph convolution layer performs corresponding processing; the last graph convolution layer passes the output data to the softmax / normalized exponential function layer so that the softmax layer outputs the first temperature perception result. It is not difficult to understand that Fig.15 This is only one of the feasible implementations. The specific reasoning process of the graph convolution model in the implementation of this application can be adjusted according to actual conditions.
[0214] In certain embodiments, see Fig.16 , step 03 includes:
[0215] 030: Input the voice data into a preset recognition network model to obtain a second temperature perception result;
[0216] Step 04 includes:
[0217] 040: Input the second temperature sensing result and the first temperature sensing result into the fully connected layer network to obtain a temperature sensing recognition result.
[0218] The second recognition module of the implementation mode of the present application is also used to input the voice data into a preset recognition network model to obtain a second temperature perception result; the determination module is also used to input the second temperature perception result and the first temperature perception result into a fully connected layer network to obtain a temperature perception recognition result.
[0219] The processor of the embodiment of the present application is also used to input the voice data into a preset recognition network model to obtain a second temperature perception result; input the second temperature perception result and the first temperature perception result into a fully connected layer network to obtain a temperature perception recognition result.
[0220] That is, the implementation of the present application will respectively complete the prediction of the first temperature perception result and the second temperature perception result based on the two pre-trained neural network models. Moreover, after obtaining the first temperature perception result and the second temperature perception result, the implementation of the present application will complete the fusion of the first temperature perception result and the second temperature perception result based on the pre-trained fully connected layer network, so that the two temperature perception results can fully consider the characteristics of the facial data and the sound characteristics when fusion.
[0221] In this way, the implementation mode of the present application enables both the first temperature perception result and the second temperature perception result to be predicted based on the pre-trained neural network model, so that the accuracy of the first temperature perception result and the second temperature perception result can be guaranteed; at the same time, because the first temperature perception result and the second temperature perception result will be fused based on the fully connected network, the credibility of the fused temperature perception recognition result can also be guaranteed.
[0222] In certain embodiments, see Fig.17 , step 040 comprises:
[0223] 0400: inputting the second temperature sensing result and the first temperature sensing result into the fully connected layer network;
[0224] 0401: for each fully connected unit of the fully connected layer network, processing the received data according to the fully connected unit to generate a probability value, and inputting the probability value to the next fully connected unit, wherein the fully connected layer network includes a plurality of fully connected units connected in sequence, and the fully connected unit includes at least one fully connected layer;
[0225] 0402: According to the probability value of the last fully connected unit, the temperature perception recognition result is obtained.
[0226] The determination module of the implementation mode of the present application is also used to input the second temperature perception result and the first temperature perception result into a fully connected layer network; for each fully connected unit of the fully connected layer network, the received data is processed according to the fully connected unit to generate a probability value, and the probability value is input to the next fully connected unit, wherein the fully connected layer network includes multiple fully connected units connected in sequence, and the fully connected unit includes at least one fully connected layer; according to the probability value of the last fully connected unit, the temperature perception recognition result is obtained.
[0227] The processor of the embodiment of the present application is also used to input the second temperature perception result and the first temperature perception result into a fully connected layer network; for each fully connected unit of the fully connected layer network, the received data is processed according to the fully connected unit to generate a probability value, and the probability value is input to the next fully connected unit, wherein the fully connected layer network includes multiple fully connected units connected in sequence, and the fully connected unit includes at least one fully connected layer; according to the probability value of the last fully connected unit, the temperature perception recognition result is obtained.
[0228] To more clearly illustrate the implementation of this application, please refer to Fig.18 , Fig.18 This is a schematic diagram of reasoning for a fully connected layer network in certain implementations of the present application.
[0229] It should be noted that Fig.18 The triangles in the leftmost column in the figure represent the first temperature perception results output by the recognition network model, and the circles in the leftmost column represent the second temperature perception results output by the network model; Fig.18 The circles in the two columns on the right represent the probability values of the fully connected layer output in the fully connected layer network. Fig.18 In the scenario shown, the fully connected layer network includes two fully connected units, one of which includes three fully connected layers and the other includes one fully connected layer.
[0230] In some embodiments, the last layer of the recognition network model is a fully connected layer, and the last layer of the graph network model is a softmax layer, that is, the first temperature perception result is based on the output result of the softmax layer in the graph network model, and the second temperature perception result is the output result of the fully connected layer in the recognition network model.
[0231] Furthermore, Fig.18 The fully connected unit on the left is the first unit, and the fully connected unit on the right is the second unit. The three fully connected layers of the first unit obtain the first temperature perception result and the second temperature perception result output by the above-mentioned graph network model and recognition network model and fuse them to generate three probability values; the three probability values will be input into a fully connected layer of the second unit for fusion again to generate a corresponding probability value as the temperature perception recognition result. Among them, the probability value output by the fully connected layer in the fully connected network can be regarded as a score.
[0232] In certain embodiments, Fig.18For example, to further improve the credibility of the temperature perception recognition result, the circles in the leftmost column represent the output results of each softmax function in the graph network model, that is, the circles in the leftmost column include the first temperature perception result and the outputs of other softmax functions in the graph network model. Thus, the temperature perception recognition result will be obtained based on high-dimensional information (such as the first temperature perception result) and low-dimensional information (such as the outputs of other softmax functions in the graph network model).
[0233] It should be understood that since each fully connected layer in the fully connected network can be regarded as a classifier, the fully connected layer network can be understood as a fusion method based on multiple decisions.
[0234] In this way, the implementation mode of the present application is based on the setting of a fully connected layer network, so that the first temperature perception result and the second temperature perception result can make full use of the prediction results corresponding to the facial data and the prediction results corresponding to the sound data when they are fused, thereby ensuring the credibility of the temperature perception recognition results.
[0235] In certain embodiments, see Fig.19 , step 03 includes:
[0236] 031: Extract features from the spectrum data in the speech data to obtain speech feature data of the speech data at different times;
[0237] 032: Obtain a second temperature perception result according to the voice feature data at different times.
[0238] The second recognition module of the implementation mode of the present application is also used to extract features from the spectral data in the voice data to obtain voice feature data of the voice data at different times; and obtain the second temperature perception result based on the voice feature data at different times.
[0239] The processor of the implementation mode of the present application is also used to extract features from the spectral data in the voice data to obtain voice feature data of the voice data at different times; and obtain a second temperature perception result based on the voice feature data at different times.
[0240] That is, the embodiment of the present application will complete the acquisition of the second temperature perception result based on the sound spectrum data. It is understandable that the sound spectrum data can reflect the relationship between the frequency and amplitude of the user at a certain moment in the process of speaking, and can also reflect whether the user's voice is not stable and whether the tone has changed.
[0241] At the same time, in order to ensure that the spectral data can effectively reflect whether the user's voice is stable and whether the tone has changed, the implementation method of the present application also extracts features from the spectral data to obtain voice feature data at different times to ensure that the subsequent calculation of the second temperature perception result can be completed based on valid feature data.
[0242] In some implementations, the speech feature data may be obtained by downsampling the spectrum data.
[0243] In this way, the implementation mode of the present application enables the second temperature perception result to be obtained based on the spectrum data and the speech feature data obtained by feature extraction of the spectrum data, so the second temperature perception result can reflect the changes in the voice line and / or tone of the target object / user.
[0244] In certain embodiments, see Fig. 20 , step 032 comprises:
[0245] 0320: for two speech feature data at the previous and next moments, according to the state vector of the speech feature data at the previous moment and the speech feature data at the next moment, the weight of the speech feature data at the next moment is obtained, wherein the state vector of the speech feature data at the first moment is a zero vector;
[0246] 0321: for two speech feature data at the previous and next moments, according to the state vector of the speech feature data at the previous moment and the weight of the speech feature data at the next moment, obtain the state vector of the speech feature data at the next moment;
[0247] 0322: Obtain a second temperature perception result according to the weight of the voice feature data at each moment.
[0248] The second recognition module of the implementation mode of the present application is also used to obtain the weight of the speech feature data at the latter moment based on the state vector of the speech feature data at the previous moment and the speech feature data at the latter moment, wherein the state vector of the speech feature data at the first moment is a zero vector; for the two speech feature data at the previous moment, obtain the state vector of the speech feature data at the latter moment based on the state vector of the speech feature data at the previous moment and the weight of the speech feature data at the latter moment; and obtain the second temperature perception result based on the weight of the speech feature data at each moment.
[0249] The processor of the embodiment of the present application is also used to obtain the weight of the voice feature data at the latter moment based on the state vector of the voice feature data at the previous moment and the voice feature data at the latter moment, with the state vector of the voice feature data at the first moment being a zero vector; for the two voice feature data at the previous moment, obtain the state vector of the voice feature data at the latter moment based on the state vector of the voice feature data at the previous moment and the weight of the voice feature data at the latter moment; and obtain the second temperature perception result based on the weight of the voice feature data at each moment.
[0250] To more clearly illustrate the implementation of the present application, please refer to the following formula, namely:
[0251] scOre t+1 =f(state(t),feature t+1 )
[0252] state(t+1)=f(state(t),score t+1 , feature t+1 )
[0253] In the formula, score t+1 represents the weight of the speech feature data at time t+1, state(t) represents the state vector of the speech feature data at time t, feature t+1 represents the speech feature data at time t+1, and f() represents the corresponding function. It can be understood that the state vector of the speech feature data at the first moment is state(0), and state(0) in the implementation mode of the present application is a zero vector.
[0254] It is understandable that the state vector in the embodiment of the present application is obtained based on the corresponding voice feature data, and the state vector at the current moment will be used for weight calculation of the voice feature data at the next moment. At the same time, the weight of the voice feature data at the current moment is also used to update the weight of the voice feature data at the next moment.
[0255] Furthermore, based on the continuous iteration of weights and state vectors, the weight of each speech feature data can contain the information of the speech feature data at the previous moment. Therefore, after the weights of each speech feature data are fused in the implementation mode of the present application, the fused second temperature perception result can characterize the information of each speech feature data from the starting moment to the end moment.
[0256] In this way, the implementation mode of the present application enables the second temperature perception result to be based on the continuous fusion of the state vector and weight of the voice feature data, so that the second temperature perception result can carry the information of the voice feature data at each moment.
[0257] To more clearly illustrate the implementation of this application, please refer to Fig.21 , Fig.21 This is a schematic diagram of the model operation of certain embodiments of the present application.
[0258] That is, the implementation mode of the present application will also be Figure 1 5, after outputting the corresponding second temperature sensing result based on adaptive pooling (i.e., steps 0321 to 03222), the first temperature sensing result and the second temperature sensing result are fused to obtain the temperature sensing recognition result.
[0259] In certain embodiments, Fig.21 As shown, the second temperature perception result is the output of the fully connected layer.
[0260] In some embodiments, in the process of outputting the second temperature sensing result based on adaptive pooling (ie, steps 0321 to 03222), a residual connection method is used to set the input and output at each moment.
[0261] In some embodiments, step 0320 includes:
[0262] For the two speech feature data at the previous and next moments, the state vector of the speech feature data at the previous moment and the pixel difference of the speech feature data at the next moment are determined, and the pixel difference is input into the preset fully connected network to obtain the weight of the speech feature data at the next moment.
[0263] The second recognition module of the embodiment of the present application is also used to determine the state vector of the speech feature data at the previous moment and the pixel difference of the speech feature data at the next moment, and input the pixel difference into a preset fully connected network to obtain the weight of the speech feature data at the next moment.
[0264] The processor of the embodiment of the present application is also used to determine the state vector of the speech feature data at the previous moment and the pixel difference of the speech feature data at the next moment, and input the pixel difference into a preset fully connected network to obtain the weight of the speech feature data at the next moment.
[0265] That is, for the speech feature data at time t+1, the implementation method of the present application will use state(t) and feature t+1 After performing pixel-level subtraction to obtain the corresponding feature map, this feature map and feature t+1 The output of the preset fully connected network is used as the weight score of the speech feature data at time t+1. t+1 .
[0266] In some embodiments, the preset fully connected network includes three fully connected layers connected in sequence, and the number of neurons in the last fully connected layer is 1.
[0267] In some embodiments, the preset fully connected network also includes a sigmoid activation function layer, which is used to perform corresponding operations based on the output probability value of the last fully connected layer in the preset fully connected network, and the result output by the sigmoid activation function layer is the weight of the speech feature data.
[0268] In certain embodiments, see Fig. 22 , step 032, further comprising:
[0269] 0323: Accumulate the weight of each speech feature data before the time to which the speech feature data belongs, and obtain the fusion feature corresponding to the speech feature data;
[0270] Step 0321, comprising:
[0271] 03210: For two speech feature data at previous and next moments, the state vector of the speech feature data at the next moment is obtained according to the fusion features and state vector corresponding to the speech feature data at the previous moment and the weight of the speech feature data at the next moment.
[0272] The second recognition module of the implementation mode of the present application is also used to accumulate the weights of each previous speech feature data before the moment to which the speech feature data belongs, and obtain the fusion feature corresponding to the speech feature data; for the two speech feature data at the previous and next moments, according to the fusion features and state vector corresponding to the speech feature data at the previous moment, and the weight of the speech feature data at the next moment, the state vector of the speech feature data at the next moment is obtained.
[0273] The processor of the implementation mode of the present application is also used to accumulate the weights of each previous speech feature data before the moment to which the speech feature data belongs, and obtain the fusion feature corresponding to the speech feature data; for the two speech feature data at the previous and next moments, according to the fusion features and state vector corresponding to the speech feature data at the previous moment, and the weight of the speech feature data at the next moment, the state vector of the speech feature data at the next moment is obtained.
[0274] To more clearly illustrate the implementation of the present application, please refer to the following formula, namely:
[0275]
[0276] In the formula, sum(t) represents the fusion feature corresponding to the speech feature data at time t.
[0277] Therefore, the implementation mode of the present application will be based on the setting of fusion features so that the state vector of the speech feature data at each moment can reuse the information of the speech feature data at all previous moments.
[0278] In some embodiments, score t+1 The acquisition process can refer to the following formula:
[0279]
[0280] In some implementations, the fusion feature corresponding to the voice feature data at the last moment represents the second temperature perception result.
[0281] The present application also provides a computer-readable storage medium containing a computer program. When the computer program is executed by one or more processors, the one or more processors execute the vehicle air conditioner control method according to the embodiment of the present application.
[0282] In the description of this specification, the descriptions with reference to the terms "certain embodiments", "in an example", "exemplarily", etc., mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.
[0283] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0284] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A method for controlling a vehicle air conditioner, characterized in that: include: Obtain facial data and voice data of the target object; Identifying, based on the facial data, a first temperature perception result of the target object regarding the current operating effect of the vehicle air conditioner; identifying, according to the voice data, a second temperature perception result of the target object regarding the current operating effect of the vehicle air conditioner; Determine, according to the first temperature sensing result and the second temperature sensing result, a temperature sensing recognition result of the target object corresponding to the current operating effect of the vehicle air conditioner; The vehicle air conditioner is controlled to change operating parameters according to the temperature sensing and recognition result.
2. The method according to claim 1, characterized in that The step of identifying, based on the facial data, a first temperature perception result of the target object regarding the current operating effect of the vehicle air conditioner comprises: The facial data is input into a preset graph network model to obtain the first temperature perception result.
3. The method according to claim 2, characterized in that The step of inputting the facial data into a preset graph network model to obtain the first temperature sensing result includes: Inputting partial facial data obtained based on different shooting angles in the facial data into corresponding branch networks in the graph network model respectively, and obtaining reference temperature perception results corresponding to each partial facial data, wherein each branch network is used to receive the corresponding partial facial data and generate the reference temperature perception results corresponding to the partial facial data; The first temperature sensing result is obtained according to the reference temperature sensing result corresponding to each of the partial facial data.
4. The method according to claim 3, characterized in that Before inputting the partial facial data obtained based on different shooting angles in the facial data into the corresponding branch networks in the graph network model to obtain the reference temperature perception results corresponding to the partial facial data, the method further includes: Using the partial facial data corresponding to the branch network as a training sample, and obtaining a data enhancement sample according to at least one of the other partial facial data; The branch network is trained according to the training samples and data enhancement samples of the branch network.
5. The method according to claim 2, characterized in that: The step of inputting the facial data into a preset graph network model to obtain the first temperature sensing result includes: Performing frame processing on the facial data to obtain multiple frames of facial images; Identify key points of the face in the facial image to obtain key point data corresponding to the facial image; The key point data corresponding to each frame of the facial image is input into the graph network model to obtain the first temperature perception result.
6. The method according to claim 5, characterized in that The step of inputting the facial data into a preset graph network model to obtain the first temperature sensing result further includes: Normalizing the key point data to obtain processed key point data; The step of inputting key point data corresponding to each frame of the facial image into the graph network model to obtain the first temperature perception result includes: The processed key point data corresponding to each frame of the facial image is input into the graph network model to obtain the first temperature perception result.
7. The method according to claim 5, characterized in that The step of inputting the facial data into a preset graph network model to obtain the first temperature sensing result further includes: Generate a relationship graph according to the connection relationship between the same two face key points in the key point data corresponding to the adjacent facial images and the connection relationship between the adjacent face key points in the key point data corresponding to each frame of the facial image; The step of inputting key point data corresponding to each frame of the facial image into the graph network model to obtain the first temperature perception result includes: The relationship graph is input into multiple graph convolution layers of the graph network model to obtain the first temperature perception result.
8. The method according to claim 7, characterized in that The step of inputting the relationship graph into a plurality of graph convolution layers of the graph network model to obtain the first temperature sensing result includes: Inputting the relationship graph into a plurality of graph convolutional layers sequentially connected in the graph network model; For each of the graph convolution layers, processing the received input data according to the graph convolution layer to obtain output data, and sending the input data and the output data to a subsequent graph convolution layer of the graph convolution layer; The first temperature sensing result is obtained according to the output data of the last graph convolution layer.
9. The method according to claim 2, characterized in that: The step of identifying, according to the voice data, a second temperature perception result of the target object regarding the current operating effect of the vehicle air conditioner includes: Inputting the voice data into a preset recognition network model to obtain the second temperature perception result; The determining, according to the first temperature sensing result and the second temperature sensing result, a temperature sensing recognition result of the target object corresponding to the current operating effect of the vehicle air conditioner includes: The second temperature sensing result and the first temperature sensing result are input into a fully connected layer network to obtain the temperature sensing recognition result.
10. The method according to claim 9, characterized in that The step of inputting the second temperature sensing result and the first temperature sensing result into a fully connected layer network to obtain the temperature sensing recognition result includes: Inputting the second temperature sensing result and the first temperature sensing result into the fully connected layer network; For each fully connected unit of the fully connected layer network, processing the received data according to the fully connected unit to generate a probability value, and inputting the probability value to the next fully connected unit, wherein the fully connected layer network includes a plurality of the fully connected units connected in sequence, and the fully connected unit includes at least one fully connected layer; The temperature perception recognition result is obtained according to the probability value of the last fully connected unit.
11. The method according to claim 1, characterized in that: The step of identifying, according to the voice data, a second temperature perception result of the target object regarding the current operating effect of the vehicle air conditioner includes: Extracting features from the spectral data in the speech data to obtain speech feature data of the speech data at different times; The second temperature perception result is obtained according to the voice feature data at different times.
12. The method according to claim 11, characterized in that The obtaining the second temperature sensing result according to the voice feature data at different times includes: For the two speech feature data at the previous and next moments, the weight of the speech feature data at the next moment is obtained according to the state vector of the speech feature data at the previous moment and the speech feature data at the next moment, wherein the state vector of the speech feature data at the first moment is a zero vector; For the two speech feature data at previous and next moments, according to the state vector of the speech feature data at the previous moment and the weight of the speech feature data at the next moment, obtain the state vector of the speech feature data at the next moment; The second temperature perception result is obtained according to the weight of the voice feature data at each moment.
13. The method according to claim 12, characterized in that The step of obtaining the weight of the speech feature data at the latter moment according to the state vector of the speech feature data at the former moment and the speech feature data at the latter moment for the two speech feature data at the former moment comprises: For the two speech feature data at the previous and next moments, determine the state vector of the speech feature data at the previous moment and the pixel difference of the speech feature data at the next moment, and input the pixel difference into a preset fully connected network to obtain the weight of the speech feature data at the next moment.
14. The method according to claim 12, characterized in that The obtaining the second temperature sensing result according to the voice feature data at different times also includes: Accumulate the weights of each of the previous speech feature data before the time to which the speech feature data belongs, to obtain a fusion feature corresponding to the speech feature data; The step of obtaining the state vector of the speech feature data at the next moment according to the state vector of the speech feature data at the previous moment and the weight of the speech feature data at the next moment for the two speech feature data at the previous moment comprises: For the two speech feature data at previous and next moments, the state vector of the speech feature data at the previous moment is obtained based on the fusion features and the state vector corresponding to the speech feature data at the previous moment, and the weight of the speech feature data at the next moment.
15. A control device for a vehicle air conditioner, characterized in that: include: An acquisition module, used to acquire facial data and voice data of a target object; A first recognition module, configured to recognize, based on the facial data, a first temperature perception result of the target object regarding the current operating effect of the vehicle air conditioner; A second recognition module is used to recognize a second temperature perception result of the target object for the current operating effect of the vehicle air conditioner according to the voice data; a determination module, configured to determine a temperature perception recognition result of the target object corresponding to a current operating effect of the vehicle air conditioner according to the first temperature perception result and the second temperature perception result; A control module is used to control the vehicle air conditioner to change operating parameters according to the temperature sensing and recognition result.
16. A vehicle, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 14 is implemented.
17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 14 is implemented.
