Methods, devices, equipment, storage media, and programs for adjusting the in-vehicle environment.

Abstract

This application relates to a method, apparatus, device, storage medium, and program product for adjusting the in-vehicle environment, and pertains to the field of intelligent vehicle technology. The method is executed by the vehicle's control system. The vehicle includes an onboard monocular camera, comprising: acquiring an image of the driver captured by the onboard monocular camera; identifying the driver's emotional information and state information based on the image; the emotional information indicating the driver's emotions; the state information indicating the driver's mental state; determining adjustment parameters for the vehicle's in-vehicle environment based on the emotional and state information; the adjustment parameters including at least one of the following: lighting adjustment parameters, seat adjustment parameters, and audio adjustment parameters; and adjusting the vehicle's in-vehicle environment according to the adjustment parameters. This solution improves the efficiency of the vehicle's control system in adjusting the in-vehicle environment.

Description

Technical Field

[0001] This application relates to the field of intelligent vehicle technology, and in particular to a method, device, equipment, storage medium, and program product for adjusting the in-vehicle environment. Background Technology

[0002] With the continuous development of the Internet, the application of intelligent sensing technology is becoming more and more widespread, especially in the intelligent vehicle industry.

[0003] In related technologies, drivers can perform specified operations on the vehicle's infotainment system display screen. After receiving the instructions from the driver, the infotainment system controls the entire vehicle to adjust the in-vehicle environment. Drivers can also adjust the in-vehicle environment by performing specified operations on the vehicle's hardware. Adjustments to the in-vehicle environment can include adjusting the brightness of the interior lights, the angle of the seats, and the volume of the in-vehicle audio system.

[0004] However, the aforementioned methods for adjusting the in-vehicle environment are relatively simple and require active operation by the driver, resulting in low efficiency in adjusting the in-vehicle environment. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and program product for adjusting the in-vehicle environment, which can improve the efficiency of adjusting the in-vehicle environment. The technical solution is as follows:

[0006] On one hand, a method for adjusting the in-vehicle environment is provided, the method being executed by the vehicle's control system, the vehicle including an onboard monocular camera, the method comprising:

[0007] Acquire the image of the driver captured by the vehicle-mounted monocular camera;

[0008] Based on the driver's image, the driver's emotional information and state information are identified; the emotional information is used to indicate the driver's emotions; the state information is used to indicate the driver's mental state.

[0009] Based on the emotional information and the state information, the adjustment parameters of the vehicle's interior environment are determined; the adjustment parameters include at least one of the following parameters: lighting adjustment parameters, seat adjustment parameters, and audio adjustment parameters;

[0010] The vehicle's interior environment is adjusted according to the aforementioned adjustment parameters.

[0011] On the other hand, a device for regulating the in-vehicle environment is provided, the device comprising:

[0012] The image acquisition module is used to acquire the image of the driver captured by the vehicle-mounted monocular camera;

[0013] The information recognition module is used to identify the driver's emotional information and state information based on the driver's image; the emotional information is used to indicate the driver's emotions; and the state information is used to indicate the driver's mental state.

[0014] The adjustment parameter determination module is used to determine the adjustment parameters of the vehicle's interior environment based on the emotion information and the state information; the adjustment parameters include at least one of the following parameters: lighting adjustment parameters, seat adjustment parameters, and audio adjustment parameters;

[0015] The in-vehicle environment adjustment module is used to adjust the in-vehicle environment of the vehicle according to the adjustment parameters.

[0016] In some embodiments, the adjustment parameter determination module is used to input the emotion information and the state information into a first model to obtain the adjustment parameters output by the first model;

[0017] The first model is a machine learning model trained using emotion information samples, state information samples, and labeled adjustment parameters.

[0018] In some embodiments, the control system includes a second model and a third model, the driver's image includes the driver's facial image and gesture image, and the information recognition module is used to input the facial image into the second model to obtain the driver's first emotion information and first state information output by the second model; wherein, the second model is a machine learning model trained by facial image samples, emotion information labeled on the facial image samples, and state information labeled on the facial image samples.

[0019] The gesture image is input into the third model to obtain the driver's second emotion information and second state information output by the third model; wherein, the third model is a machine learning model trained by gesture image samples, emotion information labeled on the gesture image samples, and state information labeled on the gesture image samples.

[0020] In some embodiments, the information recognition module further includes:

[0021] The second model output acquisition module is used to acquire the confidence level of the first emotion information and the confidence level of the first state information output by the second model.

[0022] The third model output acquisition module is used to acquire the confidence level of the second emotion information and the confidence level of the second state information output by the third model.

[0023] The first model input module is used to input the first emotion information, the confidence level of the first emotion information, the second emotion information, the confidence level of the second emotion information, the first state information, the confidence level of the first state information, the second state information, and the confidence level of the second state information into the first model to obtain the adjustment parameters output by the first model;

[0024] The first model is a machine learning model trained using the emotion information samples, the confidence scores of the emotion information samples, the state information samples, the confidence scores of the state information samples, and the labeled adjustment parameters.

[0025] In some embodiments, the apparatus further includes:

[0026] The voice prompt playback module is used to play a voice prompt before adjusting the vehicle's interior environment according to the adjustment parameters. The voice prompt is used to prompt the user to confirm whether to adjust the vehicle's interior environment.

[0027] The in-vehicle environment adjustment module is used to adjust the in-vehicle environment of the vehicle according to the adjustment parameters when a voice command confirming the adjustment of the in-vehicle environment is received.

[0028] In some embodiments, the emotional information is one of irritability, calmness, and pleasure; the state information is a tense state or a relaxed state.

[0029] In another aspect, a computer device is provided, the computer device comprising a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the in-vehicle environment adjustment method as described above.

[0030] In another aspect, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the in-vehicle environment adjustment method described above.

[0031] In another aspect, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the in-vehicle environment adjustment methods provided in the various optional implementations described above.

[0032] The technical solution provided in this application can include the following beneficial effects: the vehicle's onboard monocular camera can collect images of the driver in real time, and these images can be used to identify the driver's emotional and state information. The vehicle can automatically adjust the in-vehicle environment based on these two types of information. In the above solution, since the emotional and state information is obtained through subtle characteristics of the driver, it can more accurately represent the driver's current state. By determining the adjustment parameters of the in-vehicle environment through subtle information, the obtained adjustment parameters can be more precise, and the in-vehicle environment set based on these adjustment parameters is closer to the driver's needs. This effectively avoids overly general and singular adjustment effects, expands the methods for adjusting the in-vehicle environment, and improves the efficiency of in-vehicle environment adjustment.

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0035] Figure 1 This is a schematic diagram illustrating the application environment of a method for adjusting the in-vehicle environment according to an embodiment of this application;

[0036] Figure 2 This is a flowchart of a method for adjusting the in-vehicle environment according to an embodiment of this application;

[0037] Figure 3 This is a flowchart of a method for adjusting the in-vehicle environment according to an embodiment of this application;

[0038] Figure 4 This is a flowchart of a method for adjusting the in-vehicle environment according to an embodiment of this application;

[0039] Figure 5 This is a flowchart of a method for adjusting the in-vehicle environment according to an embodiment of this application;

[0040] Figure 6 This is a flowchart of a method for adjusting the in-vehicle environment according to an embodiment of this application;

[0041] Figure 7 This is a schematic diagram of an application of an emotion and state recognition neural network provided in one embodiment of this application;

[0042] Figure 8 This is a block diagram of an in-vehicle environment adjustment device provided in an exemplary embodiment of this application;

[0043] Figure 9This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0045] This application proposes a method for adjusting the in-vehicle environment. This method can determine the adjustment parameters of the vehicle's in-vehicle environment based on the driver's emotional and state information, and adjust the in-vehicle environment based on the adjustment parameters, effectively improving the adjustment efficiency of the in-vehicle environment. For ease of understanding, some concepts involved in this application are explained below.

[0046] 1) An automotive monocular camera is a camera system installed on a vehicle that uses only one lens to capture images. Automotive monocular cameras perform tasks such as object detection, distance estimation, and environmental perception by analyzing image data acquired from a single viewpoint. They are widely used in advanced driver assistance systems (ADAS) to achieve functions such as lane departure warning, forward collision warning, and pedestrian detection.

[0047] 2) A machine learning model is a mathematical model designed to learn patterns and regularities from data in order to make predictions or decisions based on new data. It automatically learns a mapping relationship from input data, linking the input to the output, using statistical and optimization methods. This mapping relationship can be linear or non-linear, depending on the algorithm and model type used.

[0048] 3) A DMS (Driver Monitoring System) camera system is a camera system specifically designed to monitor the driver's condition. This system is usually installed inside the vehicle and can observe the driver's behavior and physiological characteristics in real time, such as eye opening and closing, head posture, facial expressions, etc., to determine whether the driver is fatigued, distracted, or in other situations that may affect driving safety.

[0049] 4) A neural network is a computational model that mimics the structure of neurons in the human brain, used to handle complex pattern recognition tasks. It consists of a large number of simple processing units (called "neurons" or "nodes") interconnected by connection weights to form a multi-layered network structure. Neural networks can learn the complex relationships between input data and output, and continuously optimize these connection weights through the training process to improve the accuracy of prediction or classification.

[0050] 5) The IoU (Intersection over Union) loss function is a commonly used loss function in object detection tasks, used to evaluate the degree of overlap between predicted bounding boxes and ground truth bounding boxes. IoU calculates the ratio of the intersection area to the union area of ​​two rectangles, and its value ranges from 0 to 1. The larger the value, the more the two boxes overlap.

[0051] 6) The cross-entropy loss function is a commonly used loss function to measure the difference between the probability distribution predicted by a classification model and the actual labels. It is typically used in classification problems. The cross-entropy loss function is a tool for measuring the difference between the predicted probability distribution and the true distribution. By minimizing the loss function, machine learning models can progressively improve their predictive performance.

[0052] 7) CNN (Convolutional Neural Network) is a deep learning model that is particularly suitable for processing image data. It extracts features through convolutional layers and reduces feature dimensionality through pooling layers, thereby effectively capturing the spatial hierarchical structure of the data.

[0053] 8) Multilayer Perceptron (MLP) is a common artificial neural network structure consisting of multiple layers of neurons, including an input layer, one or more hidden layers, and an output layer. Neurons in each layer are fully connected to neurons in the next layer, enabling the network to capture complex patterns and relationships in the data.

[0054] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user-related data (such as facial and gesture images of the driver). These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user-related data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without receiving confirmation from the user), the steps for collecting user-related data end, meaning no user-related data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0055] Figure 1 This is a schematic diagram illustrating the application environment of a method for adjusting the in-vehicle environment according to an embodiment of this application. For example... Figure 1 As shown, the system includes a vehicle 100, which contains a control system 110. The control system 110 can be an in-vehicle system built into the vehicle, such as an in-vehicle navigation system, an in-vehicle entertainment system, an in-vehicle communication system, a smart home replication system, or a vehicle safety system.

[0056] The control system 110 and the vehicle 100 can be interconnected via a communication network. Optionally, this communication network can be a wired network or a wireless network.

[0057] Optionally, the aforementioned wireless or wired network uses vehicle communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to any of the following:

[0058] 1) Controller Area Network (CAN): CAN is a common in-vehicle communication network used to connect various electronic control units within the vehicle, such as the engine control unit, braking system control unit, and air conditioning control unit. CAN bus offers high transmission speed and reliability, and is used to transmit real-time data and commands within the vehicle.

[0059] 2) Local Area Network (LAN): A local area network is a network used to connect various electronic devices and systems within a vehicle, such as multimedia systems, navigation systems, and in-vehicle entertainment systems. LANs are typically based on Ethernet technology, providing high-speed data transmission and multi-device connectivity.

[0060] 3) Wireless Local Area Network (WLAN): Wireless Local Area Network refers to the function of providing wireless network connectivity inside the vehicle, enabling the driver and passengers to connect to the Internet, download data, and use online services.

[0061] 4) In-vehicle mobile communication network: In-vehicle mobile communication network refers to the mobile communication module integrated inside the vehicle, which is used to connect to mobile communication networks (such as 3G, 4G, 5G networks) and provide communication functions such as in-vehicle Internet, voice calls, and SMS.

[0062] 5) Vehicle-mounted satellite communication: Vehicle-mounted satellite communication refers to communication via satellite connection, used to provide communication services in remote areas or places without terrestrial mobile communication network coverage.

[0063] The aforementioned communication networks can be used individually or in combination to provide vehicles with various data transmission, communication, and internet connectivity functions, thereby enabling a more intelligent and convenient driving and riding experience.

[0064] In other embodiments, customized and / or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0065] exist Figure 1 In the vehicle 100, the vehicle also includes an onboard monocular camera 100a, headlights 100b, a seat 100c, and an audio system 100d. The control system 110 can acquire images of the driver (e.g., at least one of facial images and gesture images) captured by the onboard monocular camera 100a; based on the driver's image, it identifies the driver's current emotion and mental state; based on the emotion information used to indicate the driver's emotion and the state information used to indicate the driver's mental state, it determines the adjustment parameters of the vehicle's interior environment; and adjusts at least one of the headlights 100b, seat 100c, and audio system 100d according to the adjustment parameters to adjust the vehicle's interior environment.

[0066] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a method for adjusting the in-vehicle environment according to an embodiment of this application. The method for adjusting the in-vehicle environment can be executed by a control system in the vehicle, which includes an onboard monocular camera, and the computer device can be as described above. Figure 1 The control system 110 shown.

[0067] In this embodiment, the control system may include a camera capture subsystem, a strategy operation subsystem, a lighting adjustment subsystem, a seat adjustment subsystem, and an audio adjustment subsystem. The camera capture subsystem may include an in-vehicle monocular camera, and optionally, may also include a machine learning model, or the machine learning model may also be set in the strategy operation subsystem.

[0068] The above navigation information display method may include the following steps:

[0069] Step 210: Acquire the image of the driver captured by the vehicle-mounted monocular camera.

[0070] The aforementioned vehicle-mounted monocular camera is a camera capable of capturing images of a specified area of ​​the driver's body (e.g., facial images, hand images). The vehicle-mounted monocular camera can be installed above the rearview mirror of the vehicle or above the dashboard of the vehicle.

[0071] In this embodiment of the application, when the driver enters the vehicle and the vehicle is running, the monocular camera can capture the driver's image and save the captured image data in the database of the camera capture subsystem. Optionally, the control system can directly obtain the driver's image from the database, or the monocular camera can directly upload the driver's image to the control system after capturing it, and the control system can directly receive the driver's image.

[0072] Optionally, the control system periodically acquires images of the driver captured by the vehicle's monocular camera. For example, it acquires images of the driver captured by the vehicle's monocular camera every hour.

[0073] Optionally, the control system acquires an image of the driver from the onboard monocular camera when the vehicle is in a specified gear. The gear in question may include, but is not limited to, parking, reverse, neutral, drive, and low gear. For example, the control system acquires the image of the driver from the onboard monocular camera when the vehicle is in drive.

[0074] Optionally, the control system acquires images of the driver from the onboard monocular camera when the vehicle is traveling on designated road conditions. These road conditions may include, but are not limited to, urban roads, highways, rural roads, and mountain roads. For example, when the vehicle is traveling on an urban road, the control system acquires images of the driver from the onboard monocular camera.

[0075] In this embodiment of the application, the control system can acquire multiple images of the driver captured by the vehicle-mounted monocular camera at a time; or it can acquire one image of the driver captured by the vehicle-mounted monocular camera each time.

[0076] In this embodiment, the image of the driver captured by the vehicle-mounted monocular camera can reflect the driver's current state. The control system implements a method for adjusting the in-vehicle environment based on the real-time image of the driver captured by the vehicle-mounted monocular camera. This method can adjust the in-vehicle environment based on the driver's current state, effectively ensuring the accuracy of the adjustment.

[0077] Step 220: Based on the driver's image, identify the driver's emotional information and state information; the emotional information is used to indicate the driver's emotions; the state information is used to indicate the driver's mental state.

[0078] In this embodiment, after acquiring an image of the driver, the control system can automatically identify the driver's emotional and state information using a machine learning model. Specifically, the machine learning model can be one capable of generating the driver's emotional and state information upon receiving the image.

[0079] In this embodiment, emotional information and state information can intuitively reflect the driver's current state, thereby helping the control system to more accurately judge the current state of the driver. Based on the driver's emotional information and state information, the subsequent vehicle interior environment adjustment method can make the effect of the vehicle interior environment adjustment more in line with the current state required by the driver, and improve the accuracy of the vehicle interior environment adjustment.

[0080] Step 230: Based on the emotional and state information, determine the adjustment parameters of the vehicle's interior environment; the adjustment parameters include at least one of the following parameters: lighting adjustment parameters, seat adjustment parameters, and audio adjustment parameters.

[0081] In some embodiments, the vehicle's control system can determine the adjustment parameters of the vehicle's in-vehicle environment based on at least one of emotional information and state information.

[0082] In this embodiment of the application, after the vehicle's control system acquires emotional information and state information, it can determine the adjustment parameters of the vehicle's in-vehicle environment by reasoning based on a machine learning model through a strategy operation subsystem or by matching adjustment parameters in a database.

[0083] Optionally, the control system determines the adjustment parameters of the vehicle's in-vehicle environment through a strategy operation subsystem based on emotion information and state information. The strategy operation subsystem includes a machine learning model, which can take at least one of the emotion information and state information as input and output the adjustment parameters of the in-vehicle environment.

[0084] Optionally, the vehicle's control system includes at least one database for storing at least two in-vehicle environment adjustment parameters. Each adjustment parameter corresponds to an information group, which may include at least one of emotional information and state information. The vehicle's control system determines the in-vehicle environment adjustment parameters by querying the database based on the information group containing at least one of emotional and state information.

[0085] The aforementioned adjustment parameters are used to indicate parameters for adjusting specific components inside the vehicle. For example, the aforementioned adjustment parameters can indicate adjustments to the lights, seats, and audio system.

[0086] The aforementioned lighting adjustment parameters may include, but are not limited to: interior lighting brightness parameters (e.g., brightness 80%), interior lighting color parameters (e.g., RGB parameters 255, 0, 0), interior lighting color temperature parameters, interior lighting luminous efficacy parameters, and interior lighting brightness parameters.

[0087] The aforementioned seat adjustment parameters may include, but are not limited to: seat position parameters, seat massage intensity parameters, and seat ventilation function parameters (e.g., heating / ventilation).

[0088] The aforementioned audio adjustment parameters may include, but are not limited to: the volume parameters of the in-vehicle audio system, the type of music played by the in-vehicle audio system, and the type of sound effects played by the in-vehicle audio system.

[0089] In the embodiments of this application, the above-mentioned adjustment parameters may include a variety of parameters for adjusting different environmental components inside the vehicle, thereby expanding the adjustable range of the in-vehicle environment and effectively improving the efficiency of adjusting the in-vehicle environment.

[0090] Step 240: Adjust the vehicle's interior environment according to the adjustment parameters.

[0091] In this embodiment of the application, after the control system obtains the adjustment parameters, it can send the adjustment parameters to the hardware control system corresponding to the parameters, and the hardware control system can automatically perform the adjustment operation on the hardware according to the adjustment parameters.

[0092] For example, the control system can send lighting adjustment parameters to the lighting control system, which will then automatically adjust the lighting accordingly. Similarly, the control system can send seat adjustment parameters to the seat control system, which will then automatically adjust the seat accordingly. And so on, the control system can send audio adjustment parameters to the audio control system, which will then automatically adjust the audio system accordingly.

[0093] Optionally, after acquiring the adjustment parameters, the control system can send the adjustment parameters to the vehicle's instrument panel for display, so as to remind the driver to manually adjust the in-vehicle environment according to the adjustment parameters on the instrument panel.

[0094] In this embodiment, the vehicle's onboard monocular camera can capture images of the driver in real time. These images can be used to identify the driver's emotional and state information, allowing the vehicle to automatically adjust the in-vehicle environment based on this information. In this solution, since the emotional and state information is obtained through subtle characteristics of the driver, it can more accurately represent the driver's current state. By determining the adjustment parameters of the in-vehicle environment through this subtle information, the obtained adjustment parameters can be more precise, and the in-vehicle environment set based on these adjustment parameters is closer to the driver's needs. This effectively avoids overly general and singular adjustment effects, expands the methods for adjusting the in-vehicle environment, and improves the efficiency of in-vehicle environment adjustment.

[0095] In the scheme shown in the above embodiments, the emotional information is one of irritability, calmness, and pleasure; the state information is a tense state or a relaxed state.

[0096] In this embodiment, the vehicle control system can obtain different adjustment parameters based on the driver's different emotional information and different state information. More subtle emotional information and state information can make the generated adjustment parameters closer to the driver's needs, making the adjustment effect of the in-vehicle environment more likely to meet the driver's needs, thereby improving the adjustment efficiency of the in-vehicle environment.

[0097] based on Figure 2 Please refer to Figure 3 , Figure 3 This is a flowchart of a method for adjusting the in-vehicle environment according to an embodiment of this application. Figure 2 Step 230 in the above can be implemented as step 230a:

[0098] Step 230a: Input the emotion information and state information into the first model to obtain the adjustment parameters output by the first model.

[0099] The first model is a machine learning model trained using emotion information samples, state information samples, and labeled adjustment parameters.

[0100] The first model mentioned above can be trained based on the driver's emotion information samples, state information samples, and labeled adjustment parameters. For example, taking the first model as a multilayer connected neural network model, during the training process, the control system uses the driver's emotion information samples and state information samples as inputs to the multilayer connected neural network to obtain the predicted adjustment parameters output by the multilayer connected neural network. Then, it calculates the loss function value (e.g., by calculating the loss function value through the cross-entropy loss function) by the difference between the predicted adjustment parameters and the labeled adjustment parameters. The parameters of the multilayer connected neural network are updated by the loss function value to complete the training of the multilayer connected neural network, and the trained multilayer connected neural network is used as the first model.

[0101] In this embodiment of the application, the aforementioned emotion information samples, state information samples, and labeled adjustment parameters can be collected and obtained by the trainers of the first model.

[0102] Optionally, the aforementioned emotion information samples, state information samples, and labeled adjustment parameters can be historical data of the current driver. Specifically, for example, if a driver was detected manually performing an adjustment operation in the past, the adjustment parameters corresponding to that operation are obtained as labeled adjustment parameters, and the emotion and state information obtained from the driver's image recognition at that time are used as emotion information samples and state information samples. It should be noted that the samples obtained in this way are only matched with a specified driver; that is, different drivers will obtain different emotion and state information samples.

[0103] In this embodiment of the application, the vehicle control system can input the acquired driver's emotional information and state information into a first model with the ability to generate adjustment parameters. The first model processes the input information to obtain the adjustment parameters corresponding to the input information. For example, at least one of the driver's emotional information and state information can be input into a multilayer connected neural network to obtain the adjustment parameters output by the multilayer connected neural network.

[0104] In this embodiment, the vehicle control system outputs adjustment parameters through the first model, which expands the way adjustment parameters are obtained. The adjustment parameters generated by the trained first model are more in line with the current driver's needs and can effectively improve the adjustment efficiency of in-vehicle adjustment parameters.

[0105] Based on the solutions shown in the above embodiments, in some embodiments, the control system includes a second model and a third model, and the driver's image includes the driver's facial image and gesture image.

[0106] In this embodiment of the application, the control system can obtain the current driver's facial expression through the aforementioned facial image, obtain the current driver's hand movements through the aforementioned gesture image, and obtain emotional information and state information by analyzing the current driver's facial expression and hand movements.

[0107] Please refer to Figure 4 , Figure 4 This is a flowchart of a method for adjusting the in-vehicle environment according to an embodiment of this application. Figure 2 Step 220 can be implemented as steps 220a and 220b:

[0108] Step 220a: Input the facial image into the second model to obtain the driver's first emotion information and first state information output by the second model; wherein, the second model is a machine learning model trained by facial image samples, emotion information labeled on the facial image samples, and state information labeled on the facial image samples.

[0109] The second model mentioned above can be trained based on facial image samples, emotional information labeled on the facial image samples, and state information labeled on the facial image samples. For example, taking the second model as a CNN neural network model, during the training process, the control system uses the driver's facial image samples as input to the CNN neural network to obtain the predicted emotional information and state information output by the CNN neural network. Then, it calculates the loss function value (e.g., by calculating the loss function value through the IoU loss function) by the difference between the predicted emotional information and state information and the labeled emotional information and state information. The parameters of the CNN neural network are updated by the loss function value to complete the training of the CNN neural network, and the trained CNN neural network is used as the second model.

[0110] In this embodiment of the application, the vehicle control system can input the acquired driver's facial image into a second model with the ability to generate first emotion information and first state information. The second model processes the input facial image to obtain the first emotion information and first state information corresponding to the input facial image. For example, the driver's facial image can be input into a CNN neural network to obtain the driver's first emotion information and first state information output by the CNN neural network.

[0111] For example, the vehicle's control system can input facial images into a CNN neural network to obtain the driver's first emotion information and first state information output by the CNN neural network; wherein, the CNN neural network is a machine learning model trained by facial image samples, emotion information labeled on the facial image samples, and state information labeled on the facial image samples.

[0112] Step 220b: Input the gesture image into the third model to obtain the driver's second emotion information and second state information output by the third model; wherein, the third model is a machine learning model trained by gesture image samples, emotion information labeled on gesture image samples, and state information labeled on gesture image samples.

[0113] The aforementioned third model can be trained based on gesture image samples, emotional information labeled on the gesture image samples, and state information labeled on the gesture image samples. For example, taking the aforementioned third model as a CNN neural network model, during the training process, the control system uses the driver's gesture image samples as input to the CNN neural network to obtain the predicted emotional information and state information output by the CNN neural network. Then, it calculates the loss function value (e.g., by calculating the loss function value through the IoU loss function) based on the difference between the predicted emotional information and state information and the labeled emotional information and state information. The parameters of the CNN neural network are updated based on the loss function value to complete the training of the CNN neural network, and the trained CNN neural network is used as the third model.

[0114] In this embodiment of the application, the vehicle control system can input the acquired driver's gesture image into a third model that has the ability to generate second emotion information and second state information. The third model processes the input gesture image to obtain the second emotion information and second state information corresponding to the input gesture image. For example, the driver's gesture image can be input into a CNN neural network to obtain the driver's second emotion information and second state information output by the CNN neural network.

[0115] For example, the vehicle's control system can input gesture images into a CNN neural network to obtain the driver's second emotion information and second state information output by the CNN neural network; wherein, the CNN neural network is a machine learning model trained by gesture image samples, emotion information labeled on gesture image samples, and state information labeled on gesture image samples.

[0116] In this embodiment, the vehicle control system can obtain the driver's emotional and state information through the driver's facial and gesture images, thus expanding the methods for obtaining emotional and state information. Furthermore, since emotional and state information can be obtained through different images, it can effectively avoid the situation where emotional and state information cannot be obtained through a single type of image, effectively ensuring the subsequent acquisition steps of in-vehicle environmental adjustment parameters, expanding the methods for adjusting the in-vehicle environment, and improving the efficiency of in-vehicle environment adjustment.

[0117] based on Figure 4In the corresponding embodiment, the vehicle control system can also obtain the confidence level of the first emotion information and the confidence level of the first state information output by the second model; and obtain the confidence level of the second emotion information and the confidence level of the second state information output by the third model.

[0118] In this embodiment of the application, after receiving a facial image, the second model outputs multiple different probabilities. Each probability indicates the probability that the current facial image belongs to a certain type of emotion or a certain type of state. Different probabilities indicate different types of emotions or different types of states. After receiving the output of the second model, the control system determines the largest probability from the multiple probabilities used to indicate the emotion type as the confidence level of the first emotion information and uses the emotion corresponding to that probability as the first emotion information; it also determines the largest probability from the multiple probabilities used to indicate the state type as the confidence level of the first state information and uses the state corresponding to that probability as the first state information.

[0119] In this embodiment of the application, after receiving a gesture image, the third model outputs multiple different probabilities. Each probability indicates the probability that the current gesture image belongs to a certain type of emotion or a certain type of state. Different probabilities indicate different types of emotions or different types of states. After receiving the output of the third model, the control system determines the largest probability from the multiple probabilities used to indicate the emotion type as the confidence level of the second emotion information and uses the emotion corresponding to that probability as the second emotion information; it also determines the largest probability from the multiple probabilities used to indicate the state type as the confidence level of the second state information and uses the state corresponding to that probability as the second state information.

[0120] Please refer to Figure 5 , Figure 5 This is a flowchart of a method for adjusting the in-vehicle environment according to an embodiment of this application. It includes step 220c:

[0121] Step 220c: Input the first emotion information, the confidence level of the first emotion information, the second emotion information, the confidence level of the second emotion information, the first state information, the confidence level of the first state information, the second state information, and the confidence level of the second state information into the first model to obtain the adjustment parameters output by the first model.

[0122] The first model is a machine learning model trained using emotion information samples, the confidence level of emotion information samples, state information samples, the confidence level of state information samples, and labeled adjustment parameters.

[0123] The first model mentioned above can be trained based on emotion information samples, the confidence level of emotion information samples, state information samples, the confidence level of state information samples, and labeled adjustment parameters. For example, taking the first model as a multilayer connected neural network model, during the training process, the control system uses the driver's emotion information samples, the confidence level of emotion information samples, state information samples, and the confidence level of state information samples as inputs to the multilayer connected neural network to obtain the predicted adjustment parameters output by the multilayer connected neural network. Then, the loss function value is calculated by the difference between the predicted adjustment parameters and the labeled adjustment parameters (for example, by calculating the loss function value through the cross-entropy loss function). The parameters of the multilayer connected neural network are updated by the loss function value to complete the training of the multilayer connected neural network, and the trained multilayer connected neural network is used as the first model.

[0124] In this embodiment of the application, the vehicle control system can input the acquired first emotion information, the confidence level of the first emotion information, the second emotion information, the confidence level of the second emotion information, the first state information, the confidence level of the first state information, the second state information, and the confidence level of the second state information into a first model with the ability to generate adjustment parameters. The first model processes the input information to obtain the adjustment parameters corresponding to the input information.

[0125] For example, the in-vehicle navigation system inputs the driver's corresponding first emotion information, the confidence level of the first emotion information, the second emotion information, the confidence level of the second emotion information, the first state information, the confidence level of the first state information, the second state information, and the confidence level of the second state information into a multilayer connected neural network to obtain the adjustment parameters output by the multilayer connected neural network. The multilayer connected neural network is a machine learning model trained using the driver's first emotion information sample, the confidence level of the first emotion information sample, the second emotion information sample, the confidence level of the second emotion information sample, the first state information sample, the confidence level of the first state information sample, the second state information sample, the confidence level of the second state information sample, and the labeled adjustment parameters.

[0126] In this embodiment, the vehicle control system can obtain in-vehicle environmental adjustment parameters based on the driver's emotional information, the confidence level of the emotional information, the state information, and the confidence level of the state information through a first model. By using the confidence level, it can be ensured that the obtained emotional and state information is more accurate and closer to the current driver's real state. The in-vehicle environment set based on these adjustment parameters is closer to the driver's needs, effectively avoiding overly general and singular adjustment effects, expanding the methods for adjusting the in-vehicle environment, and improving the efficiency of in-vehicle environment adjustment.

[0127] Based on the solutions shown in the above embodiments, before the vehicle control system adjusts the vehicle's interior environment according to the adjustment parameters, the vehicle control system plays a voice prompt to ask the user to confirm whether to adjust the vehicle's interior environment; upon receiving a voice command confirming the adjustment of the vehicle's interior environment, the vehicle's interior environment is adjusted according to the adjustment parameters.

[0128] In this embodiment, after the vehicle's control system obtains the adjustment parameters, it controls the in-vehicle audio system to play a voice prompt, asking the user whether they need to adjust the vehicle's interior environment. The user can directly answer "yes" or "no" to provide feedback to the control system. When the vehicle's control system receives a voice command confirming the adjustment of the vehicle's interior environment, it adjusts the vehicle's interior environment according to the adjustment parameters; otherwise, it does not adjust the vehicle's interior environment.

[0129] Optionally, after the vehicle's control system obtains the adjustment parameters, it controls the in-vehicle audio system to play voice prompts while simultaneously displaying a prompt text on the vehicle's central control screen. The prompt text asks the user whether they need to adjust the vehicle's interior environment, and there are two options below the prompt text: "Yes" and "No." The user can provide feedback to the control system by touching one of the options. If the vehicle's control system receives confirmation of the option to adjust the vehicle's interior environment, it will adjust the vehicle's interior environment according to the adjustment parameters; otherwise, it will not adjust the vehicle's interior environment.

[0130] It should be noted that the above-mentioned users are not limited to the driver, but can also be other passengers in the vehicle.

[0131] In this embodiment of the application, voice prompts are used to confirm with the user whether to adjust the vehicle's interior environment, which can effectively avoid unnecessary adjustment operations, save vehicle resource consumption, and ensure the efficiency of adjusting the interior environment.

[0132] Based on the above Figures 2 to 5 The embodiments of this application illustrate a method and apparatus for identifying driver status using an in-vehicle monocular camera.

[0133] For example, please refer to Figure 6 The diagram illustrates a flowchart of a method for adjusting the in-vehicle environment according to an embodiment of this application.

[0134] The vehicle's control system first captures an image of the driver using a monocular camera, inputs the image into a recognition neural network, identifies the emotion and state values, and then inputs the emotion and state values ​​into a strategy output neural network. Based on the strategy output neural network, it obtains parameters for headlight adjustment, seat adjustment, and audio adjustment. Finally, based on these adjustment parameters, it adjusts the vehicle's interior environment.

[0135] The training process for the policy output neural network is as follows: First, the adjustment parameters of the driver when adjusting the lights, seat and audio within a specified time period are obtained as sample data. Then, the driver's state is used as a sample input to the policy output neural network. The adjustment parameters predicted by the policy output neural network are compared with the adjustment parameter samples, and the loss function is updated to update the training policy output neural network.

[0136] For example, please refer to Figure 7 The diagram illustrates an application of an emotion and state recognition neural network according to an embodiment of this application.

[0137] In this embodiment, the vehicle's control system can recognize image data captured by a monocular camera through a facial feature point recognition neural network and a gesture recognition neural network to obtain, for example... Figure 7 The diagram shows three emotions and two states.

[0138] In this embodiment, the facial feature point recognition neural network and gesture recognition neural network can exist in the DMS camera system, and the strategy output neural network can exist in the vehicle system. Through the mutual coordination and communication of the vehicle system, the effect of automatic control of the vehicle's lights, seats, and audio can be achieved.

[0139] In this embodiment of the application, the vehicle control system monitors the driver's state and adjusts the in-vehicle environment to enhance the driver's driving experience; using a monocular camera to monitor the driver's state reduces costs; and automatically collecting driver data and updating the strategy effectively ensures the driver's driving experience.

[0140] Although preferred embodiments of this embodiment have been disclosed for illustrative purposes, those skilled in the art will recognize that various modifications, additions, and substitutions are possible without departing from the scope and spirit of the embodiments disclosed in the appended claims.

[0141] The above is merely one embodiment of this application and should not be considered as a limitation thereof. Those skilled in the art will understand that various modifications and variations can be made to the embodiments to adapt to different application requirements. Therefore, the scope of this application should be defined by the claims appended to the claims.

[0142] Please refer to Figure 8 The diagram illustrates a block diagram of an in-vehicle environment adjustment device provided in an exemplary embodiment of this application. This in-vehicle environment adjustment device can be implemented as all or part of a computer device through hardware or a combination of hardware and software, to achieve the above-described... Figures 2 to 5 All or part of the steps in the illustrated embodiments. For example... Figure 8 As shown, the in-vehicle environment regulation device includes:

[0143] Image acquisition module 801 is used to acquire images of the driver captured by the vehicle-mounted monocular camera;

[0144] The information recognition module 802 is used to recognize the driver's emotional information and state information based on the driver's image; the emotional information is used to indicate the driver's emotions; and the state information is used to indicate the driver's mental state.

[0145] The adjustment parameter determination module 803 is used to determine the adjustment parameters of the vehicle's interior environment based on emotional information and state information; the adjustment parameters include at least one of the following parameters: lighting adjustment parameters, seat adjustment parameters, and audio adjustment parameters;

[0146] The in-vehicle environment adjustment module 804 is used to adjust the in-vehicle environment according to the adjustment parameters.

[0147] In some embodiments, the adjustment parameter determination module 803 is used to input emotion information and state information into the first model to obtain the adjustment parameters output by the first model;

[0148] The first model is a machine learning model trained using emotion information samples, state information samples, and labeled adjustment parameters.

[0149] In some embodiments, the control system includes a second model and a third model. The driver's image includes the driver's facial image and gesture image. The information recognition module 802 is used to input the facial image into the second model to obtain the driver's first emotion information and first state information output by the second model. The second model is a machine learning model trained by facial image samples, emotion information labeled on the facial image samples, and state information labeled on the facial image samples.

[0150] The gesture image is input into the third model to obtain the driver's second emotion information and second state information output by the third model; wherein, the third model is a machine learning model trained by gesture image samples, emotion information labeled on gesture image samples, and state information labeled on gesture image samples.

[0151] In some embodiments, the information recognition module 802 further includes:

[0152] The second model output acquisition module is used to acquire the confidence level of the first emotion information and the confidence level of the first state information output by the second model.

[0153] The third model output acquisition module is used to acquire the confidence level of the second emotion information and the confidence level of the second state information output by the third model.

[0154] The first model input module is used to input the first emotion information, the confidence level of the first emotion information, the second emotion information, the confidence level of the second emotion information, the first state information, the confidence level of the first state information, the second state information, and the confidence level of the second state information into the first model to obtain the adjustment parameters output by the first model.

[0155] The first model is a machine learning model trained using emotion information samples, the confidence level of emotion information samples, state information samples, the confidence level of state information samples, and labeled adjustment parameters.

[0156] In some embodiments, the apparatus further includes:

[0157] The voice prompt playback module is used to play voice prompts before adjusting the vehicle's interior environment according to the adjustment parameters. The voice prompts are used to ask the user to confirm whether to adjust the vehicle's interior environment.

[0158] The in-vehicle environment adjustment module 804 is used to adjust the in-vehicle environment according to adjustment parameters upon receiving a voice command confirming the adjustment of the in-vehicle environment.

[0159] In some embodiments, the emotional information is one of irritability, calmness, and pleasure; the state information is a tense state or a relaxed state.

[0160] Please refer to Figure 9 , Figure 9 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. The computer device 900 includes a Central Processing Unit (CPU) 901, a system memory 904 including Random Access Memory (RAM) 902 and Read-Only Memory (ROM) 903, and a system bus 905 connecting the system memory 904 and the CPU 901. The computer device 900 also includes a basic input / output system (I / O system) 906 that facilitates the transfer of information between various devices within the computer, and a mass storage device 907 for storing the operating system 913, application programs 914, and other program modules 915.

[0161] The basic input / output system 906 includes a display 908 for displaying information and an input device 909 for user input, such as a mouse or keyboard. Both the display 908 and the input device 909 are connected to the central processing unit 901 via an input / output controller 910 connected to the system bus 905. The basic input / output system 906 may also include the input / output controller 910 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 910 also provides output to a display screen, printer, or other types of output devices.

[0162] The mass storage device 907 is connected to the central processing unit 901 via a mass storage controller (not shown) connected to the system bus 905. The mass storage device 907 and its associated computer-readable media provide non-volatile storage for the computer device 900. That is, the mass storage device 907 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0163] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM (Random Access Memory), ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 904 and mass storage device 907 described above can be collectively referred to as memory.

[0164] Computer device 900 can be connected to the Internet or other network devices via network interface unit 911 connected to the system bus 905.

[0165] The memory also includes one or more programs, which are stored in the memory, and the central processing unit 901 implements these programs. Figures 2 to 5 All or some of the steps in the method shown.

[0166] In an exemplary embodiment, a chip is also provided, the chip including programmable logic circuitry and / or program instructions, which, when the chip is run on a computer device, are used to implement all or part of the steps of the methods shown in the above embodiments of this application.

[0167] In an exemplary embodiment, a computer program product is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement all or part of the steps of the methods shown in the above embodiments of this application.

[0168] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores a computer program that is loaded and executed by a processor to implement all or part of the steps of the methods shown in the above embodiments of this application.

[0169] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0170] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0171] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for adjusting the in-vehicle environment, characterized in that, The method is executed by the vehicle's control system, the vehicle including an onboard monocular camera, and the method includes: The driver's image is acquired by the vehicle-mounted monocular camera, and the driver's image includes the driver's facial image and gesture image; The facial image is input into a second model to obtain the driver's first emotion information and first state information output by the second model; the gesture image is input into a third model to obtain the driver's second emotion information and second state information output by the third model; the emotion information is used to indicate the driver's emotion; the state information is used to indicate the driver's mental state. Obtain the confidence level of the first emotion information and the confidence level of the first state information output by the second model; Obtain the confidence level of the second emotion information and the confidence level of the second state information output by the third model; The first emotion information, the confidence level of the first emotion information, the second emotion information, the confidence level of the second emotion information, the first state information, the confidence level of the first state information, the second state information, and the confidence level of the second state information are input into the first model to obtain the adjustment parameters output by the first model; the adjustment parameters include at least one of the following parameters: lighting adjustment parameters, seat adjustment parameters, and audio adjustment parameters; The vehicle's interior environment is adjusted according to the aforementioned adjustment parameters.

2. The method according to claim 1, characterized in that, The second model is a machine learning model trained using facial image samples, emotional information labeled on the facial image samples, and state information labeled on the facial image samples; the third model is a machine learning model trained using gesture image samples, emotional information labeled on the gesture image samples, and state information labeled on the gesture image samples.

3. The method according to claim 2, characterized in that, The first model is a machine learning model trained using emotion information samples, the confidence level of the emotion information samples, state information samples, the confidence level of the state information samples, and labeled adjustment parameters.

4. The method according to any one of claims 1 to 3, characterized in that, Before adjusting the vehicle's interior environment according to the adjustment parameters, the method further includes: Play a voice prompt, which is used to prompt the user to confirm whether to adjust the vehicle's interior environment; Adjusting the vehicle's interior environment according to the adjustment parameters includes: Upon receiving a voice command confirming the adjustment of the vehicle's interior environment, the vehicle's interior environment is adjusted according to the adjustment parameters.

5. The method according to any one of claims 1 to 3, characterized in that, The emotional information is one of irritability, calmness, and pleasure; The state information refers to a tense state or a relaxed state.

6. A device for regulating the in-vehicle environment, characterized in that, The device includes: An image acquisition module is used to acquire images of the driver captured by an in-vehicle monocular camera, the driver's images including facial images and gesture images; An information recognition module is used to input the facial image into a second model to obtain the driver's first emotion information and first state information output by the second model; input the gesture image into a third model to obtain the driver's second emotion information and second state information output by the third model; the state information is used to indicate the driver's mental state; The second model output acquisition module is used to acquire the confidence level of the first emotion information and the confidence level of the first state information output by the second model. The third model output acquisition module is used to acquire the confidence level of the second emotion information and the confidence level of the second state information output by the third model. The first model input module is used to input the first emotion information, the confidence level of the first emotion information, the second emotion information, the confidence level of the second emotion information, the first state information, the confidence level of the first state information, the second state information, and the confidence level of the second state information into the first model to obtain the adjustment parameters output by the first model; the adjustment parameters include at least one of the following parameters: lighting adjustment parameters, seat adjustment parameters, and audio adjustment parameters; The in-vehicle environment adjustment module is used to adjust the in-vehicle environment according to the adjustment parameters.

7. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing instructions which are executed by the processor to implement the method for adjusting the in-vehicle environment as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores instructions that are executed by a processor of a computer device to implement the in-vehicle environment adjustment method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the method for adjusting the in-vehicle environment as described in any one of claims 1 to 5.

Images