Vehicle control method, electronic equipment, vehicle, medium and product

Abstract

The invention discloses a vehicle control method, electronic equipment, a vehicle, a computer readable storage medium and a computer program product. The method comprises the steps that a vehicle is controlled according to psychological state detection information of a main driving user in a vehicle cabin space, and the psychological state detection information is determined according to behavior state information of each user in the vehicle cabin space. Therefore, the main driving psychological state detection information can be determined according to the behavior state information of each user in the cabin space of the vehicle, the vehicle is controlled, such as the related parameters of the vehicle are adjusted, and the vehicle control effect can reach the expectation to a certain extent. The main driving psychological state detection information is determined according to the behavior state information of each user in the vehicle cabin space, so that the main driving psychological state detection information can be associated with the behavior state information of the passengers in the vehicle cabin space, and the effectiveness and reliability of the main driving psychological state detection information can be guaranteed; the situation that the reliability degree of the psychological state detection result of the main driving user is low can be improved to a certain extent.

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a vehicle control method, electronic equipment, a vehicle, a computer-readable storage medium, and a computer program product. Background Art

[0002] In related technologies, vehicles can predict a driver's emotions based on behavioral information such as facial expressions and voice tone, and adjust relevant parameters such as maximum longitudinal acceleration accordingly. However, because a user's psychological state is influenced by many factors and may not be expressed through facial expressions or voice tone, the accuracy of psychological state detection based on driver behavioral information is limited, resulting in difficulties in achieving the desired vehicle control effect. Summary of the Invention

[0003] The present application provides a vehicle control method, an electronic device, a vehicle, a computer-readable storage medium, and a computer program product.

[0004] An embodiment of the present application provides a vehicle control method, the method comprising:

[0005] The vehicle is controlled according to psychological state detection information of a main driver in a vehicle cabin space, wherein the psychological state detection information is determined according to behavioral state information of each user in the vehicle cabin space.

[0006] Thus, in the embodiments of the present application, the vehicle can be controlled, such as by adjusting relevant vehicle parameters, based on the driver's psychological state detection information determined from the behavioral state information of each user in the vehicle cabin, so that the desired vehicle control effect can be achieved to a certain extent. Furthermore, because the driver's psychological state detection information is determined based on the behavioral state information of each user in the vehicle cabin, the driver's psychological state detection information can be correlated with the behavioral state information of the driver and non-driver users in the vehicle cabin. This ensures the validity and reliability of the driver's psychological state detection information, and to a certain extent improves the reliability of the driver's psychological state detection results, which can be low when the driver's psychological state is not expressed through facial expressions or tone of voice.

[0007] In certain embodiments, the method further comprises:

[0008] determining a first mental state prediction of the main driver user based on first behavioral state information of the main driver user in the vehicle cabin space;

[0009] determining a second mental state prediction of the non-driving user based on the second behavioral state information of the non-driving user in the vehicle cabin space;

[0010] The mental state detection information of the primary driver is determined based on the first mental state prediction and the second mental state prediction.

[0011] In this way, by predicting and processing the driver's first behavioral state information to obtain a first mental state prediction, and predicting and processing the passenger's second behavioral state information to obtain a second mental state prediction, the first and second mental state predictions are collaboratively processed to determine the driver's mental state detection information. This comprehensive assessment of the impact of the in-vehicle environment on the driver allows for more accurate driver mental state detection information, providing more reliable mental state detection information as an evaluation basis for vehicle control.

[0012] In some embodiments, the first behavioral status information includes at least one of facial movement data, voiceprint data, steering wheel usage status data, and seat pressure data.

[0013] In this way, by collecting the driver's multi-dimensional behavioral state, the first behavioral state information is determined. Multi-dimensional behavioral data, such as facial movement data (e.g., eyelid closure, mouth corner curvature, and other indicators of fatigue and emotion) acquired through infrared cameras, voiceprint data (speech spectrum and intonation changes) acquired through microphones, steering wheel usage (grip strength and steering characteristics) acquired through torque sensors, and sitting posture information (body posture changes) acquired through pressure sensors, can provide a more reliable basis for decision-making for vehicle control.

[0014] In some embodiments, the first behavioral status information includes at least one of facial movement data, voiceprint data, steering wheel usage status data, and seat pressure data.

[0015] In this way, the second behavioral state information is determined by collecting multi-dimensional behavioral data of the driver. This multi-dimensional behavioral data, such as facial movement data (e.g., eyelid closure, mouth curvature, and other indicators of fatigue and emotion) acquired through infrared cameras, voiceprint data (speech spectrum and intonation changes) acquired through microphones, and sitting posture information (body posture changes) acquired through pressure sensors, can provide a more reliable basis for vehicle control decisions.

[0016] In some embodiments, determining the first mental state prediction of the main driver user based on the first behavioral state information of the main driver user in the vehicle cabin space includes:

[0017] Building a model based on the first behavior state information and the pre-trained first feature, and determining first feature data corresponding to the first behavior state information;

[0018] Determine the first mental state prediction based on the first feature data and a pre-trained decoder model.

[0019] In this way, a model is constructed based on the first behavioral state information and the pre-trained first feature, determining the first feature data corresponding to the first behavioral state information. A first psychological state prediction is then determined based on the first feature data and the pre-trained decoder model. In this way, the first feature construction model extracts features from the driver's multi-dimensional behavioral data, such as facial expressions, voiceprint characteristics, and steering wheel grip, to obtain high-order temporal features that characterize their behavioral state. The decoder model decodes and predicts the state of these feature data, predicting the driver's psychological state change trend within a preset timeframe. By establishing a complete mapping chain from behavioral signals to psychological states, the vehicle's ability to predict and respond to changes in the driver's emotions is significantly improved.

[0020] In some embodiments, the first feature construction model includes a plurality of first time convolution modules connected in sequence, and the plurality of first time convolution modules are configured to perform multiple feature extractions and feature filtering on the first behavior state information to determine the first feature data.

[0021] In this way, multiple first temporal convolution modules perform convolution processing on the first behavioral state information, determining highly refined and semantically rich first feature data. Each temporal convolution module then performs deeper feature abstraction and filtering based on the primary features extracted by the preceding module, gradually removing data noise and strengthening the expression of key features, providing high-quality feature input for subsequent psychological state prediction.

[0022] In some embodiments, each of the first temporal convolution modules includes a first causal convolution layer and a first gated linear unit connected in sequence;

[0023] The first causal convolution layer is configured to perform feature extraction processing on first received information to determine a first processing result, wherein the first received data includes first behavior state information and / or includes a module output result of the first prior temporal convolution module;

[0024] The first gated linear unit is configured to filter the first processing result to determine a module output result of the first temporal convolution module.

[0025] In this way, by cascading multiple first-time convolution modules to perform multi-level feature extraction on driver behavior data, high-level features representing the driver's psychological state can be obtained. Combining causal convolution and gating mechanisms, noise can be filtered while maintaining temporal dependencies. By performing multi-level feature extraction on behavioral data such as facial expressions, voiceprints, and steering wheel operations through multiple cascaded first-time convolution modules, key temporal features representing the driver's psychological state can be gradually extracted. Causal convolution and gating mechanisms are employed within each first-time convolution module to effectively filter out noise interference while maintaining temporal dependencies. Through the stacking of modules and feature refinement, a purer and more easily decodable high-level feature representation is provided to the subsequent decoder. The resulting precise features are then fed into the decoder model, enabling accurate prediction of the driver's psychological state. This technical solution, based on deep temporal feature extraction, significantly improves the vehicle's ability to understand and respond to the driver's state by establishing a complete conversion chain from raw behavioral data to high-level psychological features.

[0026] In some embodiments, the decoder model is configured as follows:

[0027] Constructing an observation probability matrix and a state transition matrix based on the first feature data;

[0028] The first psychological state prediction is determined according to the observation probability matrix, the state transition matrix, and a plurality of predetermined user psychological states.

[0029] In this way, the decoder model processes the first feature data to determine an observation probability matrix and a state transition matrix. The observation probability matrix, state transition matrix, and user psychological state are then processed to determine a first psychological state prediction. In this way, the driver's psychological state can be accurately predicted using the observation probability matrix, which describes the probabilistic relationship between the user's psychological state and the first feature data, and the state transition matrix, which quantifies the dynamic evolution of psychological states, along with multiple predetermined user psychological states. The obtained first psychological state prediction more accurately describes the probabilistic relationship between the observation data and the psychological state, allowing the model to more reliably infer the driver's psychological state based on the current observation data, thereby providing more valuable information for subsequent intelligent driving decisions.

[0030] In some embodiments, determining the predicted second mental state of the non-driver user based on the second behavioral state information of the non-driver user in the vehicle cabin space includes:

[0031] Building a model based on the second behavior state information and the pre-trained second feature, and determining second feature data corresponding to the second behavior state information;

[0032] Determine a prediction of the second mental state based on the second feature data and a pre-trained classifier model.

[0033] In this way, a model is constructed based on the second behavioral state information and the pre-trained second feature to determine the second feature data corresponding to the second behavioral state information; and a second psychological state prediction is determined based on the second feature data and the pre-trained classifier model. In this way, the second feature construction model performs deep feature extraction on multi-dimensional behavioral data such as the passenger's facial expressions, voiceprint characteristics, and sitting posture, obtaining high-order temporal features that characterize changes in their psychological state. The classifier model is then used to classify and predict the state of this feature data, predicting the passenger's future psychological state trends. By establishing a complete mapping chain from behavioral signals to psychological states, the vehicle's ability to predict and respond to changes in the driver's emotions is significantly improved.

[0034] In some embodiments, the second behavioral state information includes multiple sub-state information, and the second feature construction model includes multiple second temporal convolution modules connected in sequence;

[0035] The second time convolution module is configured to perform feature extraction and feature filtering on each type of sub-state information to determine the second feature data corresponding to the second time convolution module.

[0036] In this way, multiple second temporal convolution modules are used to perform convolution processing on the sub-state information of the second behavioral state information. After multiple feature extractions, highly refined and semantically rich second feature data is determined, providing accurate and easily classified input data for the cascaded classifier. In this way, each temporal convolution module performs deeper feature abstraction and filtering based on the primary features extracted by the previous module, gradually removing data noise and strengthening the expression of key features. This provides high-quality feature input for subsequent psychological state prediction.

[0037] In some embodiments, each of the second temporal convolution modules includes a second causal convolution layer and a second gated linear unit connected in sequence;

[0038] The second causal convolution layer is configured to perform depthwise convolution and pointwise convolution processing on each type of second received information according to a predetermined dilation factor to determine a second processing result;

[0039] The second gated linear unit is configured to filter the second processing result to determine a module output result of the second temporal convolution module.

[0040] In this way, by cascading multiple second causal convolution layers, multi-level feature extraction is performed on passenger behavior data to obtain high-level features that represent the passenger's psychological state. By combining causal convolution with a gating mechanism, multiple cascaded second temporal convolution modules are used to group and extract multi-level features from behavioral data such as facial expressions, voiceprints, and sitting postures, reducing computational effort and improving computational efficiency, gradually extracting key temporal features that represent the psychological state. Causal convolution and a gating mechanism are employed in each temporal convolution module to effectively filter out noise interference while ensuring temporal dependencies. Through the stacking of modules and feature refinement, a purer and more easily classified high-level feature representation is provided to the subsequent classifier. The resulting precise features are then input into the classifier model, enabling accurate prediction of the passenger's psychological state. This technical solution, based on deep temporal feature extraction, significantly improves the vehicle's ability to understand and respond to passenger status by establishing a complete conversion chain from raw behavioral data to high-level psychological features.

[0041] In some embodiments, the classification model includes a pooling layer, a fully connected layer, and a classification layer connected in sequence;

[0042] The pooling layer is configured to perform downsampling processing on the second feature data and determine a downsampling processing result;

[0043] The fully connected layer is configured to determine, based on the downsampling processing result, mapping data of the downsampling processing result in each predetermined user psychological state;

[0044] The classification layer is configured to determine a second mental state prediction based on the mapped data.

[0045] Thus, based on the classifier model, the pooling layer first downsamples the second feature data, compressing and extracting its features to determine the downsampling result. Next, the fully connected layer performs mapping processing, transforming from feature space to semantic space and determining the mapping result. Finally, the classification layer transforms the mapping result to determine the probability of each state in the second psychological state prediction quantification. In this way, through the pooling, fully connected, and classification layers of the classifier model, the raw state feature data can be transformed into understandable psychological states after multi-layer processing, providing accurate and reliable prediction results for subsequent dynamic adjustment of cabin parameters (such as lowering the air conditioning temperature or playing refreshing music based on "fatigue" status).

[0046] In some embodiments, determining the mental state detection information of the primary driver based on the first mental state prediction and the second mental state prediction includes:

[0047] The mental state detection information is determined according to the first mental state prediction, the second mental state prediction and a preset coefficient.

[0048] In this way, the mental state detection information is determined by combining the first mental state prediction, the second mental state prediction, and the preset coefficient. This allows for a more comprehensive and accurate reflection of the driver's actual mental state in the entire driving environment.

[0049] In certain embodiments, the method further comprises:

[0050] The preset coefficient is determined according to the second psychological state prediction and a preset influencing factor parameter, wherein the preset influencing factor parameter is determined according to the first behavioral state information and the second behavioral state information.

[0051] In this way, a preset coefficient can be determined based on the computational processing of the second psychological state prediction and the preset influencing factor parameters. By integrating the second psychological state prediction (e.g., the passenger's psychological state) with preset influencing factors (e.g., the coordinated attention weights of driver and passenger behaviors and the environmental complexity parameter), the mutual influence of psychological states in multi-occupant scenarios can be accurately quantified.

[0052] In certain embodiments, the method further comprises:

[0053] determining a query vector according to the first behavior state information;

[0054] Determining a key vector and a value vector according to the second behavior state information;

[0055] The preset impact factor parameter is determined according to the query vector, the key vector, and the value vector.

[0056] In this way, based on aligning input features and calculating multi-head collaborative attention, the query vector can be determined by extracting and processing the features of the first behavioral state information. Secondly, the key vector is determined by extracting and processing the features of the second behavioral state information, and the value vector is determined by extracting and processing the specific features of the second behavioral state information. Finally, weights are generated by the matching degree between the query vector and the key vector, and the value vectors are weighted and summed to obtain the final impact factor. In this way, by identifying the temporal dependencies of cross-object behaviors (such as the synchronization of driver operations and passenger actions) in real time, the one-sidedness of relying solely on the features of a single object is avoided. At the same time, through the dynamic calculation of attention weights, it can adapt to complex scenarios with multiple passengers and multiple behaviors (such as multi-person conversations, children crying, etc.), providing an accurate basis for subsequent psychological state prediction and vehicle control.

[0057] In some embodiments, controlling the vehicle based on the detection information of the psychological state of the main driver in the vehicle cabin space includes:

[0058] Determine target control parameters of the vehicle based on the mental state detection information.

[0059] In this way, the target control parameters for controlling the vehicle are determined based on the psychological state detection information. In this way, by dynamically adjusting the driver's psychological state detection information in real time, precise control of the vehicle state is achieved, improving driving safety, comfort, and user experience.

[0060] In some embodiments, the target control parameter includes at least one of a vehicle following parameter, a lane change frequency, a steering wheel sensitivity, and a vehicle acceleration constraint parameter.

[0061] In this way, the target control parameters for controlling the vehicle are determined based on the psychological state detection information. In this way, by dynamically adjusting the driver's psychological state detection information in real time, precise control of the vehicle state is achieved, improving driving safety, comfort, and user experience.

[0062] In some embodiments, determining the target control parameter of the vehicle based on the mental state detection information includes:

[0063] determining an expected parameter value of the target control parameter according to the mental state detection information;

[0064] The current parameter value of the target control parameter is updated according to the expected parameter value.

[0065] This allows abstract mental states to be converted into executable, quantified instructions, controlling various vehicle systems to adapt to the driver's real-time mental state. Target control parameters encompass multiple subsystems, including vehicle powertrain, chassis, cabin environment, and safety assistance. Through real-time dynamic adjustments, precise control of vehicle status is achieved, enhancing driving safety, comfort, and user experience.

[0066] In some embodiments, updating the parameter value of the target control parameter according to the expected parameter value includes:

[0067] According to the expected parameter value, the pre-built objective function, the constraint conditions and the change state of the target control parameter at the current time, the adjustment amount of the current parameter value is determined to update the current parameter value.

[0068] In this way, the desired parameter value is determined based on the psychological state detection information; based on the desired parameter value, the current parameter value is updated to the desired parameter value. This converts the abstract psychological state into actionable control instructions, enabling dynamic updates of the vehicle control system, improving driving safety, alleviating driver stress, and optimizing the driving experience.

[0069] In certain embodiments, the method further comprises:

[0070] The target control parameter is updated according to the third behavior state information of the user in the cockpit.

[0071] Vehicle control of psychological state demands.

[0072] In this way, through quantitative optimization driven by objective functions, safety margins defined by constraints, and dynamic adaptation based on real-time state perception, vehicle parameter adjustments are both targeted and controlled, avoiding the risks associated with blind adjustments. This ensures that adjustment plans not only meet the driver's psychological needs but can also be implemented safely and smoothly in real-world driving environments.

[0073] In some embodiments, updating the target control parameter according to the third behavior state information of the user in the cabin includes:

[0074] After determining the target control parameter according to the mental state detection information, the target control parameter is updated according to the acquired third behavior state information.

[0075] In this way, the target control parameters are updated based on the user's third-party behavioral status information. This feedback optimization process can improve the ability to capture changes in vehicle status indicators and driver behavior indicators. By combining reinforcement learning with the design of a reward function, this approach enables real-time and adaptive driving strategy adjustments.

[0076] In some embodiments, updating the target control parameter according to the third behavior state information of the user in the cabin includes:

[0077] The target control parameters are updated according to the third behavior state information and a pre-trained reward model.

[0078] In this way, the pre-trained reward model quantifies the reward information for the third behavior state and updates the target control parameters. This reward model enables the system to evolve from initially passively executing preset rules with the user to an intelligent decision-maker that can proactively identify user preferences and dynamically adapt to complex scenario variables, ultimately achieving precise customization and personalized adaptation of control strategies.

[0079] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0080] An embodiment of the present application provides a vehicle, comprising the above-mentioned vehicle cabin sleep-aiding soundscape implementation device and electronic equipment, to implement the steps of the above-mentioned method.

[0081] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by one or more processors, the steps of the above method are implemented.

[0082] An embodiment of the present application provides a computer program product, including a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0083] The electronic device, vehicle, computer-readable storage medium, and computer program product provided in the embodiments of this application control the vehicle based on the driver's mental state information detected within the vehicle cabin. This allows for a more comprehensive determination of the driver's mental state by analyzing and processing the behavioral state information of each user within the cabin. Based on this driver's mental state information, vehicle control parameters can be adjusted to improve driving safety and comfort.

[0084] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0086] Figure 1 This is one of the flowcharts of the vehicle control method according to certain embodiments of the present application;

[0087] Figure 2 This is a second flow chart of a vehicle control method according to certain embodiments of the present application;

[0088] Figure 3 This is a third flow chart of a vehicle control method according to certain embodiments of the present application;

[0089] Figure 4 This is one of the module structure diagrams of the vehicle control method of certain embodiments of the present application;

[0090] Figure 5This is the second schematic diagram of the module structure of the vehicle control method in certain embodiments of the present application;

[0091] Figure 6 This is the third schematic diagram of the module structure of the vehicle control method in certain embodiments of the present application;

[0092] Figure 7 This is a fourth schematic diagram of the module structure of the vehicle control method according to certain embodiments of the present application;

[0093] Figure 8 This is the fifth schematic diagram of the module structure of the vehicle control method in certain embodiments of the present application;

[0094] Figure 9 This is the sixth schematic diagram of the module structure of the vehicle control method in certain embodiments of the present application;

[0095] Figure 10 This is a fourth flow chart of a vehicle control method according to certain embodiments of the present application;

[0096] Figure 11 This is a fifth flow chart of a vehicle control method according to certain embodiments of the present application;

[0097] Figure 12 This is the sixth flow chart of the vehicle control method according to certain embodiments of the present application;

[0098] Figure 13 This is the seventh flow chart of the vehicle control method according to certain embodiments of the present application;

[0099] Figure 14 This is the eighth flow chart of the vehicle control method according to certain embodiments of the present application;

[0100] Figure 15 This is a ninth flowchart of a vehicle control method according to certain embodiments of the present application;

[0101] Figure 16 This is a tenth flowchart of a vehicle control method according to certain embodiments of the present application;

[0102] Figure 17 This is a flowchart of a vehicle control method according to certain embodiments of the present application;

[0103] Figure 18 This is a twelfth flowchart of a vehicle control method according to certain embodiments of the present application;

[0104] Figure 19 This is the thirteenth flow chart of the vehicle control method according to certain embodiments of the present application;

[0105] Figure 20is a schematic diagram of the system module structure of a vehicle control method in certain embodiments of the present application;

[0106] Figure 21 It is a system workflow diagram of the vehicle control method of certain embodiments of the present application. DETAILED DESCRIPTION

[0107] The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and should not be understood as limiting the embodiments of the present application.

[0108] In related technologies, vehicles typically rely on explicit behavioral information such as the driver's facial expressions, voice tone, and body movements, applying computer vision and natural language processing techniques to build machine learning or deep learning models to predict driver emotions. For example, onboard cameras can capture micro-expressions such as a raised corner of the mouth or a frown, and then determine whether the driver is in an emotional state such as joy, anger, or anxiety. Control parameters such as the vehicle's maximum longitudinal acceleration and steering sensitivity can be adjusted accordingly to improve driving safety and comfort. However, this method of detecting psychological states based on explicit behavioral information has limitations. Human psychological states are inherently complex, multidimensional systems, whose formation and changes are influenced by a combination of factors such as physiological state, environmental factors, personal experience, and real-time context. Furthermore, there is no simple one-to-one correspondence between psychological state and behavioral state; many underlying psychological activities exist that are difficult to directly reflect through behavior. For example, a driver may be deeply contemplative while driving, perhaps due to thinking about work matters or reminiscing about daily life. This internal psychological activity often does not manifest itself clearly in facial expressions or voice tone, but it can distract attention and affect driving safety. At the same time, complex driving scenarios can interfere with the correlation between behavioral information and psychological state. In traffic jams or sudden emergencies, drivers may remain expressionless, focused on responding to road conditions, yet their actual psychological stress remains high. Traditional detection methods struggle to accurately identify these subtle psychological changes. Consequently, the accuracy of psychological state detection based on driver behavioral information is difficult to guarantee. Consequently, the resulting vehicle control strategy deviates from the driver's actual needs, resulting in suboptimal vehicle control performance.

[0109] Based on the above questions, please refer to Figure 1 , an embodiment of the present application provides a vehicle control method, the method comprising:

[0110] 01: Control the vehicle based on the psychological state detection information of the main driver in the vehicle cabin space, where the psychological state detection information is determined based on the behavioral state information of each user in the vehicle cabin space.

[0111] The embodiment of the present application provides a vehicle control device. The vehicle control method of the embodiment of the present application can be implemented by the vehicle control device of the embodiment of the present application. Specifically, the vehicle control device includes a psychological state prediction module and a vehicle control module. The psychological state prediction module is used to determine psychological state detection information based on the behavioral state information of each user in the vehicle cabin space. The vehicle control module is used to control the vehicle based on the psychological state detection information of the main driver in the vehicle cabin space. The psychological state detection information is determined based on the behavioral state information of each user in the vehicle cabin space.

[0112] The present application also provides an electronic device comprising a memory and a processor. The vehicle control method of the present application can be implemented by the electronic device of the present application. Specifically, the memory stores a computer program, and the processor is configured to control the vehicle based on psychological state detection information of the primary driver within the vehicle cabin. The psychological state detection information is determined based on behavioral state information of each user within the vehicle cabin.

[0113] Specifically, the primary driver refers to the driver. Psychological state detection information refers to comprehensive psychological characteristics of the driver, including emotions, cognitive load, and stress levels. This includes various psychological states, such as anxiety, excitement, and panic. The value range of this psychological state detection information can be [0, 1], with values closer to 0 indicating a more negative psychological state and values closer to 1 indicating a more positive psychological state. This psychological state detection information is determined based on the behavioral state information of each user in the vehicle cabin and represents a coordinated psychological stress index for the driver and passengers.

[0114] Behavioral status information refers to the user's movement characteristics, such as the degree of frowning and the speed of voice tone. For example, by analyzing the correlation between a passenger's rapid voice tone and the driver's frowning, it is possible to more accurately determine whether the driver is experiencing stress due to interpersonal interactions.

[0115] It is understandable that the driver's mental state is not only affected by himself, but also by other passengers in the cabin. For example, when stuck in traffic, the driver is in a bad mood. If he is affected by the complaints of other passengers, his bad mood will increase and even develop into anger.

[0116] In the embodiment of the present application, it can be considered that the cabin space includes at least one primary driving user and at least one non-primary driving user.

[0117] In the embodiments of this application, the driver's mental state detection information, or in other words, the driver's mental state detection information determined by combining the driver's and other passengers' mental state information, is used to control the vehicle, such as adjusting relevant vehicle parameters, to provide driving assistance to the driver. In one example, when the driver's mental state is detected as fatigue, the vehicle can be controlled by reducing steering wheel sensitivity, increasing following distance, and other methods, thereby avoiding accidents and improving driving safety. This avoids misjudgments caused by solely detecting the driver's behavior, thereby controlling the vehicle to reduce steering wheel sensitivity and increase the distance to the vehicle ahead, thus preventing safety accidents caused by driver agitation.

[0118] In summary, in the embodiments of the present application, the vehicle can be controlled, such as by adjusting relevant vehicle parameters, based on the driver's psychological state detection information determined from the behavioral state information of each user in the vehicle cabin, so that the desired vehicle control effect can be achieved to a certain extent. Furthermore, because the driver's psychological state detection information is determined based on the behavioral state information of each user in the vehicle cabin, the driver's psychological state detection information can be correlated with the behavioral state information of the driver and non-driver users in the vehicle cabin. This ensures the validity and reliability of the driver's psychological state detection information, and to a certain extent improves the reliability of the driver's psychological state detection results, which can be low when the driver's psychological state is not expressed through facial expressions or tone of voice.

[0119] See also Figure 2 In certain embodiments, the method further comprises:

[0120] 02: Determine the predicted first mental state of the main driver based on the first behavioral state information of the main driver in the vehicle cabin space;

[0121] 03: Determine the second psychological state prediction of the non-driver user based on the second behavioral state information of the non-driver user in the vehicle cabin space;

[0122] 04: Determine the psychological state detection information of the primary driver based on the first psychological state prediction and the second psychological state prediction.

[0123] In certain embodiments, the mental state prediction module is further configured to determine a first mental state prediction of the primary driver based on first behavioral state information of the primary driver within the vehicle cabin. The mental state prediction module is further configured to determine a second mental state prediction of a non-primary driver based on second behavioral state information of the non-primary driver within the vehicle cabin. The mental state prediction module is further configured to determine mental state detection information of the primary driver based on the first mental state prediction and the second mental state prediction.

[0124] In certain embodiments, the processor is configured to determine a first predicted mental state of the primary driver based on first behavioral state information of the primary driver within the vehicle cabin. The processor is further configured to determine a second predicted mental state of the non-primary driver based on second behavioral state information of the non-primary driver within the vehicle cabin. The processor is further configured to determine mental state detection information of the primary driver based on the first predicted mental state and the second predicted mental state.

[0125] Specifically, the first behavioral state information is the behavioral state information of the primary driver, or the driver. By predicting and processing the first behavioral state information, a first mental state prediction can be determined. The first mental state prediction refers to the driver's mental state within a preset time in the future. In one example, if it is detected that the driver's eyelid closure time increases from 10% to 30%, the change in the driver's eyelid closure time can be predicted to determine that the driver will be in a fatigued state in the next 5 seconds.

[0126] Secondary behavioral state information refers to the behavior of passengers other than the driver. Predictive processing of this second behavioral state information can be used to determine a second psychological state prediction. This second psychological state prediction refers to the passenger's psychological state within a preset timeframe. In one example, the front passenger's head-turning frequency exceeded 8 times per minute, while the rear passenger's body twisting frequency reached 5 times per minute. Based on these predicted head-turning and body twisting frequencies, it was determined that the passenger would be in a state of anxiety within the next 5 seconds.

[0127] Furthermore, the driver's mental state detection information is determined by the first and second mental state predictions. By collaboratively processing the first and second mental state predictions, the driver's mental state detection information can be determined. Understandably, a driver's mental state is influenced by both the driver and the passenger. In some examples, during a traffic jam, frequent urging from passengers can cause the driver to become more anxious or even angry.

[0128] In this way, by predicting and processing the driver's first behavioral state information to obtain a first mental state prediction, and predicting and processing the passenger's second behavioral state information to obtain a second mental state prediction, the first and second mental state predictions are collaboratively processed to determine the driver's mental state detection information. This comprehensive assessment of the impact of the in-vehicle environment on the driver allows for more accurate driver mental state detection information, providing more reliable mental state detection information as an evaluation basis for vehicle control.

[0129] In some embodiments, the first behavioral status information includes at least one of facial movement data, voiceprint data, steering wheel usage status data, and seat pressure data.

[0130] Specifically, the first behavioral state information is the driver's action state information, including facial action data, voiceprint data, steering wheel usage status data, and seat pressure data. Facial actions can include, for example, eyelid closure frequency, mouth corner curvature, eyebrow furrowing degree, and the like. Facial action data can be collected by a facial acquisition device in the cockpit, such as an infrared camera. Voiceprint data can include, for example, voice spectrum entropy, intonation speed, keyword frequency, and other sound information. Voiceprint data can be collected by a sound acquisition device, such as a microphone. Steering wheel usage status data can include, for example, grip strength change rate, steering fine-tuning frequency, sudden steering amplitude, and the like. Steering wheel usage status data can be collected by a steering wheel acquisition device, such as a torque sensor. Seat pressure data can include, for example, sitting posture adjustment frequency, hip pressure distribution asymmetry, and the like. Seat pressure data can be collected by a seat acquisition device, such as a pressure sensor.

[0131] The first behavioral state information includes one or more of facial movement data, voiceprint data, steering wheel usage data, and seat pressure data. This information is first processed through time synchronization of these multiple data sets, followed by sequential feature concatenation, for example, in the concatenation format of [micro-expression feature vector; voiceprint feature vector; steering wheel feature vector; seat feature vector]. This first behavioral state information can serve as target data for subsequent predictive processing. This multi-dimensional first behavioral state information provides an accurate and reliable basis for subsequent driver psychological prediction and decision-making.

[0132] In this way, by collecting the driver's multi-dimensional behavioral state, the first behavioral state information is determined. This multi-dimensional behavioral data, such as facial movement data (e.g., eyelid closure, mouth corner curvature, and other indicators of fatigue and emotion) acquired through infrared cameras, voiceprint data (speech spectrum and intonation changes) acquired through microphones, steering wheel usage (grip strength and steering characteristics) acquired through torque sensors, and sitting posture information (body posture changes) acquired through pressure sensors, can provide a more reliable basis for vehicle control decisions.

[0133] In some embodiments, the second behavioral state information includes at least one of facial movement data, voiceprint data, and seat pressure data.

[0134] Specifically, the second behavioral state information refers to passenger motion state information, including facial motion data, voiceprint data, and seat pressure data. Facial motion data can include, for example, eyelid closure frequency, mouth corner curvature, and eyebrow furrowing. Facial motion data can be collected by a facial acquisition device in the cabin, such as an infrared camera. Voiceprint data can include, for example, voice spectrum entropy, intonation speed, and keyword frequency. Voiceprint data can be collected by a sound acquisition device, such as a microphone. Seat pressure data can include, for example, sitting posture adjustment frequency and hip pressure distribution asymmetry. Seat pressure data can be collected by a seat acquisition device, such as a pressure sensor.

[0135] The second behavioral state information includes one or more of facial movement data, voiceprint data, and seat pressure data. This second behavioral state information is first processed through time synchronization of these multiple data sets, followed by sequential feature concatenation, for example, in the form of [micro-expression feature vector; voiceprint feature vector; steering wheel feature vector; seat feature vector]. The first behavioral state information can serve as target data for subsequent predictive processing. This multi-dimensional second behavioral state information provides an accurate and reliable basis for subsequent passenger psychology prediction and decision-making.

[0136] In this way, the second behavioral state information is determined by collecting multi-dimensional behavioral data of the driver. This multi-dimensional behavioral data, such as facial movement data (e.g., eyelid closure, mouth curvature, and other indicators of fatigue and emotion) acquired through infrared cameras, voiceprint data (speech spectrum and intonation changes) acquired through microphones, and sitting posture information (body posture changes) acquired through pressure sensors, can provide a more reliable basis for vehicle control decisions.

[0137] See also Figure 3 In some embodiments, step 02 (determining a first psychological state prediction of the main driver user based on the first behavioral state information of the main driver user in the vehicle cabin space) includes:

[0138] 021: Build a model based on the first behavior state information and the pre-trained first feature, and determine first feature data corresponding to the first behavior state information;

[0139] 022: Determine a first mental state prediction based on the first feature data and a pre-trained decoder model.

[0140] In some embodiments, the mental state prediction module is further configured to construct a model 110 based on the first behavioral state information and a pre-trained first feature, and determine first feature data corresponding to the first behavioral state information. The mental state prediction module is further configured to determine a first mental state prediction based on the first feature data and a pre-trained decoder model 120.

[0141] In some embodiments, the processor is configured to construct a model 110 based on the first behavioral state information and a pre-trained first feature, and determine first feature data corresponding to the first behavioral state information. The processor is further configured to determine a first mental state prediction based on the first feature data and a pre-trained decoder model 120.

[0142] Specifically, see Figure 4 In an embodiment of the present application, the pre-trained first feature construction model 110 may be a temporal convolutional network (TCN) model, and the pre-trained first feature construction model 110 is used to perform feature extraction processing on the first behavior state information to obtain first feature data corresponding to the first behavior state information. The first feature data is a temporal feature of the first behavior state information. In one example, by performing feature extraction processing on data of dynamic changes in signals such as facial expressions, voiceprints, and grip strength, high-order temporal feature data, i.e., long-period behavior state vector segments, such as a fatigue accumulation vector segment within 15 minutes, can be obtained.

[0143] Next, the first feature data is processed using a pre-trained decoder model 120 to obtain a first mental state prediction. Decoder model 120 is cascaded with first feature construction model 110. In one example, if the first feature data represents accumulated fatigue over 15 minutes, decoder 110 can predict the first mental state prediction, e.g., that the driver will be in a highly fatigued state in 5 seconds.

[0144] In this way, based on the first behavioral state information and the pre-trained first feature construction model 110, first feature data corresponding to the first behavioral state information is determined; based on the first feature data and the pre-trained decoder model 120, a first psychological state prediction is determined. In this way, the first feature construction model 110 extracts features from the driver's multi-dimensional behavioral data, such as facial expressions, voiceprint features, and steering wheel grip, to obtain high-order temporal features that characterize their behavioral state. Using the decoder model 120 to decode and predict the state of these feature data, the driver's psychological state change trend within a preset time period can be predicted. By establishing a complete mapping chain from behavioral signals to psychological states, the vehicle's ability to predict and respond to changes in the driver's emotions is significantly improved.

[0145] See also Figure 5 In some embodiments, the first feature construction model 110 includes a plurality of first time convolution modules 111 connected in sequence.

[0146] The multiple first time convolution modules 111 are configured to perform multiple feature extractions and feature filtering on the first behavior state information to determine first feature data.

[0147] Specifically, the first time convolution module 111 of the embodiment of the present application can perform convolution processing on the first behavior state information. The first time convolution module 111 performs multiple feature extraction and feature filtering processes on the first behavior state information to determine the first feature data. By performing convolution processing on the first behavior state information, the complexity of the first behavior state information can be reduced, its noise can be removed, its feature information can be focused, and accurate and easy-to-decode input data can be provided to the cascaded decoder 120. In an example, the first feature model 110 includes 8 first time convolution modules 111 connected in sequence. By performing 8 convolution processes on the first behavior state information, first feature data with high-level temporal features can be obtained.

[0148] It should be noted that the number of first temporal convolution modules 111 in this application is for illustrative purposes only and should not be construed as limiting the number. In other examples, the number of first temporal convolution modules 111 can also be 6, 9, or 10 layers, etc., which is not limited here and is set based on actual conditions.

[0149] In this way, multiple first temporal convolution modules 111 perform convolution processing on the first behavioral state information, determining highly refined and semantically rich first feature data. Each temporal convolution module 111 thus performs deeper feature abstraction and filtering based on the primary features extracted by the preceding module, gradually removing data noise and strengthening the expression of key features, providing high-quality feature input for subsequent psychological state prediction.

[0150] See also Figure 6 In some embodiments, the first temporal convolution module 111 includes a first causal convolution layer 1111 and a first gated linear unit 1112 connected in sequence.

[0151] The first causal convolution layer 1111 is configured to perform feature extraction processing on the first received information according to a predetermined first dilation factor to determine a first processing result, where the first received data includes first behavior state information and / or a module output result of the first temporal convolution module 111;

[0152] The first gated linear unit 1112 is configured to perform filtering processing on the first processing result to determine the module output result of the first temporal convolution module 111 .

[0153] Specifically, first causal convolution layer 1111 is configured to perform causal convolution processing on first received information to determine a first processing result, where the first received data includes first behavioral state information and / or the module output result of first temporal convolution module 111. This ensures that the current output depends only on historical data, meeting real-time inference requirements.

[0154] The first expansion factor is used to expand the receptive field of the convolution kernel of the first causal convolution layer 1111. The first expansion factor can be obtained by d=2 n Represents, where n represents n convolutional layers starting from 1, which increases with the number of layers, thereby expanding the convolution receptive field.

[0155] The first gated linear unit 1112 is used to suppress irrelevant feature information such as invalid fine-tuning noise of the steering wheel, and can improve the recognition weight of key actions (such as sharp turns).

[0156] After performing convolution processing, namely after processing by the first causal convolution layer 1111 and the first gated linear unit 1112, the first temporal convolution module 1111 retains the processing results and combines the processing results of this layer with those of the previous layer, outputting the combined results. This prevents the deep network from losing key information. For example, by combining the "single braking force" obtained from the shallow layer with the "continuous sudden braking frequency" obtained from the deep layer, it can be determined that the driver may be fatigued, resulting in occasional strong braking but a slower overall vehicle response.

[0157] In one example, the first feature construction model 110 includes 8 first time convolution modules 111, the eighth first causal convolution layer 1111 has a convolution kernel size of K=3 and a dilation factor d=2. n The first behavior state information is a 10-second sequence, i.e., a one-dimensional time series signal with a length of 1000 (100 sample signals are collected per second). Based on the first causal convolution layer 1111 and the first gating unit 1112, the following is the process of the first feature construction model 110 performing convolution processing on the first behavior state information:

[0158] First layer: First, the dilation factor d=2 of the first causal convolutional layer 1111 1, the convolution kernel actually covers the first behavior state information of 3+2*(3-1)=7 historical moments, and performs causal convolution processing on the first behavior state information in the time series to obtain the first processing result that represents the feature, and outputs the obtained first processing result to the first gating unit 1112. Then, the first gating unit 1112 processes the first processing result and can distinguish important features from noise in the first processing result based on the gated linear unit (Gated Linear Unit, GLU), that is, generate the gated signal weight through the sigmoid activation function to filter out irrelevant features and output the current layer result. For example, if the sensor detects that the steering wheel is shaking violently, the GLU will enhance the weight of the feature and reduce the weight of the noise in other stable periods, thereby suppressing invalid features. Finally, the current layer result is retained, and the high temporal feature information of the first convolution module 111 of the first layer is output.

[0159] Second layer: First, the dilation factor d=2 of the first causal convolutional layer 1111 2 , the convolution kernel actually covers 3+4*(3-1)=11 historical moments, and processes the output results of the first gating unit 1112 of the first layer. Only the current moment and the 77 moments before the interval sampling are processed, which can capture longer-range dependencies (such as driving behavior patterns within the first 5 seconds), and output the first processing results to the first gating unit 1112. The first gating unit 1112 is similar to the first layer and will not be repeated here. Finally, the current layer result is combined with the upper layer result to output the high-series feature information of the first convolution module 111 of the second layer.

[0160] The subsequent layers are similar to the above layers and will not be described in detail. 1 The receptive field increases exponentially. At the eighth layer, the convolution kernel covers 3+2568(3-1)=512 moments, which can capture longer time series dependencies (such as the persistent low response mode during fatigue driving).

[0161] Therefore, the first causal convolution layer 1111 and the first gating unit 1112 perform convolution processing on the first behavioral state information, which can reduce the complexity of the first behavioral state information, remove its noise, and focus on its feature information, thereby providing accurate and easy-to-decode input data for the cascaded decoder 120.

[0162] The high-level time series feature output by the last first time convolution module 111, namely the first feature data, can be expressed by the following formula:

[0163] H∈R T8D

[0164] Where T is the time series information, such as the 1st, 2nd, 3rd, and 4th seconds. D is the feature dimension of each time point, such as the dimension of the features at the 1st, 2nd, 3rd, and 4th seconds.

[0165] In this way, by performing multi-level feature extraction on the driver's behavioral data through multiple cascaded first-time convolution modules 111, high-level features representing the driver's psychological state can be obtained. Combining causal convolution and gating mechanisms, noise can be filtered while maintaining temporal dependencies. By performing multi-level feature extraction on the driver's behavioral data, such as facial expressions, voiceprints, and steering wheel operations, key temporal features representing the psychological state can be gradually extracted. By employing causal convolution and gating mechanisms in each first-time convolution module 111, noise interference can be effectively filtered while maintaining temporal dependencies. Through the stacking of modules and feature refinement, a purer and more easily decodable high-level feature representation can be provided to the subsequent decoder. The ultimately extracted precise features are input into the decoder model 120, enabling accurate prediction of the driver's psychological state. This technical solution based on deep temporal feature extraction significantly improves the vehicle's ability to understand and respond to the driver's state by establishing a complete conversion chain from raw behavioral data to high-level psychological features.

[0166] In some implementations, the decoder model 120 is configured to:

[0167] According to the first characteristic data, construct an observation probability matrix and a state transfer matrix;

[0168] A first psychological state prediction is determined based on the observation probability matrix, the state transition matrix, and a plurality of predetermined user psychological states.

[0169] Specifically, the first feature data is an observation sequence that characterizes the temporal characteristics of the first behavioral state information. The observation probability matrix is used to describe the probabilistic relationship between the user's psychological state and the first feature data (such as observation data such as facial action unit intensity and voice fundamental frequency). It can be obtained by calculating and processing the observation sequence using a Gaussian mixture model. The state transition matrix is used to quantify the dynamic evolution of psychological states. The state transition matrix is dynamically learned through driver behavior data and can promptly reflect the changing trends of the driver's psychological state (e.g., due to environmental factors such as traffic jams and high-speed driving).

[0170] In this embodiment, the decoder model 120 can perform prediction processing on the first feature data based on the hidden Markov model to obtain a first psychological state prediction. The hidden Markov model can infer the implicit state sequence from the observable sequence data, and model the transition rules between states and the relationship between states and observations. In the embodiment of this application, the driver's psychological state information cannot be directly observed, so the hidden Markov model can be used to process the observable behavioral state to infer the driver's psychological state information. The following is the decoding process of the decoder model 120 based on the hidden Markov model:

[0171] First, the continuous psychological state (such as anxiety and fatigue) is discretized into latent states (such as S1 = mild anxiety, S2 = high anger) for subsequent calculation of the state probability matrix;

[0172] Next, the Gaussian mixture model is used to analyze the observation sequence H∈R T*D By performing calculations, the observation probability of each state s can be generated and the observation probability matrix can be obtained, which can be expressed by the following formula:

[0173] B(H)=Softmax(W i ,H t +b i ),W i ∈R D*K

[0174] Among them, K is the label category, that is, the category of psychological state, such as anxiety, irritability, etc.

[0175] Next, we construct a state transition matrix, which can be used to describe the dynamic changes of psychological states (such as the transition probability of "fatigue→anger") and solve the nonlinear mapping problem between behavioral signals and psychological states. The state transition matrix can be expressed by the following formula:

[0176] A_ij=P(S_(t+1)=j|S_t=i)

[0177] Finally, based on the predetermined psychological state, the observation probability matrix and the state transition matrix are multiplied to determine the largest path as the first psychological state prediction, providing accurate basis data for subsequent vehicle control.

[0178] Furthermore, the decoder model 120 can also construct a loss function and parameter optimization strategy.

[0179] The loss function includes the loss function of the first time convolution module 110 and the loss function of the decoder model 120. The loss function is used to measure the deviation between the model prediction result and the actual result, providing a clear goal for parameter optimization. The loss function of the decoder model 120 can use the forward algorithm to calculate the sequence probability, maximize the log likelihood, and ensure that the probability match between the hidden state sequence (psychological state) predicted by the model and the observed data (biological signal) is the highest. It can be expressed by the formula L HMM = -log P(X|λ), λ = (A, B, π). The loss function of the first time convolution module 110 is used to force the features to distinguish different mental states through cross entropy loss. It can be expressed by the formula In addition, by combining the two loss functions, L total =0.5L HMM +0.5L TCN , a weighted sum can be used to balance the contributions between the two.

[0180] The parameter optimization strategy includes a parameter optimization strategy for the first time convolution module 110 and a parameter optimization strategy for the decoder model 120. The parameter optimization strategy for the first time convolution module 110 can use the Adam optimizer, a learning rate of 3e-4, and a gradient clipping threshold of 1.0. The parameter optimization strategy for the decoder model 120 can use the Baum-Welch algorithm to update the transfer matrix A online. In addition, by alternately updating the parameters of the first time convolution module 110 and the decoder model 120 every epoch, the joint optimization is prevented from falling into a local minimum. In one example, the decoding process of the decoder model 120 is as follows:

[0181] The first feature data (time window T=10 seconds) includes:

[0182] Facial micro-expressions: eyelid closure (blink) at the 3rd second, continuous eye closure for 1.5 seconds at the 6th second (exceeding the fatigue threshold), and slight drooping of eyebrows (muscle relaxation) at the 8th second;

[0183] Physiological signals: Steering wheel grip force gradually decreased from an initial 80N to 50N (starting to decrease at the 5th second), and the forward lean angle of the sitting posture changed from 0° to 20° (backward leaning, detected at the 7th second);

[0184] Voice characteristics: When answering the navigation prompt in the 4th second, the speaking speed drops to 120 words / minute (normal 180 words / minute), and the tone is smooth and without fluctuations (lack of emotional fluctuations).

[0185] First, the user's mental state is set to S1: awake, S2: mildly fatigued, and S3: severely fatigued. The first feature data is then processed using a Gaussian mixture model. The results show that the observed probabilities of 1.5 seconds of continuous eye closure at the 6th second are 0.1, 0.8, and 0.9 for S1: awake, S2: mildly fatigued, and S3: severely fatigued, respectively. The observed probabilities of decreased steering wheel grip strength at the 5th second are 0.2, 0.7, and 0.8 for S1: awake, S2: mildly fatigued, and S3: severely fatigued, respectively. Then, under the default S1 awake state distribution of [0.8, 0.2, 0], a decrease in grip strength in the first 5 seconds triggers a state transition, with a probability of 0.3 for the transition from S1: awake to S2: mildly fatigued. After 6 seconds, due to the continued eye closure, the probability of transitioning from S2: mildly fatigued to S3: severely fatigued increases to 0.6, indicating an increased probability of increasing fatigue.

[0186] By performing dot product processing on the observation probability matrix and the state transition matrix, it is obtained that the probability of severe fatigue at t = 10 seconds is 75%, and the probability of severe fatigue state is predicted to continue to increase at t = 15 seconds, providing a quantitative decision-making basis for vehicle control.

[0187] In this manner, decoder model 120 processes the first feature data to determine an observation probability matrix and a state transition matrix. The observation probability matrix, state transition matrix, and user psychological state are then processed to determine a first psychological state prediction. In this way, the driver's psychological state can be accurately predicted using the observation probability matrix, which describes the probabilistic relationship between the user's psychological state and the first feature data, the state transition matrix, which quantifies the dynamic evolution patterns between psychological states, and multiple predetermined user psychological states. The obtained first psychological state prediction more accurately describes the probabilistic relationship between the observed data and the psychological state, allowing the model to more reliably infer the driver's psychological state based on the current observed data, thereby providing more valuable information for subsequent intelligent driving decisions.

[0188] See also Figure 7 In some embodiments, step 03 (determining a predicted second psychological state of the non-driver user based on the second behavioral state information of the non-driver user in the vehicle cabin space) includes:

[0189] 031: constructing a model 130 based on the second behavior state information and the pre-trained second feature, and determining second feature data corresponding to the second behavior state information;

[0190] 032: Determine a second mental state prediction based on the second feature data and the pre-trained classifier model 140.

[0191] In certain embodiments, the mental state prediction module is further configured to construct a model 130 based on the second behavioral state information and a pre-trained second feature, and determine second feature data corresponding to the second behavioral state information. The mental state prediction module is further configured to determine a second mental state prediction based on the second feature data and a pre-trained classifier model 140.

[0192] In some embodiments, the processor is further configured to construct a model 130 based on the second behavioral state information and a pre-trained second feature, and determine second feature data corresponding to the second behavioral state information. The processor is further configured to determine a second mental state prediction based on the second feature data and a pre-trained classifier model 140.

[0193] Specifically, see Figure 8 In the embodiments of the present application, a pre-trained second feature construction model 130 is used to perform feature extraction processing on the second behavioral state information to obtain second feature data corresponding to the second behavioral state information. The second feature data is a temporal feature of the second behavioral state information. In one example, by performing feature extraction processing on dynamic change data of signals such as facial expressions, voiceprints, and sitting posture, high-order temporal feature data, i.e., vector segments of long-term behavioral state trends, can be obtained.

[0194] Next, the pre-trained classifier model 140 is used to perform prediction processing on the second feature data to obtain a second mental state prediction. The classifier model 140 is cascaded to the second feature construction model 130.

[0195] In this way, based on the second behavioral state information and the pre-trained second feature construction model 130, the second feature data corresponding to the second behavioral state information is determined; based on the second feature data and the pre-trained classifier model 140, the second psychological state prediction is determined. In this way, the second feature construction model 130 performs deep feature extraction on the passenger's multi-dimensional behavioral data, such as facial expressions, voiceprint characteristics, and sitting posture, to obtain high-order temporal features that represent changes in their psychological state. Using the classifier model 140 to classify and predict the state of this feature data, the passenger's future psychological state change trend can be predicted. By establishing a complete mapping chain from behavioral signals to psychological states, the vehicle's ability to predict and respond to changes in the driver's emotions is significantly improved.

[0196] See also Figure 9 In some embodiments, the second behavior state information includes multiple sub-state information, and the second feature construction model 130 includes multiple second time convolution modules 131 connected in sequence.

[0197] The second time convolution module 131 is configured to perform feature extraction and feature filtering on each sub-state information to determine second feature data corresponding to the second time convolution module 131 .

[0198] Specifically, the sub-state information refers to facial action data, voiceprint data, and seat pressure data. The second temporal convolution module 131 can perform causal convolution processing on each sub-state information separately, which can compress the number of parameters and improve calculation efficiency.

[0199] The second temporal convolution module 131 can perform convolution processing on each sub-state information based on the TCN. The second temporal convolution module 131 performs multiple grouped feature extraction and feature filtering on the sub-state information to determine the second feature data corresponding to the sub-state information. In one example, performing grouped convolution processing on the sub-state information can provide accurate and easily classified input data to the cascaded classifier 140.

[0200] In one example, the second feature construction model 130 includes six second time convolution modules 131 connected in sequence. By performing six convolution processes on the sub-state information of the second behavior state information, second feature data with high-level temporal features can be obtained.

[0201] It should be noted that the number of second temporal convolution modules 131 in this application is for illustrative purposes only and should not be construed as limiting the number. In other examples, the number of second temporal convolution modules 131 can also be 4, 7, or 8 layers, etc., which is not limited here and is set based on actual conditions.

[0202] In this way, multiple second temporal convolution modules 131 are used to perform convolution processing on the sub-state information groups of the second behavioral state information. After multiple feature extractions, highly refined and semantically rich second feature data is determined, providing accurate and easily classified input data for the cascaded classifier 140. In this way, each temporal convolution module 131 performs deeper feature abstraction and filtering based on the primary features extracted by the previous module, gradually removing data noise and strengthening the expression of key features. This provides high-quality feature input for subsequent psychological state prediction.

[0203] See also Figure 10 In some embodiments, each second temporal convolution module 131 includes a second causal convolution layer 1311 and a second gated linear unit 1312 connected in sequence.

[0204] The second causal convolution layer 1311 is configured to perform depthwise convolution and pointwise convolution processing on each type of second received information according to a predetermined second dilation factor to determine a second processing result;

[0205] The second gated linear unit 1312 is configured to perform filtering processing on the second processing result to determine the module output result of the second temporal convolution module 131 .

[0206] Specifically, the second causal convolution layer 1311 is used to perform grouped causal convolution processing on the second received information to determine the second processing result, ensuring that the current output depends only on historical data to meet real-time reasoning requirements. The second dilation factor expands the receptive field of the convolution kernel by multiplying it.

[0207] The second gated linear unit 1312 is used to distinguish important features from noise through a gating mechanism, retaining signals strongly related to psychological states while enhancing the module's ability to represent complex behavioral patterns. In one example, the second gated linear unit processes in-vehicle noise such as steering wheel vibration (50-100Hz mechanical noise) and air conditioning wind noise (steady-state broadband noise), suppressing the weight of irrelevant signals to 0.1-0.3 (such as air conditioning wind noise) and increasing the weight of key behavioral features (such as sudden changes in grip strength) to 0.7-0.9. In one example, a passenger suddenly panics. The second received information is causally convolved through the second causal convolution layer 1311 to independently extract the voiceprint scream (sound frequency greater than 2000Hz) and seat tilt (seat pressure reaches 300Pa) features. The second gated linear unit 1321 then increases the weight of the scream feature (0.95) and filters the air conditioning wind noise (weight 0.1). The final output is the second temporal convolution module 131 output, which triggers a "panic" state classification with high confidence.

[0208] After performing grouped convolution processing, namely, after processing by the second causal convolution layer 1311 and the second gated linear unit 1312, the second temporal convolution module 121 can retain the processing results and combine the processing results of this layer with the processing results of the upper layer, and output the combined results. This can prevent the deep network from losing key information. For example, the second received information is the steering wheel fine-tuning frequency and blinking frequency increasing from 0.2Hz to 0.8Hz for 5 consecutive minutes. Through causal convolution processing, the shallow layer (d=1) captures the single blink action, and the deep layer (d=16) associates the long-term fatigue trend, which can more accurately determine the second processing result.

[0209] In one example, the second feature construction model 130 includes six second temporal convolution modules 131, and the sixth first causal convolution layer 1311 has a convolution kernel size of K=3 and a dilation factor d=1. Based on the second causal convolution layer 1311 and the second gating unit 1312, the second feature construction model 130 performs group convolution processing on the second behavior state information similarly to the first feature construction model 110. The second dilation factor of the second temporal convolution module 131 changes from 1, 2, 4, 8, 16, to 32 per layer, so the receptive field of the convolution kernel is different, and the number of parameters for group processing is 1 / 3 of that of the first feature construction model 110. The convolution process of each layer of the second feature construction model 130 is not repeated here.

[0210] In this way, by performing multi-level feature extraction on passenger behavior data through multiple cascaded second causal convolution layers 1311, high-level features representing the passenger's psychological state can be obtained. By combining causal convolution with a gating mechanism, multiple cascaded second temporal convolution modules 131 perform multi-level feature extraction on passenger behavior data, such as facial expressions, voiceprints, and sitting postures. This reduces computational effort, improves computational efficiency, and gradually extracts key temporal features representing the psychological state. Causal convolution and a gating mechanism are employed in each temporal convolution module, effectively filtering out noise interference while ensuring temporal dependencies. Through the stacking of modules and feature refinement, a purer and more easily classified high-level feature representation is provided for the subsequent classifier. The ultimately extracted precise features are input into the classifier model, enabling accurate prediction of the passenger's psychological state. This technical solution, based on deep temporal feature extraction, significantly improves the vehicle's ability to understand and respond to passenger status by establishing a complete conversion chain from raw behavioral data to high-level psychological features.

[0211] In some implementations, the classifier model 140 includes a pooling layer, a fully connected layer, and a classification layer that are sequentially connected.

[0212] The pooling layer is configured to perform downsampling processing on the second feature data and determine a downsampling processing result;

[0213] The fully connected layer is configured to determine mapping data of the downsampling processing result in each predetermined user psychological state based on the downsampling processing result;

[0214] The classification layer is configured to determine a second mental state prediction based on the mapped data.

[0215] Specifically, the pooling layer is used to compress the time series features of the second feature data (time series features) into a fixed-dimensional global feature vector through downsampling and maximum pooling, and then determine the downsampling result. The pooling layer can eliminate the length differences of the time series, retain the peak features of each time step (such as strong micro-expressions or sudden changes in speech at a certain moment), filter out redundant time series information, and form a compact feature representation suitable for classification.

[0216] The fully connected layer is used to map the downsampling processing results to the user's psychological state (such as "fatigue", "anxiety", "relaxation", etc.), realize the conversion from feature space to semantic space, and output the classification score corresponding to the user's psychological state one by one, providing a basis for subsequent probability calculation.

[0217] The classification layer is used to transform the classification scores, determine the second psychological state prediction, obtain the probability distribution of n types of psychological states, and quantify the possibility of each state (such as the probability of "fatigue" is 85%, and the probability of "anxiety" is 10%). It can be implemented based on functions such as softmax and SIGMOD.

[0218] Thus, based on classifier model 140, the pooling layer first downsamples the second feature data, compressing and extracting its features to determine the downsampling result. Next, the fully connected layer performs mapping processing, transforming from feature space to semantic space and determining the mapping result. Finally, the classification layer transforms the mapping result to determine the probability of each state in the second psychological state prediction quantification. In this way, through the pooling, fully connected, and classification layers of classifier model 140, the raw state feature data can be transformed into understandable psychological states after multi-layer processing, providing accurate and reliable prediction results for subsequent dynamic adjustment of cabin parameters (e.g., lowering the air conditioning temperature or playing refreshing music based on "fatigue" status).

[0219] See also Figure 11 In some embodiments, step 04 (determining the driver's mental state detection information based on the first mental state prediction and the second mental state prediction) includes:

[0220] 041: Determine the psychological state detection information based on the first psychological state prediction, the second psychological state prediction and the preset coefficient

[0221] In some embodiments, the vehicle control module is configured to determine mental state detection information based on the first mental state prediction, the second mental state prediction, and a preset coefficient.

[0222] In some embodiments, the processor is further configured to determine mental state detection information based on the first mental state prediction, the second mental state prediction, and a preset coefficient.

[0223] Specifically, the preset coefficient is used to characterize the mutual influence relationship between the first psychological state prediction and the second psychological state prediction, which can ensure the validity of the psychological state detection information. Because the psychological states of the driver and the passenger are not independent of each other, but have a mutual influence relationship. Therefore, the psychological state detection information can be determined by the first psychological state prediction, the second psychological state prediction and the preset coefficient.

[0224] In one example, when a passenger shows nervousness and anxiety, it may be conveyed to the driver through words or actions, making the driver more nervous; on the contrary, if the passenger is in a good mood and communicates easily, it may relieve the driver's stress. Therefore, the psychological state detection information of the main driver is determined by the driver's psychological state, i.e., the first psychological state prediction, and the passenger's psychological state, i.e., the second psychological state. The preset coefficient can be used to characterize this mutual influence relationship, and then the psychological state detection information is jointly determined by the first psychological state prediction, the second psychological state prediction and the preset coefficient to ensure the accuracy of the psychological state detection information. In one example, the formula ESI can be used. syn =ESIdriver +βPSI indicates that ESI syn is the mental state detection information, and ESI is predicted by the first mental state driver , the second psychological state prediction PSI and the preset coefficient β are determined.

[0225] In this way, the mental state detection information is determined by combining the first mental state prediction, the second mental state prediction, and the preset coefficient. This allows for a more comprehensive and accurate reflection of the driver's actual mental state in the entire driving environment.

[0226] See also Figure 12 In certain embodiments, the method further comprises:

[0227] 042: Determine a preset coefficient based on the second psychological state prediction and preset influencing factor parameters, wherein the preset influencing factor parameters are determined based on the first behavioral state information and the second behavioral state information.

[0228] In some embodiments, the vehicle control module is configured to determine mental state detection information based on the first mental state prediction, the second mental state prediction, and a preset coefficient.

[0229] In certain embodiments, the processor is configured to determine a preset coefficient, the second mental state prediction, and the preset coefficient based on the second mental state prediction and the preset influencing factor parameter, and determine the mental state detection information.

[0230] Specifically, the preset influencing factor parameter refers to the multi-head collaborative attention weight of the time-series synchronization of the first and second behavioral state information, integrating the influence of multiple features in the first and second behavioral state information. The preset coefficient is determined by the second psychological state prediction and the preset influencing factor parameter. Because the mutual influence relationship between the first and second psychological state predictions is complex, the preset parameter is determined by processing the second psychological state prediction and the preset influencing factor parameter.

[0231] In one example, a bidirectional state transfer matrix is first constructed based on the first psychological state prediction and the second psychological state prediction. The bidirectional state transfer matrix is used to represent the mutual mapping relationship between the driver's psychological state and the passenger's psychological state. It is the n-value of the psychological state combination. 2 The joint state is expressed by the following formula:

[0232]

[0233] Then, the preset impact factor is determined according to the bidirectional state transfer matrix;

[0234] The preset coefficient is then determined jointly by the second psychological state prediction and the preset influencing factor parameters.

[0235] Furthermore, we can use the formula β=tan(w1·α dp +w2·ΔPSI+w3·C) represents the process of determining the preset coefficient. Among them, β is the preset coefficient, which is determined by the preset impact factor parameter α dp The second predicted mental state change ΔPSI is determined by operation. The second predicted mental state change ΔPSI is the change over a period of time, for example, ΔPSI=PSI t -PSI t-10 is the passenger pressure change within a 10-frame window; C represents the complexity of the environmental scene, with a value range of [0, 1]. For example, a value closer to 0 indicates a small number of people in the cabin and fewer people talking, which has a greater impact on the driver; a value closer to 1 indicates a large number of people in the cabin and more people talking, which has a greater impact on the driver; w1, w2, and w3 represent pre-learned weight parameters. By calculating and processing the second mental state prediction and the preset influencing factor parameters, the complex interaction between the first and second mental state predictions can be more comprehensively and accurately determined.

[0236] In this way, a preset coefficient can be determined based on the computational processing of the second psychological state prediction and the preset influencing factor parameters. By integrating the second psychological state prediction (e.g., the passenger's psychological state) with preset influencing factors (e.g., the coordinated attention weights of driver and passenger behaviors and the environmental complexity parameter), the mutual influence of psychological states in multi-occupant scenarios can be accurately quantified.

[0237] See also Figure 13 In certain embodiments, the method further comprises:

[0238] 043: Determine a query vector based on the state information of the first line;

[0239] 044: Determine a key vector and a value vector according to the second behavior state information;

[0240] 045: Determine the preset impact factor parameters based on the query vector, key vector, and value vector.

[0241] In certain embodiments, the vehicle control module is further configured to determine a query vector based on the first behavioral state information, a key vector and a value vector based on the second behavioral state information, and a preset impact factor parameter based on the query vector, the key vector, and the value vector.

[0242] In some embodiments, the processor is further configured to determine a query vector based on the first behavior state information, a key vector and a value vector based on the second behavior state information, and a preset impact factor parameter based on the query vector, the key vector, and the value vector.

[0243] Specifically, the query vector is determined by extracting features from the first behavioral state information and is used to query other passengers' behaviors for key information related to the driver. For example, the query vector is determined by extracting features from the driver's steering wheel angle, braking frequency, and line of sight deviation. The query vector can be used to determine which passenger behaviors (such as not wearing a seatbelt or frequent conversation) affect the driver during rapid acceleration, for example.

[0244] The key vector is determined by extracting behavioral features from the second behavioral state information and is used to match the query vector and determine the correlation between the first and second behavioral state information. The value vector is determined by extracting specific features from the second behavioral state information and is used to weight the relevant features in the subsequent extraction.

[0245] Before this, it is necessary to dynamically time-warp the first and second behavioral state information. By aligning the driver and passenger psychological state sequences through dynamic time-warping, synchronization can be achieved in the time domain, eliminating the difference in reaction delay between the two. Calculate to obtain.

[0246] It is also necessary to calculate the influence heads of three types of influences. Because the behavioral state information of the driver and the passenger will affect each other, it is also necessary to perform attention calculation on each behavioral state information to obtain the influence relationship between them. It should be noted that the driver obtains one more steering wheel state than the passenger user's behavioral state. Since the steering wheel force information and the seat force information belong to the same physical force information, they can be regarded as the same physiological influence. The three types of influences include voice influence, visual influence and physiological influence. The influence head can process part of the vector segmentation in parallel. The following is the calculation formula of the three types of influence heads:

[0247] Voice influence head: Q = driver's voiceprint feature,

[0248] Visual impact head: Q = driver's micro-expression,

[0249] Physiological influence head: Q = driver posture characteristics,

[0250] Among them, Q is the query vector, K is the key vector, and V is the value vector.

[0251] Based on the query vector, key vector, and value vector, the degree of impact of passenger behavior on the driver can be quantified to determine the preset impact factor parameters. For example, when the driver brakes suddenly, the passenger's screaming behavior will increase the impact factor, prompting the system to strengthen the warning. Further, it can be expressed as follows:

[0252]

[0253] Among them, d h is the vector dimension of the matrix KQ.

[0254] In this way, based on aligning input features and calculating multi-head collaborative attention, the query vector can be determined by extracting and processing the features of the first behavioral state information. Secondly, the key vector is determined by extracting and processing the features of the second behavioral state information, and the value vector is determined by extracting and processing the specific features of the second behavioral state information. Finally, weights are generated by the matching degree between the query vector and the key vector, and the value vectors are weighted and summed to obtain the final impact factor. In this way, by identifying the temporal dependencies of cross-object behaviors (such as the synchronization of driver operations and passenger actions) in real time, the one-sidedness of relying solely on the features of a single object is avoided. At the same time, through the dynamic calculation of attention weights, it can adapt to complex scenarios with multiple passengers and multiple behaviors (such as multi-person conversations, children crying, etc.), providing an accurate basis for subsequent psychological state prediction and vehicle control.

[0255] See also Figure 14 In certain embodiments, step 01 (controlling the vehicle based on the detection information of the driver's mental state in the vehicle cabin space) includes:

[0256] 011: Determine the target control parameters of the vehicle based on the psychological state detection information.

[0257] In some embodiments, the vehicle control module is further configured to determine target control parameters of the vehicle based on the mental state detection information.

[0258] In some embodiments, the processor is further configured to determine target control parameters of the vehicle based on the mental state detection information.

[0259] Specifically, target control parameters refer to specific quantifiable control indicators that control various vehicle systems to adapt to the driver's real-time psychological state. Different psychological state detection information corresponds to different vehicle target control parameters, which can convert abstract psychological states into executable quantified instructions, namely target control parameters, to control the vehicle. In one example, based on the driver's fatigue state, parameters such as extending the following distance from the default 2 seconds to 3 seconds and lowering the air conditioning temperature from 24°C to 22°C are adjusted to make driving safer and help the driver stay awake. For example, adjusting the following distance to give a tired driver more reaction time or using cold air stimulation to help the driver wake up, or lowering the temperature to allow cold air stimulation to help the driver wake up.

[0260] In this way, the target control parameters for controlling the vehicle are determined based on the psychological state detection information. In this way, by dynamically adjusting the driver's psychological state detection information in real time, precise control of the vehicle state is achieved, improving driving safety, comfort, and user experience.

[0261] In certain embodiments, the target control parameter includes at least one of a vehicle following parameter, a lane change frequency, a steering wheel sensitivity, and a vehicle acceleration constraint parameter.

[0262] Specifically, the following parameter refers to the time interval maintained between a vehicle and the vehicle ahead, which directly determines the safety margin during emergency braking. Lane change frequency refers to the number of active lane changes per unit time, reflecting the aggressiveness of driving style. Steering wheel sensitivity refers to the gain coefficient between steering angular velocity and steering wheel angle, which determines the vehicle's responsiveness to steering operations. Vehicle acceleration constraint parameters include the upper limit of longitudinal acceleration and the limit on the rate of change of lateral acceleration.

[0263] Table 1 below is the target control parameterization strategy library:

[0264] Table 1

[0265]

[0266] This allows abstract mental states to be converted into executable, quantified instructions, enabling the vehicle's various systems to adapt to the driver's real-time mental state. Target control parameters encompass multiple subsystems, including vehicle powertrain, chassis, cabin environment, and safety assistance. Through real-time dynamic adjustments, precise control of the vehicle's state is achieved, enhancing driving safety, comfort, and user experience.

[0267] See also Figure 15 In some implementations, step 011 (determining target control parameters of the vehicle based on psychological state detection information) includes:

[0268] 0111: Determine the expected parameter value of the target control parameter based on the psychological state detection information;

[0269] 0112: Update the current parameter value of the target control parameter according to the expected parameter value.

[0270] In some embodiments, the vehicle control module is further configured to determine an expected parameter value of the target control parameter based on the mental state detection information, and to update a current parameter value of the target control parameter based on the expected parameter value.

[0271] In certain embodiments, the processor is further configured to determine an expected parameter value of the target control parameter based on the mental state detection information, and to update a current parameter value of the target control parameter based on the expected parameter value.

[0272] Specifically, the expected parameter value refers to the ideal target parameter value calculated based on psychological state detection information through algorithms or preset rules to guide vehicle system adjustments. The expected parameter value can transform abstract psychological states (such as emotions, stress, fatigue, etc.) into quantifiable control instructions. Its dynamic adjustment can be expressed by the following formula:

[0273] T target =T base +k T Sigmoid(ESI syn -50)

[0274]

[0275] Where T represents the following distance, L represents the lane change frequency, S represents the directional sensitivity, target represents the target value, and base represents the original value. In this way, the original data can be dynamically updated through different calculation methods.

[0276] In one example, when the psychological state detection information shows that the driver is in a nervous or anxious state, the driver can be given more reaction time and operating pressure can be reduced by increasing the following distance, reducing the frequency of lane changes, and reducing the steering wheel sensitivity.

[0277] The current value refers to the real-time state of the target control parameter before adjustment. Based on the desired parameter value, the parameter adjustment amount is calculated in real time, enabling dynamic updates and stable transitions in the vehicle control system. In one example, when the mental state detection information indicates fatigue, the current parameter value is adjusted in real time based on the desired parameter value, automatically adjusting the seat angle from the current 10° to 15°. The updated value then becomes the desired parameter value, and the lumbar support level is increased from level 3 to level 4, overwriting the current value with the desired value.

[0278] In this way, the desired parameter value is determined based on the psychological state detection information; based on the desired parameter value, the current parameter value is updated to the desired parameter value. This converts the abstract psychological state into actionable control instructions, enabling dynamic updates of the vehicle control system, improving driving safety, alleviating driver stress, and optimizing the driving experience.

[0279] See also Figure 16 In some embodiments, step 0112 (updating the current parameter value of the target control parameter according to the expected parameter value) includes:

[0280] 01121: Determine the adjustment amount of the current parameter value based on the pre-built expected parameter value, objective function, constraints, and the change state of the target control parameter at the current time to update the current parameter value.

[0281] In some embodiments, the vehicle control module is further configured to determine an adjustment amount for the current parameter value based on pre-built desired parameter values, objective functions, constraints, and the changing state of the target control parameters at the current time, so as to update the current parameter value.

[0282] In some embodiments, the processor is further configured to determine an adjustment amount for the current parameter value based on the pre-built expected parameter value, objective function, constraint conditions, and the change state of the target control parameter at the current time, so as to update the current parameter value.

[0283] Specifically, the objective function is used to translate abstract requirements into concrete, calculable goals. This provides a clear optimization direction for parameter adjustments and ensures that the adjustment strategy aligns with the core requirements of psychological state detection by minimizing control mutations and comfort loss. For example, when adjusting the seat angle when the driver is fatigued, the weighted sum of safety risk and comfort loss is minimized, and the seat angle is adjusted to 15°.

[0284] In the embodiment of the present application, the objective function can be expressed by the following formula:

[0285]

[0286] Among them w1, w2, w3 are weight coefficients, It is the changing state of the target control parameter at the current time.

[0287] Constraints are used to constrain the safety boundaries of parameter changes, eliminate unreasonable adjustment plans, and avoid new risks caused by excessive parameter adjustments. For example, the seat angle can only be between 8° and 20° (to avoid excessive angles that affect braking). The changing state is used to dynamically correct the adjustment strategy based on the vehicle's real-time motion state, environmental information, and driver operation. For example, the parameter adjustment rate is slowed down at high speeds to avoid driver misjudgment due to system intervention.

[0288] In the embodiment of the present application, the constraint condition can be expressed by the following formula:

[0289]

[0290] Furthermore, based on the pre-constructed expected parameter values, objective functions, constraints and the changing state of the target control parameters at the current time, it can be achieved through a fuzzy PDI controller, that is, the center of gravity method can be used to calculate the PDI to determine the adjustment amount of the current parameter value. In this way, when the parameter exceeds the safety range, integral separation can be started to avoid new risks caused by excessive parameter adjustment.

[0291] In this way, through quantitative optimization driven by objective functions, safety margins defined by constraints, and dynamic adaptation based on real-time state perception, vehicle parameter adjustments are both targeted and controlled, avoiding the risks associated with blind adjustments. This ensures that adjustment plans not only meet the driver's psychological needs but can also be implemented safely and smoothly in real-world driving environments.

[0292] See also Figure 17 In certain embodiments, the method further comprises:

[0293] 012: Update the target control parameters according to the third behavior status information of the user in the cockpit.

[0294] In some embodiments, the vehicle control module is further configured to update the target control parameter according to third behavior state information of the user in the cabin.

[0295] In some embodiments, the processor is further configured to update the target control parameter according to third behavior state information of the user in the cabin.

[0296] Specifically, the third behavioral state information refers to the user's feedback information, which is used to update the target control parameters. The third behavioral state information can include explicit feedback and implicit feedback. Explicit feedback refers to the driver's active rating (such as 1-5 points) and voice commands (such as "driving too slowly"). Implicit feedback refers to state information that needs to be judged, such as steering wheel grip force variance. Excessive grip force fluctuations may indicate a mismatch in the driver's feedback strategy or a time difference in manual takeover. It can be expressed by the following formula ΔT takeover =T 实际接管 -T 系统建议接管 Excessive variance indicates that the user is dissatisfied with the current system takeover, leading to actual takeover. When the user is dissatisfied with vehicle control, such as with a low rating or when voice commands do not match vehicle control parameters, the target control parameters are updated based on factors such as steering wheel grip and the time difference between manual takeovers, thereby achieving vehicle control that meets the driver's psychological needs.

[0297] In this way, the target control parameters are updated based on the user's third-party behavioral status information. This feedback optimization process can improve the ability to capture changes in vehicle status indicators and driver behavior indicators. By combining reinforcement learning with the design of a reward function, this approach enables real-time and adaptive driving strategy adjustments.

[0298] See also Figure 18 In some implementations, step 012 (updating the target control parameter based on the third behavior state information of the user in the cabin) includes:

[0299] 0121: After determining the target control parameter according to the mental state detection information, the target control parameter is updated according to the third behavioral state information.

[0300] In some embodiments, the vehicle control module is further configured to update the target control parameter according to the third behavioral state information and a pre-trained reward model after determining the target control parameter according to the psychological state detection information.

[0301] In some embodiments, the processor is further configured to, after determining the target control parameter based on the mental state detection information, update the target control parameter based on the third behavioral state information and a pre-trained reward model.

[0302] Specifically, after the original corresponding target control parameters have been determined based on the psychological state detection information, the target control parameters are updated according to the user's third behavioral state information. When the first feedback information of the third behavioral state information is satisfactory with the target control parameters and the second feedback information is inconsistent with the ideal information, new target control parameters are determined, and the original target control parameters mapped by the psychological state detection information are replaced with new target control parameters; when the first feedback information of the third behavioral state information is satisfactory with the target control parameters and the second feedback information is consistent with the ideal information, the target control parameters corresponding to the psychological state detection information are not changed. The first feedback information is the driver's voice command and rating feedback, and the second feedback information refers to the first state monitoring information. By updating the target control parameters, the updated target control parameters are more accurate in controlling the vehicle state and more in line with the driver's expectations, thereby improving driving safety and comfort.

[0303] In one example, in a congested road section during the morning rush hour, the driver's mental state detection information is fatigue, and the system adjusts the target control parameter to follow the vehicle with a time distance of T according to the fatigue state. cur = 2.0s, according to the third behavior state information, the target control parameters are updated including:

[0304] First, when the target control parameter is T cur = 2.0s, due to the frequent occurrence of cutting in during peak hours on congested roads, vehicles often brake suddenly to maintain the following distance.

[0305] Then, based on the first feedback information representing the driver's third behavioral state information, such as the driver's voice "Why do you always brake?" or the second feedback information that the steering wheel grip force variance fluctuates sharply between 80-350N (the normal fatigue scenario should be 50-200N), the system will extend the following distance from 2.0s to 2.3s. This can avoid triggering the system's sudden braking due to the vehicle in front cutting in, and meet the driver's demand for less aggressive braking in the third behavioral state information.

[0306] Then, if the driver does not complain again or the steering wheel grip force variance drops to 120N (normal value), the system will set the original corresponding target control parameter T cur =2.0s is fixed as the updated target control parameter and the following distance is T cur =2.3s.

[0307] Finally, when the mental state detection information is fatigue state again, the system adjusts the target control parameter to follow the vehicle distance T according to the fatigue state. cur = 2.3s. The third behavioral state information is then used to repeatedly determine whether to update the target control parameters. Once the original corresponding target control parameters have been determined based on the psychological state detection information, the target control parameters are updated based on the user's third behavioral state information. This updating of the target control parameters allows the system to evolve from mechanically executing the preset control strategy in the initial stage to an intelligent carrier that can accurately understand user needs and dynamically adapt to complex scenarios, ultimately achieving personalized adaptive optimization of the target control parameters.

[0308] See also Figure 19 In some implementations, step 012 (updating the target control parameter based on the third behavior state information of the user in the cabin) includes:

[0309] 0122: Update the target control parameters based on the third behavior state information and the pre-trained reward model.

[0310] In some instances, the vehicle control device is further configured to update target control parameters based on the third behavior state information and a pre-trained reward model.

[0311] In some instances, the processor is further configured to update the target control parameters according to the third behavior state information and a pre-trained reward model.

[0312] Specifically, the reward model is used to convert the third behavior state information into a quantifiable reward signal. The reward signal may include a comfort score + a takeover time difference penalty item. The reward signal guides the update of the control parameters to achieve deep adaptation of the strategy to the user's real needs and complex scenarios. In one example, based on the first feedback information such as the driver's active score of 1-5 points and the second feedback information such as steering wheel grip fluctuations and manual takeover time difference, the reward model is integrated and processed into a specific reward value. The comfort score can be directly used as a positive reward to encourage the system to generate a control strategy that is more in line with the driver's preferences; the manual takeover time difference can be used as a penalty item (the more frequent the takeover, the greater the penalty), prompting the system to reduce unnecessary intervention and enhance user trust. In the implementation scheme of the present application, the PPO algorithm can be used to update the target control parameters, and the process can be expressed by the following formula:

[0313] In this way, the pre-trained reward model quantifies the reward information for the third behavior state and updates the target control parameters. This reward model enables the system to evolve from initially passively executing preset rules with the user to an intelligent decision-maker that can proactively identify user preferences and dynamically adapt to complex scenario variables, ultimately achieving precise customization and personalized adaptation of control strategies.

[0314] The following describes the system module design and workflow of this application with a complete example:

[0315] See also Figure 20 The system modules include driver state monitoring module, passenger state detection module, driver psychological state prediction model, passenger information state prediction model, driver-passenger collaborative prediction module, dynamic coupling control module and closed-loop feedback optimization module.

[0316] The driver state monitoring module is used to collect the multi-dimensional behavior state of the driver and determine the first behavior state information;

[0317] The passenger status detection module is used to collect the multi-dimensional behavior status of the passenger and determine the second behavior status information;

[0318] The driver mental state prediction model is used to determine a first mental state prediction of the main driver user based on the first behavioral state information of the main driver user in the vehicle cabin space;

[0319] The passenger information state prediction model is used to determine a second psychological state prediction of the non-driving user based on the second behavioral state information of the non-driving user in the vehicle cabin space;

[0320] The driver-passenger collaborative prediction module is used to determine the psychological state detection information of the main driver according to the first psychological state prediction and the second psychological state prediction;

[0321] The dynamic coupling control module is used to determine the target control parameters of the vehicle based on the psychological state detection information;

[0322] The closed-loop feedback optimization module is used to update the target control parameters according to the third behavioral state information of the user in the cabin.

[0323] In this way, through the coordinated action of the above modules, the vehicle can be controlled based on the psychological state detection information of the main driver in the vehicle cabin space.

[0324] The explanation of the corresponding modules can refer to the description of the relevant content in the aforementioned implementation method, which will not be repeated here.

[0325] See also Figure 21 The process of the vehicle control method in the embodiment of the present application is roughly as follows:

[0326] First, multi-dimensional behavioral data of the driver and passengers is collected in real time through the DMS camera, in-vehicle microphone, seat pressure sensor, steering wheel grip sensor, etc. In the embodiment of the present application, the DMS camera can collect facial movement data of the driver and passengers, the in-vehicle microphone can collect voiceprint data of the driver and passengers, the seat pressure sensor can collect seat pressure data of the driver and passengers, and the steering wheel grip sensor can collect steering wheel usage status data of the driver. One or more of the above data can constitute the first behavioral status information and the second behavioral status information.

[0327] Then, a psychological state prediction model is constructed. Based on the hidden Markov model and fusion framework, the psychological state is regarded as a hidden state sequence, and the state migration trend is inferred through observation data to achieve the psychological state prediction within a preset time in the future. In the embodiment of the present application, the first feature construction model performs convolution processing on the first behavioral state information and the second behavioral state information based on TCN, extracts their features, and determines the first feature data and the second feature data respectively. The decoder model is based on the hidden Markov model, infers the first psychological state prediction through the first feature data, and predicts the second feature information based on the state classifier to determine the second psychological prediction.

[0328] Then, through temporal synchronization and spatial correlation, the state characteristics of the driver and passenger are aligned. An attention network is used to construct an interactive influence network between the driver's state and the passenger's state, identifying the correlation between the passenger's state and the driver's psychological state, and accurately determining indirect state changes caused by the passenger. In this implementation, temporal synchronization is achieved by aligning the first and second psychological predictions through dynamic time warping calculations. Spatial modeling determines attention weights by calculating three types of influence heads. This allows the establishment of a transfer matrix between the passenger's state and the driver's psychological state, which is used to determine the relationship between the two.

[0329] Next, a library of driving strategy parameters (such as following distance, lane change frequency, and steering wheel sensitivity) is defined. These parameters are dynamically adjusted based on the psychological state probability. A fuzzy PID controller is also employed to achieve smooth parameter transitions, avoiding driving discomfort caused by policy jumps. In this implementation, parameters are dynamically adjusted based on the passenger and driver's collaborative psychological state index.

[0330] Finally, the system collects changes in vehicle status indicators and driver behavior indicators, combines them with reinforcement learning, and designs a reward function to achieve real-time and adaptive driving strategy adjustments. In this application's implementation, a closed-loop feedback loop is implemented through a dual-channel feedback mechanism, and a reward model is designed based on online reinforcement learning to optimize strategy updates.

[0331] An embodiment of the present application further provides a vehicle, which includes the above-mentioned electronic device.

[0332] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, the above-mentioned vehicle control method is implemented.

[0333] The embodiment of the present application also provides a computer program product, including a computer program / instruction, which implements the above-mentioned vehicle control method when executed by a processor.

[0334] The present application also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the aforementioned steps of controlling a vehicle based on psychological state detection information of a primary driver within the vehicle cabin, wherein the psychological state detection information is determined based on behavioral state information of each user within the vehicle cabin.

[0335] It is understood that a computer program includes computer program code. The computer program code may be in source code form, object code form, executable file, or some intermediate form. Computer-readable storage media may include any entity or device capable of carrying computer program code, recording media, USB flash drives, mobile hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media.

[0336] In the description of this specification, the descriptions with reference to the terms "particularly", "further", "particularly", "understandably", etc. are intended to mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms are not intended to refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0337] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code that includes one or more executable requests for implementing a specific logical function or step of a process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0338] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A vehicle control method, characterized in that: include: The vehicle is controlled according to psychological state detection information of a main driver in a vehicle cabin space, wherein the psychological state detection information is determined according to behavioral state information of each user in the vehicle cabin space.

2. The method according to claim 1, characterized in that The method further comprises: determining a first mental state prediction of the main driver user based on first behavioral state information of the main driver user in the vehicle cabin space; determining a second mental state prediction of the non-driving user based on the second behavioral state information of the non-driving user in the vehicle cabin space; The mental state detection information of the primary driver is determined based on the first mental state prediction and the second mental state prediction.

3. The method according to claim 2, characterized in that The first behavior status information includes at least one of facial movement data, voiceprint data, steering wheel usage status data and seat pressure data.

4. The method according to claim 2, characterized in that The second behavior state information includes at least one of facial movement data, voiceprint data, and seat pressure data.

5. The method according to claim 2, characterized in that The determining, based on the first behavioral state information of the main driver user in the vehicle cabin space, a first mental state prediction of the main driver user includes: Building a model based on the first behavior state information and the pre-trained first feature, and determining first feature data corresponding to the first behavior state information; Determine the first mental state prediction based on the first feature data and a pre-trained decoder model.

6. The method according to claim 5, characterized in that The first feature construction model includes a plurality of first time convolution modules connected in sequence, and the plurality of first time convolution modules are configured to perform multiple feature extractions and feature filtering on the first behavior state information to determine the first feature data.

7. The method according to claim 6, characterized in that Each of the first temporal convolution modules includes a first causal convolution layer and a first gated linear unit connected in sequence; The first causal convolution layer is configured to perform feature extraction processing on first received information according to a predetermined first dilation factor to determine a first processing result, wherein the first received data includes first behavioral state information and / or a module output result of the first prior temporal convolution module; The first gated linear unit is configured to filter the first processing result to determine a module output result of the first temporal convolution module.

8. The method according to claim 5, characterized in that The decoder model is configured as follows: Constructing an observation probability matrix and a state transition matrix based on the first feature data; The first psychological state prediction is determined according to the observation probability matrix, the state transition matrix, and a plurality of predetermined user psychological states.

9. The method according to claim 2, characterized in that The determining, based on the second behavior state information of the non-driving user in the vehicle cabin space, a second mental state prediction of the non-driving user includes: Building a model based on the second behavior state information and the pre-trained second feature, and determining second feature data corresponding to the second behavior state information; Determine a prediction of the second mental state based on the second feature data and a pre-trained classifier model.

10. The method according to claim 9, characterized in that The second behavior state information includes multiple sub-state information, and the second feature construction model includes multiple second time convolution modules connected in sequence; The second time convolution module is configured to perform feature extraction and feature filtering on each type of sub-state information to determine the second feature data corresponding to the second time convolution module.

11. The method according to claim 10, characterized in that Each of the second temporal convolution modules includes a second causal convolution layer and a second gated linear unit connected in sequence; The second causal convolution layer is configured to perform depthwise convolution and pointwise convolution processing on each type of second received information according to a predetermined second dilation factor to determine a second processing result; The second gated linear unit is configured to filter the second processing result to determine a module output result of the second temporal convolution module.

12. The method according to claim 9, characterized in that The classification model includes a pooling layer, a fully connected layer, and a classification layer connected in sequence; The pooling layer is configured to perform downsampling processing on the second feature data and determine a downsampling processing result; The fully connected layer is configured to determine, based on the downsampling processing result, mapping data of the downsampling processing result in each predetermined user psychological state; The classification layer is configured to determine a second mental state prediction based on the mapped data.

13. The method according to claim 2, characterized in that The determining the mental state detection information of the primary driver according to the first mental state prediction and the second mental state prediction includes: The mental state detection information is determined according to the first mental state prediction, the second mental state prediction and a preset coefficient.

14. The method according to claim 13, characterized in that The method comprises: The preset coefficient is determined according to the second psychological state prediction and a preset influencing factor parameter, wherein the preset influencing factor parameter is determined according to the first behavioral state information and the second behavioral state information.

15. The method according to claim 14, characterized in that The method further comprises: determining a query vector according to the first behavior state information; Determining a key vector and a value vector according to the second behavior state information; The preset impact factor parameter is determined according to the query vector, the key vector, and the value vector.

16. The method according to claim 1, wherein The controlling of the vehicle according to the detection information of the psychological state of the main driver in the vehicle cabin space includes: Determine target control parameters of the vehicle based on the mental state detection information.

17. The method according to claim 16, characterized in that The target control parameter includes at least one of a vehicle following parameter, a lane changing frequency, a steering wheel sensitivity, and a vehicle acceleration constraint parameter.

18. The method according to claim 16, characterized in that Determining the target control parameters of the vehicle according to the mental state detection information includes: determining an expected parameter value of the target control parameter according to the mental state detection information; The current parameter value of the target control parameter is updated according to the expected parameter value.

19. The method according to claim 18, characterized in that The updating of the parameter value of the target control parameter according to the expected parameter value includes: According to the pre-constructed objective function, constraint conditions and the change state of the target control parameter at the current time, the adjustment amount of the current parameter value is determined to update the current parameter value.

20. The method according to claim 16, wherein The method further comprises: The target control parameter is updated according to the third behavior state information of the user in the cockpit.

21. The method according to claim 20, characterized in that The updating of the target control parameter according to the third behavior state information of the user in the cabin includes: After determining the target control parameter according to the mental state detection information, the target control parameter is updated according to the acquired third behavior state information.

22. The method according to claim 20, characterized in that The updating of the target control parameter according to the third behavior state information of the user in the cabin includes: The target control parameters are updated according to the third behavior state information and a pre-trained reward model.

23. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 22 is implemented.

24. A vehicle, characterized in that: The vehicle includes the apparatus of claim 23.

25. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 22 is implemented.

26. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 22 is implemented.

Images