Method for recognizing state of object in vehicle, vehicle, and storage medium

Abstract

Embodiments of the present application provide a method for identifying a state of an object in a vehicle, a vehicle and a storage medium, wherein the method comprises: obtaining image data corresponding to a target object in a process in which the vehicle responds to a control operation triggered by the target object; determining target image data of a target part of the target object from the image data, wherein image content of the target image data at least includes the target part, and the target part is used to reflect a change process of a state of the target object; determining a photoplethysmography signal of the target part based on the target image data, wherein the photoplethysmography signal is used to represent a volume change degree of a physiological channel inside the target part; and determining a physiological state of the target object based on the photoplethysmography signal, wherein the physiological state is used to at least represent a safety degree of the target object triggering the control operation. The present application solves the technical problem of low accuracy of state identification of an object in a vehicle.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method for recognizing the state of objects in a vehicle, a vehicle, and a storage medium. Background Technology

[0002] Currently, relevant technologies for monitoring the state of the target object (driver) mainly rely on cameras to capture image data and identify the driver's explicit features from the image data, such as eyelid closure and head posture, to assess the driver's state. However, the above methods ignore subtle and imperceptible changes in the driver's physiological signals in the image data, such as heart rate and respiratory rate, which are often important clues for judging fatigue and distraction. Relying solely on explicit feature identification leads to insufficient accuracy in state assessment. Therefore, the technical problem of low accuracy in identifying the state of objects in vehicles remains.

[0003] There is currently no good solution to the above problems. Summary of the Invention

[0004] This application provides a method for recognizing the state of objects in a vehicle, a vehicle, and a storage medium, to at least solve the technical problem of low accuracy in recognizing the state of objects in a vehicle.

[0005] According to one aspect of the embodiments of this application, a method for state recognition of an object in a vehicle is provided. The method may include: acquiring image data corresponding to the target object during a vehicle response to a control operation triggered by the target object, wherein the image content of the image data includes at least the target object; determining target image data of a target part of the target object from the image data, wherein the image content of the target image data includes at least the target part, and the target part is used to reflect the state change process of the target object; determining a photoplethysmography (PPG) signal of the target part based on the target image data, wherein the PPG signal is used to represent the degree of volume change of the physiological channels within the target part; and determining the physiological state of the target object based on the PPG signal, wherein the physiological state is used to at least represent the safety level of the target object triggering the control operation.

[0006] Furthermore, based on the photoplethysmography (PPG) signal, the physiological state of the target object is determined, including: using a state recognition model to analyze the PPG signal to obtain the physiological state, wherein the state recognition model is obtained by training a deep learning model.

[0007] Furthermore, the state recognition model is a time series analysis model. Using the state recognition model, the photoplethysmography (PPG) signal is analyzed to obtain the physiological state, including: inputting the PPG signal into the time series analysis model, and using the time series analysis model to analyze the degree of change of the PPG signal over time to obtain the physiological state.

[0008] Furthermore, the time series analysis model includes a first input layer, a convolutional layer, a pooling layer, a recurrent layer, a first fully connected layer, and a first output layer. The photoplethysmography (PPG) signal is input into the time series analysis model, which analyzes the changes in the PPG signal over time to obtain the physiological state. This includes: responding to the PPG signal received by the first input layer, using the convolutional layer to extract features from the PPG signal to obtain local feature information; using the pooling layer to perform dimensionality reduction on the local feature information to obtain dimensionality-reduced local feature information; using the recurrent layer to obtain the correlation between the dimensionality-reduced local feature information and time, where the correlation represents the relationship between the dimensionality-reduced local feature information and time; using the first fully connected layer to integrate the relationship, local feature information, and dimensionality-reduced local feature information to obtain an integrated result, and mapping the integrated result to a target space to obtain a preset physiological state value, where different dimensions of the target space correspond to different preset physiological state values; and using the activation function in the first output layer to process the preset physiological state value to obtain the physiological state.

[0009] Furthermore, the state recognition model is an attention mechanism model. Using the state recognition model, the photoplethysmography (PPG) signal is analyzed to obtain the physiological state, including: inputting the PPG signal into the attention mechanism model, using the attention mechanism model to determine the importance of different PPG signals to the physiological state, and obtaining the physiological state.

[0010] Furthermore, the attention mechanism model includes a second input layer, a feature extraction layer, an attention layer, a second fully connected layer, and a second output layer. The photoplethysmography (PPG) signal is input into the attention mechanism model, which analyzes the PPG signal to obtain the physiological state. This includes: in response to the second input layer receiving the PPG signal, the feature extraction layer extracts key feature information from the PPG signal, where the importance of the key feature information to the physiological state is greater than the importance of other feature information; the attention layer performs weighted processing on the key feature information to obtain weighted key feature information; the second fully connected layer maps the weighted key feature information to corresponding preset physiological state values; and the second output layer evaluates the preset physiological state values ​​to obtain the physiological state.

[0011] Furthermore, the method may further include: identifying image data using a preset recognition model to obtain physiological state labels for the target object; and training the state recognition model using the physiological state labels and image data to obtain a trained state recognition model, wherein the physiological state labels are used to represent the physiological state determined by the preset recognition model; or, analyzing photoplethysmography (PPG) signals using the state recognition model to obtain physiological states, including: identifying image data using a preset recognition model to obtain physiological state labels for the target object; and analyzing PPG signals using the state recognition model to obtain initial physiological states, wherein the initial physiological states are the physiological states determined by the state recognition model; and selecting physiological states from the physiological state labels and initial physiological states.

[0012] Furthermore, the method may also include at least one of the following: in response to an abnormal physiological state, controlling the vehicle to output a prompting message, wherein the prompting message is used to prompt the target object to switch from the abnormal physiological state to a normal physiological state; in response to an abnormal physiological state, or if the target object fails to switch from the abnormal physiological state to a normal physiological state, adjusting the control operation, wherein the adjusted control operation is used to improve the safety of the target object performing the control operation; in response to an abnormal physiological state, or if the target object fails to switch from the abnormal physiological state to a normal physiological state, adjusting the safe distance between the vehicle and the obstacle object based on the vehicle's driving speed and the speed of the obstacle object moving in the vehicle's driving direction; or, determining target image data from image data, including: identifying the image data to determine the location of the target part in the image data; identifying the image data of the location of the target part to obtain key points of the target part, and determining the image data of the area where the key points are located as the target image data.

[0013] According to another aspect of the embodiments of this application, a state recognition device for an object in a vehicle is also provided. This device may include: an acquisition module, configured to acquire image data corresponding to the target object during a vehicle response to a control operation triggered by the target object, wherein the image content of the image data at least includes the target object; a first determination module, configured to determine target image data of a target part of the target object from the image data, wherein the image content of the target image data at least includes the target part, and the target part is used to reflect the state change process of the target object; a second determination module, configured to determine a photoplethysmography (PPG) signal of the target part based on the target image data, wherein the PPG signal is used to represent the degree of volume change of the physiological channels inside the target part; and a third determination module, configured to determine the physiological state of the target object based on the PPG signal, wherein the physiological state at least represents the safety level of the target object triggering the control operation.

[0014] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods in various embodiments of this application.

[0018] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0019] In this embodiment, during the process of the target object performing control operations on the vehicle, image data corresponding to the target object can be acquired in real time. The target part reflecting the state change process of the target object, and the target image data corresponding to that target part, can be identified from the image data. Based on the target image data, the degree of volume change of the physiological channels inside the target part can be analyzed to obtain the corresponding photoplethysmography (PPG) signal. The PPG signal can be used to determine the physiological state of the target object. In other words, in this application, by acquiring image data of the target object in real time during the process of performing control operations on the vehicle, the target part reflecting state changes is accurately located, and the PPG signal is extracted and analyzed to comprehensively evaluate the physiological state of the target object. The above method overcomes the limitations of related technologies that rely solely on explicit features, effectively captures and utilizes the physiological signal changes implicit in the image data, significantly improving the accuracy and comprehensiveness of state recognition, especially in early warning of fatigue and distraction, thereby achieving the technical effect of improving the accuracy of state recognition of objects in vehicles and solving the technical problem of low accuracy in state recognition of objects in vehicles. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a flowchart of a method for recognizing the state of an object in a vehicle according to an embodiment of this application;

[0022] Figure 2 This is a flowchart of a driver's emotional state prediction method according to an embodiment of this application;

[0023] Figure 3(a) is a schematic diagram of a state recognition model according to an embodiment of this application;

[0024] Figure 3(b) is a schematic diagram of another state recognition model according to an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of a vehicle object state recognition device according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] According to an embodiment of this application, a method for recognizing the state of an object in a vehicle is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] This embodiment provides a method for recognizing the state of objects in a vehicle. Figure 1 This is a flowchart of a method for recognizing the state of an object in a vehicle according to an embodiment of this application, such as... Figure 1 As shown, it may include vehicle A, network B, and server C. Figure 1 As shown, the method may include the following steps.

[0030] Step S102: During the process of the vehicle responding to the control operation triggered by the target object, image data corresponding to the target object is acquired.

[0031] In the technical solution provided by step S102 of the embodiments of this application, the image content of the image data includes at least the target object.

[0032] In this embodiment, the target object can refer to an individual in the vehicle whose state needs to be monitored, such as the driver. The state of the target object (such as fatigue, distraction, or physiological abnormality) affects the safety and controllability of vehicle operation. Control operations can refer to operations triggered by the target object that affect the state or behavior of the vehicle, and may include, but are not limited to, control operations such as pressing the accelerator, braking, steering, accelerating, or decelerating. In the driving scenario of the vehicle, control operations can reflect the driver's intentions and immediate reactions.

[0033] Optionally, image data can refer to visual information of a target object captured by the vehicle's image acquisition equipment (such as an onboard camera or other vision sensors), and can also be called driver image data. Image data can include the appearance features of the target object, as well as its dynamic behavior during control operations. Image data can be used to assess the state of the target object while driving the vehicle.

[0034] Optionally, image data is acquired in real time during the execution of control operations because the driver's body movements and facial expressions at this time best reflect the driver's immediate level of attention, reaction speed, and physiological state.

[0035] For example, high-resolution, wide-angle cameras installed inside the vehicle can capture the driver's full face and some of their body movements. Ideally, the camera should have good lighting adaptability, operating stably in different driving environments (such as daytime, nighttime, or various weather conditions) to ensure the quality of the image data remains unaffected. The camera's field of view can be set to cover the driver's head to shoulders, with particular emphasis on clear visibility of the facial area, as facial features (such as expressions and eye contact) are used to assess the driver's attention and fatigue level.

[0036] Optionally, in order to accurately capture changes in the driver's state during control operations, the camera can continuously acquire image data at a sufficiently high frame rate to ensure that minute changes in the driver's instantaneous state (such as briefly closing the eyes or slightly shaking the head) can be recorded.

[0037] Step S104: Determine the target image data of the target part of the target object from the image data.

[0038] In the technical solution provided by step S104 of the embodiments of this application, the image content of the target image data includes at least the target region. The target region is used to reflect the change process of the state of the target object.

[0039] In this embodiment, the target area can refer to a body region in the image data that is directly related to the state assessment of the target object. The target area can be a region where the driver's physiological signals and behavioral expressions are prominent, such as the facial region (e.g., the eye and mouth areas), neck region, or wrist region. No specific limitations are imposed here; any body region that can ensure vehicle control safety and the accuracy of target object state recognition is within the scope of protection of this application embodiment. The selection of the target area can be based on the characteristics of the target area that reflect changes in the target object's state. For example, the frequency and duration of eye closure can be used to determine fatigue, while changes in facial expressions and color can reveal emotional fluctuations or physiological health status of the target object. By focusing on the aforementioned target area, the parts related to the target object's state changes can be acquired more efficiently and accurately, avoiding redundant analysis of irrelevant areas in the image data other than the target area, thereby improving the accuracy and real-time performance of state assessment.

[0040] Optionally, target image data can refer to image segments or datasets containing the target area extracted from the original image data. Target image data can be images of skin regions. For example, if the target area is the face, target image data can include image data showing facial features, particularly the eye and mouth areas, to facilitate accurate subsequent capture of subtle changes such as blinking frequency and drooping corners of the mouth, which can indicate fatigue, distraction, or changes in emotional state.

[0041] Optionally, after obtaining the image data corresponding to the target object, the target image data of the target part of the target object can be determined from the image data.

[0042] Optionally, before identifying the target area, the raw image data can be preprocessed to improve the image data quality and recognition accuracy. Preprocessing steps may include, but are not limited to: adjusting the brightness, contrast, and chroma of the image data; removing noise from the image data; and stabilizing the image data to eliminate blurring caused by vibrations during vehicle movement. Furthermore, the size and resolution of the image data can also be adjusted to suit the needs of subsequent processing algorithms while reducing computational burden.

[0043] Optionally, computer vision and deep learning techniques can be used to detect feature points in the image data, such as facial contours, eyes, mouth, and eyebrows. These feature points can be used to locate target areas, helping computer vision and deep learning techniques determine the approximate location of the target area in the image data. Based on the detection results of the feature points, region segmentation can be further performed on the image data to accurately define the target area. For example, by detecting eye and mouth feature points, the upper half (eye region) and lower half (mouth region) of the face can be segmented, and these two regions can be used to assess driver fatigue and emotional state.

[0044] For example, deep learning models can be used to further refine the identification of target parts. Through training, deep learning models can automatically learn and identify image features related to changes in driver state, thereby accurately locating target parts.

[0045] Optionally, once the target region is accurately located, an image segment containing the target region is cropped from the original image data to form target image data. The cropping process ensures the target region is intact and clearly visible, preserving sufficient detail for subsequent analysis. After cropping, the target image data can undergo additional quality checks and enhancements, such as automatic brightness and contrast adjustments, to guarantee the clarity and contrast of the image content, facilitating accurate identification of subtle changes in the target region.

[0046] Step S106: Based on the target image data, determine the photoplethysmography pulse signal of the target location.

[0047] In the technical solution provided by step S106 of the embodiments of this application, the photoplethysmography (PPG) signal is used to represent the degree of volume change of the physiological channel inside the target site. The physiological channel can be a blood vessel.

[0048] In this embodiment, photoplethysmography (PPG) signals can be used to monitor changes in tissue volume caused by blood flow. PPG signals are based on the principle that when light shines on the skin surface, some light is absorbed by the skin tissue, while the rest is reflected or scattered back. Since the heartbeat can periodically cause changes in the amount of blood flowing to the skin, thus affecting tissue volume and light absorption characteristics, by measuring changes in the intensity of transmitted or reflected light, hemodynamic information, including physiological parameters such as heart rate and blood oxygen saturation, can be indirectly inferred.

[0049] Optionally, the photoplethysmography (PPG) signal can be a remote photoplethysmography signal (rPPG). The PPG signal is extracted from the target image data, and by analyzing subtle color changes in the target object's face or other skin areas, the fluctuations in blood flow through that area can be indirectly measured. This is because red blood cells in the blood absorb light of specific wavelengths (such as green light), and as the blood volume changes periodically with the heartbeat, the intensity of the light reflected back to the camera also changes accordingly.

[0050] Optionally, after determining the target image data of the target region from the image data, the photoplethysmography (PPG) signal of the target region can be determined based on the target image data.

[0051] Optionally, noise reduction processing can be performed on the target image data to remove interference signals caused by changes in lighting and camera shake. This can be achieved by using median filtering, Gaussian filtering, or other image processing techniques to ensure image clarity and facilitate subsequent analysis. Since changes in ambient light may affect signal quality, it is necessary to standardize the image brightness to ensure consistency in rPPG signal extraction under different lighting conditions.

[0052] Optionally, the skin region of the target area can be located and segmented. The above process is used to accurately extract rPPG signals from complex backgrounds. Deep learning techniques, such as convolutional neural networks, can be used for more accurate skin region segmentation, improving the extraction accuracy of rPPG signals. The target image data can be converted into light intensity data, focusing on the green light channel, as changes in the reflection of green light on the skin surface best reflect blood fluctuations. The average brightness value of a specific skin region in the image at each time point is calculated. Methods such as Fourier transform or wavelet transform are used to analyze the frequency characteristics of light intensity changes to identify signal components corresponding to pulse frequency. Since reflected light signals can contain various factors (such as pulse signals, respiratory signals, environmental noise, etc.), effective signal separation methods can be used to separate independent signal sources from the mixed signals, thereby extracting rPPG signals more accurately.

[0053] In this embodiment of the application, the rPPG signal extracted from the target image data can reflect the degree of volume change of the physiological channels inside the target area, which can be used for real-time monitoring of the driver's state, thereby realizing early warning and intervention and helping to improve driving safety.

[0054] Step S108: Determine the physiological state of the target object based on the photoplethysmography pulse signal.

[0055] In the technical solution provided by step S108 in the embodiments of this application, the physiological state is used to at least indicate the safety level of the target object triggering control operations.

[0056] In this embodiment, physiological state can be used to represent the health and functional performance of the target object during the triggering control operation, and may include, but is not limited to, vital signs such as heart rate, blood pressure, blood oxygen level, respiratory rate, and body temperature, as well as deeper physiological changes, such as fatigue level, mental stress state, and mood fluctuations. Physiological state can be used to assist the vehicle in determining whether the driver is suitable to continue driving, or to take warning or proactive intervention measures. In this embodiment, if the target object is a driver, the physiological state can be the driving state.

[0057] Optionally, after determining the photoplethysmography (PPG) signal of the target region based on the target image data, the physiological state of the target object can be determined based on the PPG signal.

[0058] Optionally, rPPG signals are susceptible to interference from ambient light, camera shake, and facial expression changes. Therefore, in the preprocessing stage, filtering techniques (such as bandpass filters) can be applied to remove or reduce noise in the rPPG signal while retaining information related to physiological state. The rPPG signal may contain multiple physiological signals such as heart rate, respiratory rate, and blood pressure. Signal separation techniques (such as independent component analysis) can be used to decompose the mixed signal and extract independent physiological signal components. These physiological signal components may include heart rate, respiratory rate, blood oxygen saturation, and blood pressure.

[0059] Optionally, heart rate can be determined by identifying peaks in the rPPG signal. For example, the time interval between peaks can be used to calculate the heart rate. In the frequency domain, the heart rate frequency is identified using Fourier transform or wavelet transform to further confirm the heart rate value. Respiratory rate is represented as lower-frequency fluctuations in the rPPG signal. Through spectral analysis, the frequency of the respiratory rate component is identified and calculated, thereby determining the respiratory rate. Certain characteristics of the rPPG signal, such as the slope and width of the waveform, can be used to estimate blood pressure. By comparing with individuals whose blood pressure values ​​are known, a model can be built to predict blood pressure levels.

[0060] Optionally, prolonged driving or work can cause changes in physiological parameters such as heart rate, respiratory rate, and blood pressure. By comparing baseline and current values, the driver's level of fatigue can be assessed. For example, increases in heart rate and respiratory rate, and subtle changes in blood pressure, can indicate fatigue. Emotional fluctuations, such as anger, anxiety, or relaxation, can also leave traces in rPPG signals. Emotional excitement can cause significant changes in heart rate and blood pressure; by analyzing patterns of these changes, different emotional states can be identified. rPPG signals can reflect the driver's health status, such as the presence of hypoxia or cardiovascular disease. For example, abnormal blood oxygen saturation or heart rate change patterns can indicate underlying health problems.

[0061] In this embodiment, real-time analysis of rPPG signals enables continuous monitoring of the physiological state of the target object, instantly identifying potential risks or anomalies. Once a physiological state detrimental to safe driving (such as fatigue or health problems) is detected, the system immediately activates an early warning mechanism, alerting the driver through visual, auditory, or tactile signals. If necessary, proactive intervention measures are taken, such as automatic deceleration, emergency stopping, or calling for roadside assistance. By analyzing the physiological state of the target object from rPPG signals using the above method, crucial early warning information can be provided to the intelligent driving assistance system, helping to improve driving safety and prevent accidents.

[0062] In steps S102 to S108 of this embodiment, during the process of the target object performing control operations on the vehicle, image data corresponding to the target object can be acquired in real time. The target part reflecting the state change process of the target object, and the target image data corresponding to that target part, can be identified from the image data. Based on the target image data, the degree of volume change of the physiological channels inside the target part can be analyzed to obtain the corresponding photoplethysmography (PPG) signal. The PPG signal can be used to determine the physiological state of the target object. In other words, in this application, by acquiring image data of the target object in real time during the process of performing control operations on the vehicle, the target part reflecting state changes is accurately located, and the PPG signal is extracted and analyzed to comprehensively evaluate the physiological state of the target object. The above method overcomes the limitations of relying solely on explicit features in related technologies, effectively capturing and utilizing the physiological signal changes implicit in the image data, significantly improving the accuracy and comprehensiveness of state recognition, especially in early warning of fatigue and distraction, thereby achieving the technical effect of improving the accuracy of state recognition of objects in vehicles and solving the technical problem of low accuracy in state recognition of objects in vehicles.

[0063] The embodiments of this application will be described in detail below with reference to the steps described above.

[0064] As an optional implementation, step S108, determining the physiological state of the target object based on the photoplethysmography (PPG) signal, includes: analyzing the PPG signal using a state recognition model to obtain the physiological state, wherein the state recognition model is obtained by training a deep learning model.

[0065] In this embodiment, during the process of determining the physiological state based on the photoplethysmography (PPG) signal, a state recognition model can be used to analyze the PPG signal to obtain the physiological state.

[0066] Alternatively, a deep learning model can refer to a neural network containing multiple hidden layers, capable of automatically learning complex features from data. For rPPG signal analysis, deep learning models such as convolutional neural networks, recurrent neural networks, and long short-term memory networks can be used.

[0067] Optionally, data preprocessing can be performed before analyzing the rPPG signal to optimize the quality of the input data. Preprocessing steps may include: applying filtering techniques to remove noise and interference from the signal, such as ambient light variations and motion artifacts; and normalizing the rPPG signal to the same range or unit to eliminate signal amplitude differences, which is beneficial for model training. Although deep learning models can automatically learn features, moderate feature engineering is still beneficial, such as extracting the frequency components of the signal or calculating statistical indicators to provide initial clues for model training.

[0068] Optionally, an rPPG signal dataset containing various physiological states is collected. Each signal sample should have a corresponding target physiological state label, such as "normal," "fatigue," or "emotional excitement." The model is trained using this labeled dataset. By adjusting internal parameters (such as weights and biases), the model learns how to identify the target physiological state from rPPG signals. After selecting a suitable model architecture, hyperparameters (such as learning rate, batch size, and hidden layer size) can be tuned to achieve good training results.

[0069] Optionally, after the model training is complete, the corresponding physiological state category can be output based on the input rPPG signal.

[0070] In this embodiment, the process of analyzing the physiological state of a target object using a deep learning model based on rPPG signals demonstrates the application potential of artificial intelligence in the field of non-contact health monitoring. This method not only provides real-time health status assessment but also continuously improves accuracy through an adaptive learning mechanism, laying the foundation for intelligent driving assistance systems and health management platforms.

[0071] As an optional implementation method, the state recognition model is a time series analysis model. The state recognition model is used to analyze the photoplethysmography (PPG) signal to obtain the physiological state. This includes: inputting the PPG signal into the time series analysis model, and using the time series analysis model to analyze the degree of change of the PPG signal over time to obtain the physiological state.

[0072] In this embodiment, if the state recognition model is a time series analysis model, the time series analysis model can be used to analyze the degree of change of the photoplethysmography pulse signal over time to obtain the physiological state.

[0073] Optionally, time series analysis models can be used to process time-varying photoplethysmography (PPG) signals, capturing trends, periodicity, and randomness in the PPG signals.

[0074] Optionally, noise in the rPPG signal, such as interference from ambient light changes, facial expressions, or muscle activity, should be removed. The continuous rPPG signal should be segmented, with each segment containing sufficient temporal information for model analysis. Key features in the rPPG signal, such as intensity, frequency, and phase variations, should be extracted, as these features are closely related to physiological states. A dataset of rPPG signals under various physiological states should be constructed, with each signal sample having a corresponding physiological state label. The preprocessed rPPG signal should be used as input, and the labeled physiological state should be used as output to train the time series analysis model to learn the mapping relationship between the signal and the physiological state. Model parameters should be adjusted based on the training results to optimize model performance and ensure that it can accurately distinguish signal features under different physiological states.

[0075] Optionally, rPPG signals are continuously collected and input into a time-series analysis model. The time-series analysis assesses the changes in rPPG signals over time, identifying patterns or abnormalities related to physiological states. Based on dynamic analysis of rPPG signals, the physiological state of the target individual, such as fatigue, stress, or health abnormalities, is predicted. The physiological state information output by the time-series analysis model is visualized for real-time detection and understanding. Based on the physiological state assessment results, early warning signals can be issued, or intelligent driving intervention measures can be implemented, such as reminding for rest, automatic deceleration, or emergency braking.

[0076] In this embodiment, the physiological state of a target object can be effectively analyzed from rPPG signals using a time series analysis model. The process encompasses not only signal preprocessing, feature extraction, and model training, but also real-time signal analysis, physiological state prediction, result application, and continuous model optimization. Furthermore, the time series analysis-based method enables real-time monitoring and early warning of the driver's health status.

[0077] As an optional implementation, the time series analysis model includes a first input layer, a convolutional layer, a pooling layer, a recurrent layer, a first fully connected layer, and a first output layer. The photoplethysmography (PPG) signal is input into the time series analysis model, which analyzes the changes in the PPG signal over time to obtain the physiological state. This includes: in response to the first input layer receiving the PPG signal, using the convolutional layer to extract features from the PPG signal to obtain local feature information; using the pooling layer to perform dimensionality reduction on the local feature information to obtain dimensionality-reduced local feature information; using the recurrent layer to obtain the correlation between the dimensionality-reduced local feature information and time, where the correlation represents the relationship between the dimensionality-reduced local feature information and time; using the first fully connected layer to integrate the relationship, local feature information, and dimensionality-reduced local feature information to obtain an integrated result, and mapping the integrated result to a target space to obtain a preset physiological state value, where different dimensions of the target space correspond to different preset physiological state values; and using the activation function in the first output layer to process the preset physiological state value to obtain the physiological state.

[0078] In this embodiment, the time series analysis model may include a first input layer, a convolutional layer, a pooling layer, a recurrent layer, a first fully connected layer, and a first output layer.

[0079] Optionally, the first input layer receives the rPPG signal as input, which is presented in the form of a time series, i.e., a series of values ​​that change over time. This ensures that the input data format meets the requirements of subsequent layers, providing a direct signal data source for the time series analysis model. The convolutional layer extracts local features from the rPPG signal through convolution operations. The convolutional kernel can slide across the rPPG signal, calculating a weighted sum of local regions to identify specific patterns, such as heartbeat waveforms. The convolutional layer can capture the basic structure and patterns in the rPPG signal. The pooling layer can perform dimensionality reduction on the local feature information extracted by the convolutional layer, for example, by taking the maximum or average value within a region, reducing the amount of data while retaining important feature information. This reduces computational complexity, accelerates model training, avoids overfitting, and ensures the generalization ability of the time series analysis model.

[0080] Optionally, the recurrent layer can be a Long Short-Term Memory (LSTM) network, a Gated Recurrent Unit (GRU), or a Recurrent Neural Network (RNN). The recurrent layer can analyze the correlation between dimensionality-reduced feature information and time. It can remember information from the sequence and use this memory to understand long-term dependencies, thus identifying physiological states that change over time in the rPPG signal. It captures the dynamic changes of the rPPG signal over time and identifies indicators reflecting physiological states such as heart rate variability and respiratory rate.

[0081] Optionally, the first fully connected layer can integrate temporal correlation information from the recurrent layer and dimensionality reduction feature information from the pooling layer to form a higher-level feature representation. This transforms the complex information of the time series into an interpretable intermediate representation, facilitating the final classification or regression task. The output of the first fully connected layer is mapped to the target space to obtain preset values ​​corresponding to different physiological states.

[0082] Optionally, the first output layer may include an activation function, such as a softmax function, to transform the integrated result into a probability distribution or continuous value, representing the likelihood or prediction of different physiological states. This transforms the model's internal representation into intuitive physiological state predictions, facilitating system understanding and appropriate action.

[0083] Optionally, the rPPG signal enters the model from the first input layer and undergoes preprocessing steps (such as denoising). Convolutional layers extract local features from the signal, followed by pooling layers for dimensionality reduction, retaining the most significant feature information. Recurrent layers capture the relationship between the dimensionality-reduced features and time, understanding the patterns of time-varying changes in the signal. The first fully connected layer integrates time-dependent and local feature information, mapping the aforementioned high-dimensional features to the target space to obtain preset values ​​corresponding to different physiological states. The activation function in the first output layer converts the preset values ​​into easily understandable physiological state predictions, such as fatigue level, emotional state, and health status.

[0084] In this embodiment, the time series analysis model is designed to automatically learn and identify the physiological state of a target object from continuous rPPG signals, providing real-time monitoring and early warning functions for intelligent driving assistance systems and health management platforms. Through layer-by-layer processing, the model can handle complex time-series data, capture long-term dependencies, and effectively reduce the impact of noise and redundant information, ultimately achieving accurate physiological state assessment.

[0085] As an optional implementation, the state recognition model is an attention mechanism model. The state recognition model is used to analyze the photoplethysmography (PPG) signal to obtain the physiological state. This includes: inputting the PPG signal into the attention mechanism model, using the attention mechanism model to determine the importance of different PPG signals to the physiological state, and obtaining the physiological state.

[0086] In this embodiment, if the state recognition model is an attention mechanism model, the attention mechanism model can be used to determine the importance of different photoplethysmography pulse signals to the physiological state, thereby obtaining the physiological state.

[0087] Alternatively, attention mechanisms can be used to emphasize certain parts of sequence data while ignoring other, less important parts. Applying attention mechanisms to rPPG signal analysis can more accurately capture signal segments that are important for physiological state assessment, thereby improving the accuracy and robustness of state identification.

[0088] Optionally, the rPPG signal is input as time-series data into the first layer of the model. The signal may have undergone preliminary preprocessing, such as denoising and normalization, to facilitate model analysis. Convolutional layers or recurrent neural network layers (such as LSTM or GRU) may be included, which are used to extract key features from the rPPG signal, such as dynamic patterns of heart rate, respiratory rate, etc.

[0089] In this embodiment, the attention mechanism can automatically identify key information segments in rPPG signals, improving the accuracy and efficiency of the model's assessment of physiological states. For rPPG signals with long durations, the attention mechanism can handle long signal sequences. The dynamic adjustment of attention weights enables the model to maintain flexibility and adaptability across different scenarios and individuals, improving the model's generalization ability.

[0090] As an optional implementation, the attention mechanism model includes a second input layer, a feature extraction layer, an attention layer, a second fully connected layer, and a second output layer. The photoplethysmography (PPG) signal is input into the attention mechanism model, and the model analyzes the PPG signal to obtain a physiological state. This includes: in response to the second input layer receiving the PPG signal, the feature extraction layer extracts key feature information from the PPG signal, wherein the importance of the key feature information to the physiological state is greater than the importance of other feature information; the attention layer performs weighted processing on the key feature information to obtain weighted key feature information; the second fully connected layer maps the weighted key feature information to a corresponding preset physiological state value; and the second output layer evaluates the preset physiological state value to obtain the physiological state.

[0091] In this embodiment, the second input layer is used to acquire the rPPG signal. The rPPG signal may have undergone preliminary preprocessing, such as noise reduction and normalization, to improve the efficiency and effectiveness of subsequent processing. The feature extraction layer is used to extract key feature information, specifically extracting key feature information highly correlated with physiological state from the original rPPG signal. Key feature information may include, but is not limited to, time-series data such as heart rate, respiratory rate, and blood oxygen level. The feature extraction layer may consist of multiple convolutional layers and pooling layers, used to capture local patterns in the signal and perform dimensionality reduction processing, thereby extracting key feature information. The accurate extraction of key feature information directly affects the model's performance and prediction accuracy.

[0092] Optionally, the attention layer can be used to weight key feature information. The core of the attention layer is to assign weights to the key feature information obtained from the feature extraction layer. These weights reflect the importance of these key feature information to the final physiological state prediction. The attention layer can use the Softmax function to calculate the attention weights of key features. The key feature information is multiplied by a specific query vector to generate a similarity score. These scores are then normalized using the Softmax function to obtain the weight of each feature. Through weighted processing, more attention can be paid to feature information closely related to physiological state assessment, while irrelevant or noisy features are ignored, improving the efficiency and accuracy of the attention mechanism model.

[0093] Optionally, the second fully connected layer can be used to integrate the weighted features. Here, the fully connected layer integrates the feature information after weighting by the attention layer, combining multiple features into a lower-dimensional space to facilitate subsequent physiological state prediction. Each neuron linearly combines the weighted feature information and then generates an output through an activation function. Preset values ​​corresponding to different physiological states are generated, representing the probability or degree of a specific physiological state.

[0094] Optionally, the second output layer is used to assess physiological states, transforming the output of the second fully connected layer into the final physiological state assessment result. An activation function is used to generate a probability distribution representing the likelihood of different physiological states. Each node in the output layer typically corresponds to a physiological state, and the activation level of the node reflects the predicted probability or degree of that physiological state. The output of the second output layer can be used to monitor the physiological states of the driver or other individuals in real time, such as fatigue, anxiety, or low blood oxygen, and then take appropriate intervention measures, such as reminding them to rest, automatically adjusting vehicle settings, or notifying health services.

[0095] Optionally, the rPPG signal enters from the second input layer, passes through a feature extraction layer to mine potential physiological state-related features, and then enters an attention layer to weight these features to emphasize key information. The weighted features are then fed into the second fully connected layer for integration, and finally transformed into a physiological state prediction result through an activation function in the second output layer. This process fully utilizes the advantages of the attention mechanism, effectively identifying the most critical segments in the rPPG signal for physiological state assessment, resulting in higher accuracy and robustness of the model.

[0096] In this embodiment of the application, by integrating the attention mechanism into the deep learning model, not only can key features related to physiological state be extracted from rPPG signals more effectively, but the importance of these features can also be highlighted through weighted processing, thereby achieving a more accurate and timely assessment and early warning of physiological state.

[0097] As an optional implementation, the method may further include: recognizing image data using a preset recognition model to obtain physiological state labels for the target object; and training the state recognition model using the physiological state labels and image data to obtain a trained state recognition model, wherein the physiological state labels are used to represent the physiological state determined by the preset recognition model; or, analyzing photoplethysmography (PPG) signals using the state recognition model to obtain physiological states, including: recognizing image data using the preset recognition model to obtain physiological state labels for the target object; and analyzing PPG signals using the state recognition model to obtain initial physiological states, wherein the initial physiological states are the physiological states determined by the state recognition model; and selecting physiological states from the physiological state labels and initial physiological states.

[0098] In this embodiment, the pre-trained recognition model can refer to a pre-trained deep learning model, such as a convolutional neural network or other visual recognition model, specifically designed to identify the physiological state features of a target object from image data. Such models are typically trained on large labeled datasets and learn how to infer physiological states from visual cues such as facial expressions, posture, and skin color changes. The pre-trained recognition model processes image data, identifies the physiological state of the target object, and generates corresponding physiological state labels. These labels provide supervisory information for subsequent training of the state recognition model, helping the model learn its ability to identify physiological states from rPPG signals. The pre-trained recognition model can also be referred to as a traditional algorithm.

[0099] Optionally, the physiological state label is the result of image data analysis by a preset recognition model, and can be used to represent the classification or evaluation of the target object's physiological state by the preset recognition model. The physiological state label can be a specific physiological state category (such as "fatigue," "anxiety," "relaxation," etc.) or a state severity rating (such as fatigue level from 0 to 10). In this embodiment, the physiological state label has two main uses: as supervised information to train the state recognition model, allowing the model to learn the association between rPPG signals and specific physiological states; and by comparing the physiological state obtained from the state recognition model's analysis of rPPG signals with the initial physiological state, a more accurate physiological state assessment can be selected.

[0100] Optionally, the initial physiological state can be a physiological state prediction result directly obtained by the state recognition model based on photoplethysmography pulse signal analysis. The initial physiological state provides a physiological state assessment perspective based on biosignals, and can be used alone as a physiological state prediction result, or integrated with physiological state labels to improve the accuracy and reliability of the final physiological state assessment.

[0101] Optionally, this embodiment combines image data and photoplethysmography (PPG) signals to comprehensively assess the physiological state of the target object, aiming to improve the accuracy and robustness of the assessment.

[0102] Optionally, a pre-defined recognition model utilizes deep learning techniques (such as convolutional neural networks, CNNs) to process image data and identify features related to physiological states, such as fatigue, inattention, and tension. Based on the identified features, the pre-defined model outputs physiological state labels. The image data and the physiological state labels generated by the pre-defined recognition model are integrated as training samples to train the state recognition model. The goal of the above steps is to enable the model to predict physiological states based on image data. Supervised learning methods, such as the backpropagation algorithm, are used to adjust model parameters based on the difference between the model's prediction results and the physiological state labels to minimize prediction errors and improve the model's prediction performance.

[0103] Optionally, the rPPG signal is input into a state recognition model. The model uses parameters previously trained on visual data to analyze physiologically relevant features in the rPPG signal, obtaining a physiological state assessment result based on biosignals, i.e., the initial physiological state. The physiological state label generated by the preset recognition model is compared with the initial physiological state obtained by the state recognition model based on rPPG signal analysis. The two models produce different assessment results due to their different analytical perspectives. According to the consistency principle, if the two models provide consistent physiological state assessment results for the same time period, this assessment result will be adopted; if the results are inconsistent, a comprehensive judgment can be made based on factors such as model confidence and historical accuracy to select the most reliable assessment result. Through the above screening process, a final physiological state assessment combining visual information and biosignals can be obtained, and this assessment result is more comprehensive and accurate than using information from either modality alone.

[0104] In this embodiment, visual information (image data) and biosignals (rPPG signals) play different roles in physiological state assessment. Combining the two can overcome the limitations of a single information source and improve the accuracy of the assessment. Lighting conditions or facial occlusion may affect the reliability of visual information, while biosignals are more stable; conversely, the same applies. By integrating these two types of information, high robustness can be maintained under various conditions. Through the above steps, the integrated approach using bimodal information can more comprehensively and accurately assess the physiological state of the target object, providing technical support for intelligent assisted driving, health monitoring, and work and life status management.

[0105] As an optional implementation, the method may further include at least one of the following: in response to an abnormal physiological state, controlling the vehicle to output a prompting message, wherein the prompting message is used to prompt the target object to change from an abnormal physiological state to a normal physiological state; in response to an abnormal physiological state, or if the target object fails to change from an abnormal physiological state to a normal physiological state, adjusting the control operation, wherein the adjusted control operation is used to improve the safety of the target object performing the control operation; in response to an abnormal physiological state, or if the target object fails to change from an abnormal physiological state to a normal physiological state, adjusting the safe distance between the vehicle and the obstacle object based on the vehicle's driving speed and the speed of the obstacle object moving in the vehicle's driving direction; or, step S104, determining target image data from image data, including: identifying the image data to determine the location of the target part in the image data; identifying the image data of the location of the target part to obtain key points of the target part, and determining the image data of the area where the key points are located as the target image data.

[0106] This embodiment illustrates how to ensure driving safety through feedback and active control of intelligent vehicles when an abnormal physiological state of a target object (driver or passenger) is detected.

[0107] Optionally, when the state recognition model determines that the target is in an abnormal physiological state, such as fatigue, anxiety, or poor health, the vehicle will be triggered to output specific prompts. These prompts can be audible warnings, tactile feedback (such as seat vibration), screen display suggestions, or automatically played soothing music. The aim is to remind the target of their current physical condition and encourage them to take action to restore a normal physiological state, such as taking a short rest, taking deep breaths, or adjusting their driving posture. Timely prompts help the target become aware of their abnormal state and proactively take measures to improve it, thereby reducing driving risks caused by physiological abnormalities.

[0108] Optionally, if the abnormal physiological state of the target object persists, or if the target object fails to successfully transition from the abnormal physiological state to a normal state, the vehicle control system will automatically adjust the driving strategy or the level of autonomous driving intervention. Adjustments to control operations may include automatic deceleration, smooth driving, switching to autonomous driving mode, or restricting certain vehicle operations (such as preventing acceleration). This helps reduce sudden dangers caused by poor driver condition and improves overall driving safety. By actively adjusting control operations, the vehicle can automatically take measures when the target object's physiological state is abnormal, reducing the risk of accidents and protecting the safety of occupants and pedestrians.

[0109] Optionally, the vehicle uses sensors such as radar, lidar, and cameras to perceive the surrounding environment in real time, including the position, speed, and direction of obstacles, as well as the vehicle's own speed and direction. Based on this information, the vehicle control system can dynamically calculate the safe distance to be maintained between the vehicle and obstacles. If an abnormal physiological state of the target object is detected, the system will tend to increase the safe distance to cope with possible slow reaction or misjudgment. By dynamically adjusting the safe distance to obstacles, even when the driver's physiological state is abnormal, driving safety can be maximized, and collisions can be avoided.

[0110] Optionally, a preset face detection algorithm or other localization method is used to analyze the image data to determine the location of the target object's (usually the driver's) face or other parts of interest in the image. Key points of the target parts, such as the eyes, nose, mouth, and ears, are further precisely located; these key points are used for subsequent physiological state assessment. Based on the location of these key points, image data containing these key points and their surrounding areas is extracted; this image data can be used as target image data. It contains visual information closely related to physiological state assessment, such as facial expressions and eyelid closure. By accurately extracting target image data, interference from irrelevant information can be reduced, improving the accuracy and efficiency of subsequent physiological state assessment.

[0111] In this embodiment, the method described above demonstrates how physiological state assessment results can be applied to an intelligent driving assistance system. Through timely feedback, proactive control adjustments, and dynamic safe distance management, it ensures that the vehicle can maintain safe driving even when faced with abnormal driver physiological states. Simultaneously, by optimizing the image data processing flow, the accuracy and robustness of physiological state detection are enhanced.

[0112] Below, we will further explain the method for recognizing the state of objects in the vehicle in the above embodiments of this application, in conjunction with the driver's emotional state prediction method.

[0113] Currently, with the rapid advancement of the automotive industry, driver safety has gradually become a focus of attention for both the industry and the public. To ensure driving safety, various driver monitoring systems have emerged on the market. These systems are mainly divided into two categories: active monitoring systems and passive monitoring systems. Active monitoring systems determine whether a driver is fatigued or distracted by monitoring their behavior and physiological state in real time, such as eyelid closure, head position, and heart rate. Passive monitoring systems typically react after the driver exhibits abnormal behavior or state, such as alerting the driver through lane departure warnings or collision warnings. These systems, to a certain extent, improve driving safety and reduce the occurrence of traffic accidents.

[0114] However, existing driver monitoring systems still have significant shortcomings and deficiencies. Most systems only begin monitoring and reacting after the driver has already exhibited abnormal behavior or condition. This lag can lead to untimely warnings and interventions, thus failing to effectively prevent traffic accidents. These systems often fail to accurately identify early physiological abnormalities in drivers, such as slight changes in heart rate or blood pressure fluctuations. These physiological abnormalities may be precursors to driver fatigue, distraction, or illness. Because they cannot capture these early signals in time, existing systems struggle to issue warnings before the driver's condition deteriorates. Furthermore, the accuracy and reliability of existing systems are also affected by environmental factors and individual driver differences, sometimes resulting in false alarms or missed alarms. Therefore, developing a method that can predict driver physiological abnormalities in advance, enabling earlier warnings and interventions, is significant for further improving driving safety and preventing traffic accidents.

[0115] Existing systems often react only after drivers have exhibited obvious abnormal behavior or conditions, resulting in delayed warnings and an inability to effectively prevent impending traffic accidents. Current technology struggles to accurately capture early physiological abnormalities in drivers, such as subtle changes in heart rate and blood pressure fluctuations, which could be early warning signs of driver fatigue, distraction, or health problems. Existing systems are also susceptible to environmental factors (such as lighting and weather) and individual driver differences, leading to inaccurate monitoring results and false alarms or missed alarms. Due to these issues, existing systems cannot provide timely and effective warnings and interventions before a driver's condition deteriorates, thus increasing the risk of traffic accidents.

[0116] This application provides a method for predicting driver physiological abnormalities in advance, aiming to achieve earlier and more accurate early warning and intervention, effectively solving the aforementioned technical problems, thereby significantly improving driving safety and preventing traffic accidents. It provides a method for predicting and detecting abnormal physiological states of drivers, issuing timely warnings when drivers experience fatigue, anger, or excessive tension without obvious facial changes. The method includes the following steps: acquiring continuous frame images of the driver in real time using a camera installed in the vehicle; processing the acquired images in real time using artificial intelligence algorithms to accurately identify the location of faces in the images; extracting effective skin modules from the identified face regions using one or more image processing methods; analyzing micro-changes in the skin surface using a specific algorithm to detect the driver's rPPG signal; inputting the detected rPPG signal into a specific algorithm module to calculate the driver's drowsiness, fatigue, hypoxia, anger, and other states detrimental to safe driving; and based on the prediction results, the vehicle system issues corresponding warnings, alarms, and intelligent driving intervention actions, such as reminding the driver to rest, automatically slowing down or stopping, and calling for emergency assistance.

[0117] The methods of the embodiments of this application will be further illustrated below.

[0118] Figure 2 This is a flowchart of a driver's emotional state prediction method according to an embodiment of this application, such as... Figure 2 As shown, the method may include the following steps:

[0119] Step S201: Acquire consecutive frames of image data of the driver.

[0120] In this embodiment, a camera installed inside the vehicle continuously captures images of the driver's face. Acquiring these consecutive frames of image data allows for real-time monitoring of the driver's emotions and physiological changes. By capturing these consecutive frames, the algorithm can identify minute changes over time.

[0121] Optionally, after acquiring the first continuous image of the driver, a second continuous image of the driver can be continuously acquired, and the driver's state can be monitored using a traditional algorithm on the second continuous image. The traditional algorithm result of the second continuous image is used as the data result label of the first image, and the algorithm of step S204 is trained using the dataset of step S204.

[0122] Optionally, after acquiring the first continuous image of the driver, the second continuous image of the driver can be continuously acquired. The algorithm module of this method and the traditional algorithm are deployed on the vehicle at the same time. The first continuous image and the second continuous image are used as a set of data to form an adversarial network. During the use of the vehicle, the accuracy of the algorithm gradually improves with the increase of time.

[0123] Step S202: Use a face recognition algorithm to determine the face location and key point data.

[0124] In this embodiment, facial recognition algorithms, such as deep learning models, are applied to the acquired image data to locate the driver's facial region and identify key feature points such as the eyes, nose, and mouth. These steps prepare for subsequent, more accurate emotion state analysis. Extracting key point data helps the algorithm focus on key areas of facial expressions, improving the accuracy of emotion recognition.

[0125] Step S203: Identify the skin area of ​​interest.

[0126] In this embodiment, based on the acquired key point data, the skin areas of the driver's face are determined, especially those areas that can reflect emotional changes, such as the forehead and cheeks. The image information contained in the skin areas of interest can be used for subsequent rPPG signal detection, as well as the analysis of possible physiological parameters such as facial temperature and blood flow, which are directly related to emotional state.

[0127] Step S204: Determine the driver's status using the algorithm module.

[0128] In this embodiment, the data obtained in steps S201 to S203 are input into an algorithm module. This module may include deep learning models such as CNN, RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory Network), or GRU (Gated Recurrent Unit) to analyze the driver's emotions and physiological state. This step aims to predict the driver's emotional state, such as fatigue, anxiety, or pleasure, and possible physiological abnormalities, such as heart rate changes and blood pressure fluctuations, based on the collected data. The algorithm module may include a state recognition model, such as a time series analysis model and an attention mechanism model.

[0129] Figure 3(a) is a schematic diagram of a state recognition model according to an embodiment of this application. As shown in Figure 3(a), if the state recognition model is a time series analysis model, it may include an input layer 301, a convolutional layer + pooling layer 302, an RNN / LSTM / GRU 303, a fully connected layer 304, and an input layer 305. The input layer 301 receives the original rPPG signal data. The convolutional layer + pooling layer 302 uses a convolutional kernel to perform a sliding window operation on the rPPG signal data to extract local features. These features may include peaks, valleys, and frequency components in the signal. The pooling layer in the convolutional layer + pooling layer 302 performs dimensionality reduction on the features extracted by the convolutional layer while retaining the most important information. The RNN / LSTM / GRU 303 is used because, considering that the rPPG signal is time series data, simple convolutional and pooling layers may not be able to capture long-term temporal dependencies. Therefore, RNN, LSTM, or GRU layers are introduced at this stage to capture the temporal relationship of features within the duration of the rPPG signal. The temporal distribution of these features reflects physiological changes such as fatigue, stress, and emotional state in the tested individual. Fully connected layer 304: After processing by the above layers, the features have been sufficiently extracted and dimensionality reduced. These features are integrated and mapped to predicted values ​​of fatigue, stress, and emotional state. Input layer 305: As the final layer of the neural network, the output layer is responsible for outputting the final evaluation results. These results are obtained based on the predictions from the fully connected layers, processed by an appropriate activation function (such as the softmax function), and can be represented as a quantitative evaluation of fatigue, stress, and emotional state.

[0130] Figure 3(b) is a schematic diagram of another state recognition model according to an embodiment of this application. As shown in Figure 3(b), if the state recognition model is an attention mechanism model, it may include: an input layer 306, a feature extraction layer 307, an attention layer 308, a fully connected layer 309, and an output layer 310. The input layer 306 receives rPPG signal data. The feature extraction layer 307 extracts key feature information from the rPPG signal data; these feature information exhibit different feature value changes when there are strong emotional fluctuations and drowsiness. This layer may be a multi-layer convolutional pooling layer or a multi-layer recurrent neural network. The attention layer 308 weights the features to highlight key information. The fully connected layer 309 maps the weighted features to predicted values ​​for fatigue, stress, and emotional state. The output layer 310 outputs the evaluation results for fatigue, stress, and emotional state.

[0131] Step S205: Based on the driver's condition, intervene and issue a warning.

[0132] In this embodiment, the vehicle intelligent system takes corresponding intervention measures based on the driver's predicted state by the algorithm module. For example, if the driver is determined to be fatigued, the system may issue an audible warning to prompt the driver to rest; if the driver is in an anxious state, it can automatically play soothing music or adjust the atmosphere inside the vehicle. By monitoring the driver's emotional state in real time and taking intervention measures, the system can prevent potential driving risks and improve driving safety.

[0133] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0134] According to another aspect of the embodiments of this application, corresponding to the embodiments of the above-described method for recognizing the state of objects in a vehicle, this specification also provides a device for recognizing the state of objects in a vehicle.

[0135] Figure 4 This is a schematic diagram of a vehicle object state recognition device according to an embodiment of this application, such as... Figure 4As shown, the object state recognition device 40 in the vehicle may include: an acquisition module 402, a first determination module 404, a second determination module 406, and a third determination module 408. The acquisition module 402 is used to acquire image data corresponding to the target object during the vehicle's response to a control operation triggered by the target object. The first determination module 404 is used to determine target image data of a target part of the target object from the image data. The second determination module 406 is used to determine the photoplethysmography (PPG) signal of the target part based on the target image data. The third determination module 408 is used to determine the physiological state of the target object based on the PPG signal.

[0136] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application when it runs.

[0137] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0138] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0139] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0140] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of this application.

[0141] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0145] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0146] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for recognizing the state of objects in a vehicle, characterized in that, include: During the process of the vehicle responding to a control operation triggered by the target object, image data including the target object is acquired; From the image data, target image data of the target part of the target object is determined, wherein the image content of the target image data includes at least the target part, and the target part is used to reflect the change process of the state of the target object; Based on the target image data, the photoplethysmography (PPG) signal of the target region is determined, wherein the PPG signal is used to represent the degree of volume change of the physiological channels inside the target region. Using a state recognition model, the photoplethysmography pulse signal is analyzed to obtain a physiological state, wherein the physiological state is at least used to represent the safety level of the target object triggering the control operation; The state recognition model is an attention mechanism model, which includes a feature extraction layer, an attention layer, a second fully connected layer, and a second output layer. Using the state recognition model, the photoplethysmography (PPG) signal is analyzed to obtain the physiological state, including: using the feature extraction layer to extract key feature information from the PPG signal; using the attention layer to perform weighted processing on the key feature information; using the second fully connected layer to map the weighted key feature information to corresponding preset physiological state values; and using the second output layer to evaluate the preset physiological state values ​​to obtain the physiological state.

2. The method according to claim 1, characterized in that, The state recognition model is obtained by training a deep learning model.

3. The method according to claim 1, characterized in that, The attention mechanism model further includes a second input layer, which uses the feature extraction layer to extract key feature information from the photoplethysmography pulse signal, including: In response to the second input layer receiving the photoplethysmography (PPG) signal, the feature extraction layer extracts the key feature information from the PPG signal, wherein the key feature information is more important to the physiological state than the feature information other than the key feature information is to the physiological state.

4. The method according to claim 1, characterized in that, The method further includes: The image data is identified using a preset recognition model to obtain a physiological state label of the target object, and the physiological state label and the image data are used to train the state recognition model to obtain the trained state recognition model, wherein the physiological state label is used to represent the physiological state determined by the preset recognition model; or, Using the state recognition model, the photoplethysmography (PPG) signal is analyzed to obtain the physiological state, including: The image data is identified using a preset recognition model to obtain the physiological state label of the target object, and the photoplethysmography signal is analyzed using the state recognition model to obtain the initial physiological state, wherein the initial physiological state is the physiological state determined by the state recognition model. The physiological state is selected from the physiological state label and the initial physiological state.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes at least one of the following: In response to the physiological state being an abnormal physiological state, the vehicle is controlled to output a prompting message, wherein the prompting message is used to prompt the target object to change from the abnormal physiological state to a normal physiological state; In response to the physiological state being an abnormal physiological state, or the target object failing to transition from the abnormal physiological state to the normal physiological state, the control operation is adjusted, wherein the adjusted control operation is used to improve the safety of the target object performing the control operation; Alternatively, determining the target image data from the image data includes: The image data is identified to determine the location of the target part within the image data; The image data of the target location is identified to obtain the key points of the target location, and the image data of the area where the key points are located is determined as the target image data.

6. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 5.

Images