In-vehicle person physiological feature monitoring method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides an in-vehicle person physiological feature monitoring method and device, electronic equipment and a medium. The method comprises the steps that infrared image data and RGB image data collected for a target person in a target vehicle are acquired; inputting the infrared image data and the RGB image data into a target feature extraction model used for performing face feature extraction in combination with multiple pieces of image data; in the target feature extraction model, extracting a first face image feature based on the infrared image data and extracting a second face image feature based on the RGB image data, fusing the first face image feature and the second face image feature, and outputting target face feature data of the target person, and determining physiological feature information of the target person according to the target face feature data. Through the embodiment of the invention, feature extraction is realized through the trained target feature extraction model, the accuracy of feature extraction is improved, and accurate physiological feature information can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a method and device, electronic equipment, and medium for monitoring physiological characteristics of occupants in an intelligent cockpit. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and computer vision technology, driver physiological health characteristic signal detection technology has important significance in improving road safety and driver well-being. The primary goal of driver physiological health characteristic signal detection technology is to improve driving safety.

[0003] At present, the single-mode RGB image visual detection algorithm is greatly affected by lighting, angle, background interference and other factors in face detection. These factors can lead to a decrease in detection performance, especially when these factors are serious, the performance decrease is obvious and it is impossible to accurately detect physiological health characteristic signals. Summary of the invention

[0004] In view of the above problems, a method and device for monitoring physiological characteristics of a person in a vehicle, an electronic device, and a medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A method for monitoring physiological characteristics of a person in a vehicle, the method comprising:

[0006] Acquire infrared image data and RGB image data collected for a target person in a target vehicle;

[0007] Inputting the infrared image data and the RGB image data into a target feature extraction model for extracting facial features by combining multiple image data;

[0008] In the target feature extraction model, a first facial image feature is extracted based on the infrared image data and a second facial image feature is extracted based on the RGB image data, and the first facial image feature and the second facial image feature are fused to output target facial feature data of the target person;

[0009] The physiological characteristic information of the target person is determined according to the target facial feature data.

[0010] Optionally, determining target facial feature data of a target person based on the infrared image data and the RGB image data includes:

[0011] Inputting the infrared image data and the RGB image data into a target feature extraction model for extracting facial features by combining multiple image data;

[0012] In the target feature extraction model, a first facial image feature is extracted based on the infrared image data and a second facial image feature is extracted based on the RGB image data, and the first facial image feature and the second facial image feature are fused to output target facial feature data of the target person.

[0013] Optionally, determining the physiological characteristic information of the target person according to the target facial feature data includes:

[0014] Determine facial optical flow feature data according to the target face feature data;

[0015] The facial optical flow feature data is converted into frequency domain data using Fourier transform;

[0016] The physiological characteristic information of the target person is determined based on the peak signal in the frequency domain data.

[0017] Optionally, the physiological characteristic information includes heart rate data and heart rate variability, and determining the physiological characteristic information of the target person based on the peak signal in the frequency domain data includes:

[0018] Determine the heart rate data of the target person based on the peak signal in the frequency domain data;

[0019] A heart rate variability of the target person is determined based on the heart rate data.

[0020] Optionally, before the infrared image data and the RGB image data are input into a target feature extraction model for extracting facial features by combining a plurality of image data, the method further includes:

[0021] Preprocessing the infrared image data and the RGB image data;

[0022] Wherein, the preprocessing includes any one or more of the following:

[0023] Image scaling, random noise addition, random cropping, data enhancement, and normalization.

[0024] Optionally, it also includes:

[0025] Acquiring sample data for training the target feature extraction model;

[0026] Training the initial feature extraction model based on the sample data to obtain an iterative feature extraction model;

[0027] Using verification data selected from the sample data to perform integer calibration on the iterative feature extraction model to determine model accuracy data of the iterative feature extraction model;

[0028] When the model accuracy data does not meet the preset model accuracy condition, performing quantization training on the iterative feature extraction model;

[0029] When the model accuracy data meets the preset model accuracy condition, the iterative feature extraction model is used as the target feature extraction model.

[0030] Optionally, when the model accuracy data does not meet a preset model accuracy condition, performing quantization training on the iterative feature extraction model includes:

[0031] When the model accuracy data does not meet the preset model accuracy condition, inputting the sample data into the iterative feature extraction model;

[0032] When the sample data is used to perform training in the iterative feature extraction model, first floating-point data generated by calculation of a current training layer of the iterative feature extraction model is converted into preset integer data;

[0033] The preset shaping data is converted into second floating point data, and the second floating point data is input into the next training layer of the iterative feature extraction model for training.

[0034] Optionally, before converting the facial optical flow feature data into frequency domain data by Fourier transform, the method further includes:

[0035] Noise filtering is performed on the facial optical flow feature data.

[0036] A device for monitoring physiological characteristics of a person in a vehicle, the device comprising:

[0037] An image data acquisition module is used to acquire infrared image data and RGB image data collected for a target person in a target vehicle;

[0038] An image feature extraction module, used to determine target facial feature data of a target person based on the infrared image data and the RGB image data;

[0039] The physiological characteristic information determination module is used to determine the physiological characteristic information of the target person based on the target facial feature data.

[0040] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the above-mentioned method for monitoring physiological characteristics of people in a vehicle.

[0041] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for monitoring the physiological characteristics of a person in a vehicle as described above is implemented.

[0042] The embodiments of the present invention have the following advantages:

[0043] In an embodiment of the present invention, infrared image data and RGB image data collected for a target person in a target vehicle are obtained; the infrared image data and RGB image data are input into a target feature extraction model for extracting facial features by combining multiple image data; in the target feature extraction model, a first facial image feature is extracted based on the infrared image data and a second facial image feature is extracted based on the RGB image data, and the first facial image feature and the second facial image feature are fused, and the target facial feature data of the target person is output, and the physiological feature information of the target person is determined based on the target facial feature data. Thus, the physiological feature information can be detected based on multimodal image data through infrared image data and RGB image data, the accuracy and reliability of the detection are improved, and accurate physiological feature information is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0045] Figure 1a is a flowchart of the steps of a method for monitoring physiological characteristics of a person in a vehicle provided by an embodiment of the present invention;

[0046] Figure 1b It is a schematic diagram of a target feature extraction model structure provided by an embodiment of the present invention;

[0047] Figure 1c is a schematic diagram of a ShuffleNet V2 Block module provided by an embodiment of the present invention;

[0048] Figure 2 is a flowchart of another method for monitoring physiological characteristics of a person in a vehicle provided by an embodiment of the present invention;

[0049] Figure 3 is a flowchart of another method for monitoring physiological characteristics of a person in a vehicle provided by an embodiment of the present invention;

[0050] Figure 4 It is a structural schematic diagram of a device for monitoring physiological characteristics of a person in a vehicle provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] In recent years, with the rapid development of artificial intelligence and computer vision technology, the detection technology of driver physiological health characteristic signals has played an important role in improving road safety and driver welfare. The primary goal of the detection technology of driver physiological health characteristic signals is to improve driving safety. By monitoring the driver's physiological health characteristic signals (such as heart rate, heart rate variability, blood oxygen saturation, etc.), the system can identify the driver's health status, fatigue state, tension and other emergencies. When an abnormal situation is detected, the system can remind the driver to pay attention through alarm sounds, seat vibrations or visual prompts, and even take over by intelligent driving to a safe place or a nearby hospital when necessary, thereby reducing accidents caused by driver health problems and thus reducing risks. In order to achieve this goal, efficient and accurate physiological health characteristic signal detection technology has become the key. Especially in the vehicle environment, limited by computing resources and real-time requirements, a physiological health characteristic signal detection method that can still maintain high accuracy and high efficiency under limited hardware conditions is needed.

[0053] At present, the detection technology for physiological health characteristic signals is mainly based on sensors, such as heart rate sensors PPG, ECG, blood oxygen SPO2, and visual solutions based on face detection and optical flow combined detection algorithms, etc. These methods, especially the sensor method, have good accuracy, while the image vision-based method is still in its early stages.

[0054] In the actual implementation of the cockpit physiological health characteristic signal detection project, there are some challenges when deploying the image vision method to the vehicle side. The single-modality RGB image visual detection algorithm is greatly affected by lighting, angle, background interference and other factors in face detection. These factors can lead to a decline in detection performance, especially when these factors are serious, the performance decline is obvious and it is impossible to accurately detect physiological health characteristic signals in the future. In addition, when calculating the optical flow features of the face based on the optical flow algorithm, the amount of calculation is very huge. The hardware equipment and power consumption of the vehicle system are higher, so in practical applications, it is necessary to balance the equipment resources and the optimization of the detection algorithm. In summary, how to better maintain the real-time and high accuracy of vision-based physiological health characteristic signal detection, and thus truly improve the cockpit driving experience and safety, has become a problem that needs to be solved urgently.

[0055] The main driving force for the development of the vehicle industry has gradually shifted from being driven by products and technologies on the supply side to being driven by ever-increasing user demand. Users' perception of vehicles has also gradually shifted from a single means of transportation to a living space. The cockpit is a space that users directly experience. Improving the intelligence of the cockpit and making it a multi-functional experience area can provide users with a better vehicle riding experience. In an embodiment of the present invention, in the cockpit, infrared image data and RGB image data collected for a target person in a target vehicle can be acquired, and a target feature extraction model that combines multiple image data for facial feature extraction can be used to fuse the features of the infrared image data and the RGB image data. Then, the physiological feature information of the target person can be determined based on the fused target facial feature data, thereby realizing the detection of the physiological feature information of the occupant based on multimodal image data, and improving the accuracy and reliability of physiological feature detection.

[0056] Reference Figure 1a , shows a flowchart of a method for monitoring physiological characteristics of a person in a vehicle provided by an embodiment of the present invention, which is applied in the field of smart cockpits and may specifically include the following steps:

[0057] Step S101, acquiring infrared image data and RGB image data collected for a target person in a target vehicle;

[0058] In practical applications, an image acquisition device may be installed in the target vehicle, and the image acquisition device is used to acquire image data of the target person. For example, the image acquisition device may be an infrared detection device for acquiring infrared image data, or the image acquisition device may be an in-vehicle camera for acquiring RGB image data. The target person may include any one or more of the driver and passengers in the vehicle.

[0059] Step S102, inputting the infrared image data and the RGB image data into a target feature extraction model for extracting facial features by combining multiple image data;

[0060] After the infrared image data and the RGB image data are acquired, feature extraction may be performed in combination with the infrared image data and the RGB image data to determine target facial feature data of the target person.

[0061] In practical applications, RGB images require a good lighting environment to form images normally, otherwise the image quality is very poor, which can easily cause overexposure or underexposure of the image, resulting in the inability to clearly see the image content, and thus the inability to stably and accurately detect facial data. In an embodiment of the present invention, the defects of a single RGB image can be overcome by combining infrared image data and RGB image data for feature extraction, thereby obtaining accurate facial feature data.

[0062] Specifically, a target feature extraction model for extracting facial features by combining multiple image data can be pre-trained, wherein the input data of the target feature extraction model is infrared image data and RGB image data. The output data is the target facial feature data of the target person. The target feature extraction model is a model that can be used for feature extraction and feature fusion of image data.

[0063] The acquired infrared image data and RGB image data are then input into the target feature extraction model for feature extraction.

[0064] In one embodiment of the present invention, the infrared image data and the RGB image data may be preprocessed before being input into the target feature extraction model. The image data preprocessing facilitates subsequent feature extraction and fusion of the image data.

[0065] Among them, preprocessing includes any one or more of the following: image scaling adjustment processing, random noise addition processing, random cropping processing, data enhancement processing, and normalization processing.

[0066] In practical applications, the generalization ability and robustness of the model can be significantly improved by scaling the images, adding random noise, random cropping, data augmentation and normalization.

[0067] Specifically, when the preprocessing method is to scale the image, the image processing tool can be used to scale the image to a specified size. Common scaling methods include bilinear interpolation, bicubic interpolation, etc. When the preprocessing method is to add random noise processing, random noise, such as Gaussian noise, salt and pepper noise, etc., can be added to the image. The intensity and type of noise can be adjusted according to specific needs. When the preprocessing method is random cropping, a region can be randomly selected from the image for cropping, and the size and position of the cropped region can be randomly changed. When the preprocessing method is data enhancement processing, the data enhancement methods that can be used include but are not limited to random rotation, random flipping, random brightness adjustment, random contrast adjustment, etc. These transformations can be randomly applied to the image during training. When the preprocessing method is normalization processing, common normalization methods can include scaling pixel values ​​to a pre-value range, for example, a [0,1] range or a [-1,1] range. Normalization can be achieved by subtracting the mean and dividing by the standard deviation.

[0068] Step S103, in the target feature extraction model, extract the first facial image feature based on the infrared image data and extract the second facial image feature based on the RGB image data, and fuse the first facial image feature and the second facial image feature to output the target facial feature data of the target person.

[0069] In one example, the target feature extraction model may be a model composed of ShuffleNet V2 Block modules, and its model structure may be as follows: Figure 1b As shown in the figure, the input data is 224*224*6 RGB image data and infrared image data. After convolution processing of multiple ShuffleNet V2 Blocks, infrared features and RGB features are obtained, and then the infrared features are fused with the RGB features to obtain multi-mode fusion features, and then the multi-mode fusion features are output. The specific output can be divided into three categories. The first category is the probability of having a face, the second category is the position of the face frame, and the third category is the position of the key points of the face.

[0070] Among them, the schematic diagram of the ShuffleNet V2 Block module is as follows Figure 1c As shown in the figure, the channel splitting of ShuffleNet V2. Based on the block of ShuffleNet v1, ShuffleNet v2 block introduces a simple operator called ChannelSplit to achieve the purpose of not increasing the amount of FLOPs calculation. At the beginning of each unit (block), the c channels of the input feature map can be split into two branches (branch): cc^ channels. According to G3, the network fragmentation is as small as possible, one of the branches remains unchanged (shortcut connection), and the other branch contains three convolutions with the same number of channels to satisfy G1. Unlike ShuffleNet v1, the two convolutions of ShuffleNet v2 block no longer use grouped convolutions, partly because it satisfies G2, and partly because the initial channel splitting (split) operation has completed the grouping effect.

[0071] Finally, the results of the two branches are concatenated, so that the number of input and output channels of the convolution block is the same, which conforms to the G1 principle. Like ShuffleNet v1, the channel shuffle operation is used to ensure that the information of the two branches interacts, so as to fully extract the feature information of the input image. This not only improves the speed of feature extraction, but also ensures that the extracted features retain rich texture features and will not cause the problem of feature information loss.

[0072] In one embodiment of the present invention, the training process of the target feature extraction model includes the following steps:

[0073] Step S11, obtaining sample data for training a target feature extraction model;

[0074] In practical applications, the sample data may include infrared image data and RGB image data collected from the target person, as well as corresponding facial feature data.

[0075] Step S12, training the initial feature extraction model based on the sample data to obtain an iterative feature extraction model;

[0076] Step S13, using the verification data selected from the sample data to perform integer calibration on the iterative feature extraction model to determine the model accuracy data of the iterative feature extraction model;

[0077] Step S14: when the model accuracy data does not meet the preset model accuracy condition, quantization training is performed on the iterative feature extraction model;

[0078] The model accuracy condition can be set according to the actual scenario requirements. For example, when the model accuracy is less than the preset accuracy value, it is determined that the model accuracy data does not meet the preset model accuracy condition. The model accuracy can be the accuracy rate of the output data obtained during model verification.

[0079] Step S15, when the model accuracy data meets the preset model accuracy condition, the iterative feature extraction model is used as the target feature extraction model.

[0080] Step S104, determining the physiological characteristic information of the target person according to the target facial feature data.

[0081] After obtaining the target facial feature data, the target facial feature data may reflect the physiological feature information of the passenger or driver, wherein the physiological feature information may include heart rate, heart rate variability, and the like.

[0082] In an embodiment of the present invention, in order to solve the high latency limitation of the existing visual face detection model combined with optical flow feature extraction and analysis, a low-memory high-speed inference mechanism of model quantization is introduced, and then the general FPN (feature pyramid) multi-scale feature fusion is abandoned, and ShuffleNet is used to extract features, which significantly improves the detection capability of the model and the processing speed of detection.

[0083] In one embodiment of the present invention, when the model accuracy data does not meet the preset model accuracy conditions, the iterative feature extraction model is quantized and trained. Quantization training is a technique used in deep learning to reduce model size and accelerate reasoning. By converting the floating-point weights and activation values ​​in the model into lower-precision integer representations (such as 8-bit integers), memory usage and computational complexity are reduced. The specific quantization training process is as follows:

[0084] Step S21, when the model accuracy data does not meet the preset model accuracy condition, the sample data is input into the iterative feature extraction model;

[0085] In practical applications, when the model accuracy data does not meet the preset model accuracy conditions, in order to improve the model accuracy, the model training can be continued by using quantization training.

[0086] During the quantization training process, continue to use the previous sample data as the initial input data for training.

[0087] Step S22, when the sample data is used to perform training in the iterative feature extraction model, converting the first floating point data generated by the current training layer of the iterative feature extraction model into preset integer data;

[0088] After the sample data is input into the iterative feature extraction model, calculations can be performed in different training layers in the iterative extraction model for the input sample data.

[0089] In the previous iterative training process of the model, sample data can be input into the first training layer, and after corresponding calculation processing is performed in the first training layer, the data is output, and the output data is input into the second training layer for corresponding cloud processing. In this way, after multiple rounds of processing in the iterative extraction model, the final fused feature data is obtained.

[0090] During the quantization process in the embodiment of the present invention, the data input and output of each training layer can be quantized according to preset quantization rules to improve the model training accuracy.

[0091] Among them, the quantization rule can be a specific conversion rule between floating-point data and integer data output by the model, for example, converting between float32 data and int8 data to achieve quantization training.

[0092] The iterative extraction model may include multiple training layers, each training layer input and output is floating-point data, and the quantization process is: the first floating-point data output by each training layer is converted into preset integer data, and the data quantization processing is achieved by converting the floating-point data into integer data.

[0093] Step S23, converting the preset shaping data into second floating point data, and inputting the second floating point data into the next training layer of the iterative feature extraction model for training.

[0094] Since the input data of the iterative feature extraction model is floating-point data, the quantized integer data can be converted into second floating-point data through dequantization, and then the second floating-point data is input into the next training layer of the iterative feature extraction model to continue training.

[0095] According to the above quantization and inverse quantization processing, the forward propagation process of the iterative feature extraction model is completed to obtain the newly predicted fusion feature data, and then the model parameter iteration of the iterative feature extraction model is realized through back propagation, and the verification data is used to verify the iterative model, the model accuracy is calculated, and steps S14 to S15 are executed until the accuracy of the generated iterative feature extraction model meets the preset model accuracy condition, and the iterative feature extraction model is used as the target feature extraction model.

[0096] In an embodiment of the present invention, infrared image data and RGB image data collected for a target person in a target vehicle are obtained; the infrared image data and the RGB image data are input into a target feature extraction model for extracting facial features by combining multiple image data; in the target feature extraction model, a first facial image feature is extracted based on the infrared image data and a second facial image feature is extracted based on the RGB image data, and the first facial image feature and the second facial image feature are fused, and target facial feature data of the target person is output, and physiological feature information of the target person is determined based on the target facial feature data, thereby realizing feature extraction through a trained target feature extraction model, improving the accuracy of feature extraction, and obtaining accurate physiological feature information.

[0097] Reference Figure 2 , shows a flowchart of another method for monitoring physiological characteristics of a person in a vehicle provided by an embodiment of the present invention, which may specifically include the following steps:

[0098] Step S201, acquiring infrared image data and RGB image data collected for a target person in a target vehicle;

[0099] Step S202, inputting the infrared image data and the RGB image data into a target feature extraction model for extracting facial features by combining multiple image data;

[0100] Step S203, in the target feature extraction model, extracting a first facial image feature based on the infrared image data and extracting a second facial image feature based on the RGB image data, and fusing the first facial image feature with the second facial image feature to output target facial feature data of the target person;

[0101] Step S204, determining facial optical flow feature data according to the target face feature data;

[0102] After acquiring the facial feature data, the LK (Lucas-Kanade Optical Flow, optical flow estimation algorithm) optical flow technology can be used to calculate the facial optical flow features.

[0103] In practical applications, optical flow refers to the movement of objects between consecutive frames in an image sequence. It estimates the motion vector field of pixels based on the change in image intensity over time. The core idea of ​​optical flow calculation is that the movement of objects in an image over a short period of time can be represented as the change in image intensity. Its basic assumption is that the brightness of the image remains constant over a short period of time. This assumption can be expressed as:

[0104] I(x,y,t)=I(x+δx,y+δy,t+δt)

[0105] Among them, I(x, y, t) is the pixel intensity at position (x, y) at time t. (x+δx, y+δy, t+δt) is the new position of the pixel at position (x, y) at time t+δt. Based on this basic principle, the present invention adopts the Lucas-Kanade optical flow method and uses a (3×3) window to solve the optical flow equations of all pixels. The window-based method is simple to calculate and is suitable for processing local images such as face areas, thereby better matching the real-time performance requirements of the vehicle-mounted device.

[0106] Step S205, converting the facial optical flow feature data into frequency domain data using Fourier transform;

[0107] Among them, Fourier transform can be used to convert time domain signals into frequency domain signals.

[0108] In one embodiment of the present invention, before converting the facial optical flow feature data into frequency domain data using Fourier transform, the method further includes: performing noise filtering on the facial optical flow feature data.

[0109] In practical applications, bandpass filtering can be used to filter the facial optical flow feature data into time series numbers, retain the preset frequency components, remove noise, and further improve the accuracy of physiological feature detection.

[0110] Step S206: determining the physiological characteristic information of the target person based on the peak signal in the frequency domain data.

[0111] In one embodiment of the present invention, the physiological characteristic information includes heart rate data and heart rate variability. The physiological characteristic information of the target person is determined based on the peak signal in the frequency domain data, including: determining the heart rate data of the target person based on the peak signal in the frequency domain data; determining the heart rate variability of the target person based on the heart rate data.

[0112] In the embodiment of the present invention, the face detection module based on the improved RetinaFace (a face detection algorithm) multi-modal image data fusion combined with the architecture of the optical flow algorithm is intended to provide efficient detection function of the driver's physiological health characteristic signal for the equipment in the real vehicle cockpit, which is particularly beneficial for the vehicle-mounted OMS or DMS system that needs to operate in a resource-constrained environment. Through the steps of feature extraction, multi-modal image data fusion and feature enhancement extraction, model quantization, etc., the method can realize efficient physiological health characteristic signal detection on the embedded end, and improve the accuracy and reliability of detection.

[0113] In addition, the embodiment of the present invention takes into account the limitations of the skin color material of the sensor, and adopts a visual solution to introduce a multi-mode feature detection and extraction solution of infrared images and RGB images, thereby significantly solving the problem of being unable to normally detect physiological health characteristic signals due to skin color problems.

[0114] At the same time, the model involved in the embodiment of the present invention performs well on a variety of hardware platforms, including mobile CPUs, NPUs, GPUs, etc., which means that the method for monitoring the physiological characteristics of occupants in a vehicle in the embodiment of the present invention can be flexibly applied to different types of devices and systems.

[0115] The present invention combines a variety of structures, has versatility and flexibility, can adapt to different scenarios and needs, and brings a new solution to the field of cockpit pilot physiological health characteristic signal detection.

[0116] In an embodiment of the present invention, infrared image data and RGB image data collected for a target person in a target vehicle are obtained; target facial feature data of the target person is determined based on the infrared image data and the RGB image data; facial optical flow feature data is determined based on the target facial feature data; the facial optical flow feature data is converted into frequency domain data using Fourier transform; physiological feature information of the target person is determined based on a peak signal in the frequency domain data, thereby performing feature extraction based on the dual-modal image data to improve the accuracy of the physiological feature information.

[0117] Reference Figure 3 , shows a schematic diagram of a physiological characteristic monitoring process in an embodiment of the present invention, and the process can be divided into two stages: Stage 1 (Stage 1) and Stage 2 (Stage 2).

[0118] Among them, stage 1 includes the following processes:

[0119] (1) Input data: The input object of this method is the infrared and RBG cockpit face data collected by the vehicle-mounted OMS or DMS camera.

[0120] (2) Data processing: The two modal RGB and infrared images are scaled and adjusted in equal proportions. Random noise is added to the RGB image, and random cropping is performed for data enhancement. Finally, the two modal RGB and infrared images are normalized.

[0121] (3) Feature extraction: The processed image is input into ShuffleNetV2 0.5x for feature extraction.

[0122] (4) Feature fusion: The features after fusion of infrared image and RBG image are input into SSH module for enhanced feature extraction.

[0123] (5) Feature output: The model has three output task heads, one corresponding to the key point position of the face target, another corresponding to the upper left and lower right coordinates of the face target box, and the third corresponding to the probability value of whether there is a face in the target box.

[0124] (6) Model quantization: The model obtained in the steps of the entire stage 1 is quantized and trained to int8 to reduce memory consumption and increase the speed of the entire feature extraction output.

[0125] The specific quantization training process is as follows: When a deep learning model is obtained, it is generally saved in Float32 format, that is, the model is calculated in floating point type. Based on this model, the face dataset can be used for int8 data calibration, and finally the model float32 type is changed to int8 type, which is the offline fine-tuning method of model quantization.

[0126] After data calibration, if the accuracy of the model does not meet our goal, we need to perform quantization-aware training (Quant Aware Transform) to quantize the model. Quantization-aware training takes the face dataset as input and normally inputs it into our original float32 model. During the model calculation process, the output result data is quantized to int8, and then dequantized to restore the int8 data to float32 as the next layer of model calculation input, and this cycle continues until the last layer of the model output. Quantization-aware training repeats this process on the face dataset until the accuracy meets the goal, and the final output is the quantized model.

[0127] Stage 2 includes the following processes:

[0128] (1) LK optical flow: Calculate facial optical flow features based on the face area data output by the feature.

[0129] (2) Signal processing: Use bandpass filtering to filter the facial optical flow features into time series numbers, retain the frequency components from 0.7 to 4 Hz, and remove noise.

[0130] (3) Frequency domain analysis: Use FFT (Fast Fourier Transform) to convert the time series number into the frequency domain and find the frequency peak. This frequency peak is the heart rate for calculating physiological health characteristics. The heart rate variability (HRV Heart Rate Variability) can be further calculated from the heart rate, thereby obtaining a series of physiological health characteristics information.

[0131] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0132] Reference Figure 4 , shows a schematic diagram of the structure of a device for monitoring physiological characteristics of a person in a vehicle provided by an embodiment of the present invention, which may specifically include the following modules:

[0133] The image data acquisition module 401 is used to acquire infrared image data and RGB image data collected for a target person in a target vehicle;

[0134] A model input module 402, used for inputting the infrared image data and the RGB image data into a target feature extraction model for extracting facial features by combining multiple image data;

[0135] The model processing submodule 403 is used to extract the first facial image feature based on the infrared image data and the second facial image feature based on the RGB image data in the target feature extraction model, and fuse the first facial image feature with the second facial image feature to output the target facial feature data of the target person.

[0136] The physiological characteristic information determination module 404 is used to determine the physiological characteristic information of the target person according to the target facial feature data.

[0137] In an embodiment of the present invention, the physiological characteristic information determination module 404 may include the following submodules:

[0138] A facial optical flow feature data determination submodule, used to determine facial optical flow feature data according to the target face feature data;

[0139] A Fourier transform submodule, used for converting the facial optical flow feature data into frequency domain data by using Fourier transform;

[0140] The physiological characteristic information determination submodule is used to determine the physiological characteristic information of the target person based on the peak signal in the frequency domain data.

[0141] In one embodiment of the present invention, the physiological characteristic information includes heart rate data and heart rate variability, and the physiological characteristic information determination submodule may include the following units:

[0142] a heart rate data determining unit, configured to determine the heart rate data of a target person based on a peak signal in the frequency domain data;

[0143] A heart rate variability determination unit is used to determine the heart rate variability of a target person based on the heart rate data.

[0144] In one embodiment of the present invention, the device may further include:

[0145] A preprocessing module, used for preprocessing the infrared image data and the RGB image data;

[0146] Wherein, the preprocessing includes any one or more of the following:

[0147] Image scaling, random noise addition, random cropping, data enhancement, and normalization.

[0148] In one embodiment of the present invention, the device further comprises:

[0149] A sample data acquisition module, used to acquire sample data for training the target feature extraction model;

[0150] A model training module, used for training the initial feature extraction model based on the sample data to obtain an iterative feature extraction model;

[0151] A model calibration module, used to perform integer calibration on the iterative feature extraction model using verification data selected from the sample data to determine model accuracy data of the iterative feature extraction model;

[0152] A quantization training module, used for performing quantization training on the iterative feature extraction model when the model accuracy data does not meet the preset model accuracy condition;

[0153] The target feature extraction model determination module is used to use the iterative feature extraction model as the target feature extraction model when the model accuracy data meets the preset model accuracy condition.

[0154] In one embodiment of the present invention, the quantization training module may include:

[0155] An iterative training submodule, used for inputting the sample data into the iterative feature extraction model when the model accuracy data does not meet the preset model accuracy condition;

[0156] An integer data conversion submodule, used for converting first floating-point data generated by calculation of a current training layer of the iterative feature extraction model into preset integer data when the sample data is used for training in the iterative feature extraction model;

[0157] The floating point data conversion submodule is used to convert the preset shaping data into second floating point data, and input the second floating point data into the next training layer of the iterative feature extraction model for training.

[0158] In an embodiment of the present invention, the physiological characteristic information determination module 404 may further include:

[0159] The noise filtering submodule is used to perform noise filtering on the facial optical flow feature data.

[0160] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the above-mentioned method for monitoring the physiological characteristics of people in a vehicle is implemented.

[0161] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned method for monitoring the physiological characteristics of people in a vehicle is implemented.

[0162] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0163] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0164] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0165] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0166] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0168] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0169] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0170] The above is a detailed introduction to a method and device for monitoring the physiological characteristics of people in a vehicle, an electronic device, and a medium. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for monitoring physiological characteristics of people in a vehicle, characterized in that: The method comprises: Acquire infrared image data and RGB image data collected for a target person in a target vehicle; Inputting the infrared image data and the RGB image data into a target feature extraction model for extracting facial features by combining multiple image data; In the target feature extraction model, a first facial image feature is extracted based on the infrared image data and a second facial image feature is extracted based on the RGB image data, and the first facial image feature and the second facial image feature are fused to output target facial feature data of the target person; The physiological characteristic information of the target person is determined according to the target facial feature data.

2. The method according to claim 1, characterized in that Determining the physiological characteristic information of the target person according to the target facial feature data includes: Determine facial optical flow feature data according to the target face feature data; The facial optical flow feature data is converted into frequency domain data using Fourier transform; The physiological characteristic information of the target person is determined based on the peak signal in the frequency domain data.

3. The method according to claim 2, characterized in that The physiological characteristic information includes heart rate data and heart rate variability, and the determining of the physiological characteristic information of the target person based on the peak signal in the frequency domain data includes: Determine the heart rate data of the target person based on the peak signal in the frequency domain data; A heart rate variability of the target person is determined based on the heart rate data.

4. The method according to any one of claims 1 to 3, characterized in that Before the infrared image data and the RGB image data are input into a target feature extraction model for extracting facial features by combining multiple image data, the method further includes: Preprocessing the infrared image data and the RGB image data; Wherein, the preprocessing includes any one or more of the following: Image scaling, random noise addition, random cropping, data enhancement, and normalization.

5. The method according to any one of claims 1 to 3, characterized in that Also includes: Acquiring sample data for training the target feature extraction model; Training the initial feature extraction model based on the sample data to obtain an iterative feature extraction model; Using verification data selected from the sample data to perform integer calibration on the iterative feature extraction model to determine model accuracy data of the iterative feature extraction model; When the model accuracy data does not meet the preset model accuracy condition, performing quantization training on the iterative feature extraction model; When the model accuracy data meets the preset model accuracy condition, the iterative feature extraction model is used as the target feature extraction model.

6. The method according to any one of claims 5, characterized in that When the model accuracy data does not meet the preset model accuracy condition, quantizing the iterative feature extraction model, including: When the model accuracy data does not meet the preset model accuracy condition, inputting the sample data into the iterative feature extraction model; When the sample data is used to perform training in the iterative feature extraction model, first floating-point data generated by calculation of a current training layer of the iterative feature extraction model is converted into preset integer data; The preset shaping data is converted into second floating point data, and the second floating point data is input into the next training layer of the iterative feature extraction model for training.

7. The method according to claim 2, characterized in that Before converting the facial optical flow feature data into frequency domain data using Fourier transform, the method further includes: Noise filtering is performed on the facial optical flow feature data.

8. A device for monitoring physiological characteristics of people in a vehicle, characterized in that: The device comprises: An image data acquisition module is used to acquire infrared image data and RGB image data collected for a target person in a target vehicle; An image feature extraction module, used to determine target facial feature data of a target person based on the infrared image data and the RGB image data; The physiological characteristic information determination module is used to determine the physiological characteristic information of the target person based on the target facial feature data.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for monitoring the physiological characteristics of a person in a vehicle as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for monitoring the physiological characteristics of a person in a vehicle as described in any one of claims 1 to 7 is implemented.

Images