Passenger safety monitoring method and device for unmanned vehicle

Abstract

The invention provides a passenger safety monitoring method and device for an unmanned vehicle. The method comprises the steps that radar reflection signals of passengers are collected through a radar sensor deployed in the unmanned vehicle; feature engineering is carried out on the radar reflection signals, and multi-dimensional time sequence features of passengers are extracted; inputting the multi-dimensional time sequence characteristics into a bidirectional long-short-term memory network model, and obtaining bidirectional time sequence dependence characteristics of the passenger state; fusing the bidirectional time sequence dependency features based on a generalized ordered weighted logarithmic average fusion mechanism to obtain fusion features; and inputting the fusion feature into a full connection layer and a classifier to obtain a safety monitoring result of the passenger. According to the invention, accuracy and real-time performance of passenger safety state monitoring can be improved.

Description

Technical Field

[0001] This invention belongs to the field of unmanned driving safety monitoring technology, specifically relating to a method and device for monitoring passenger safety in unmanned vehicles. Background Technology

[0002] With the rapid development of autonomous driving technology, driverless taxis have gradually entered the commercial trial operation stage. However, in a fully driverless operating environment, the safety monitoring of passengers inside the vehicle has become a new technical challenge. In traditional manned vehicles, the driver can promptly detect and take emergency measures when a passenger experiences a sudden physical abnormality (such as heart attack, fainting, shock, etc.). However, driverless taxis lack such human monitoring mechanisms. If a passenger experiences a sudden situation during the trip, the vehicle system usually cannot detect it, which may lead to serious safety accidents and social risks.

[0003] On the other hand, while camera surveillance can achieve passenger status recognition to some extent, this method involves the collection and processing of sensitive information such as facial recognition and posture tracking, which can easily raise passenger privacy concerns and legal compliance issues. Given the increasingly stringent requirements for personal privacy protection in today's society, video-based in-vehicle monitoring solutions face significant obstacles in practical implementation.

[0004] Therefore, there is an urgent need for a new technology that can both enable real-time monitoring of passenger safety status and avoid privacy violations caused by video collection. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and device for monitoring passenger safety in unmanned vehicles, so as to achieve accurate monitoring of passenger safety status and timely triggering of early warning response while protecting passenger privacy.

[0006] In a first aspect, the present invention provides a method for monitoring passenger safety in an unmanned vehicle, the method comprising the following steps: Radar reflection signals from passengers are collected by radar sensors deployed inside the driverless vehicle; Feature engineering is performed on the radar reflection signal to extract multi-dimensional temporal features of passengers; the multi-dimensional temporal features include the passenger status at multiple times. Multi-dimensional temporal features are input into a bidirectional long short-term memory network model to obtain bidirectional temporal dependency features of passenger states; the bidirectional temporal dependency features characterize the temporal dependency relationship between the passenger state at each time step and the passenger states at previous and subsequent time steps. The bidirectional temporal dependency features are fused based on a generalized ordered weighted logarithmic average fusion mechanism to obtain fused features. The fusion mechanism includes sorting the bidirectional temporal dependency features by amplitude, logarithmic transformation to compress noise, and weighted aggregation based on weights to enhance the response to weak radar reflection signals from passengers. The weight values ​​are adaptively optimized by a cloud-based reinforcement learning agent based on feedback from the performance of abnormal state recognition and then sent to the vehicle. The fused features are input into a fully connected layer and a classifier to obtain passenger safety monitoring results; these results are used to trigger safety warning responses for autonomous vehicles.

[0007] Optionally, feature engineering can be performed on the radar reflection signal to extract multi-dimensional temporal features of passengers, including: Perform range fast Fourier transform and Doppler fast Fourier transform on the radar reflection signal to construct the first range Doppler image; Static clutter in the first range Doppler image is removed using the mean cancellation method to obtain the second range Doppler image; A constant false alarm rate (CFAR) detection algorithm is used to detect passenger reflection points from the second range Doppler image and generate a two-dimensional radar point cloud. Micro-Doppler signals are extracted from two-dimensional radar point clouds, and the passenger status at each moment is constructed based on the micro-Doppler signals; the passenger status includes respiratory rate, respiratory amplitude, heart rate, limb movement energy, center of gravity height, and duration of stillness; By combining passenger states from multiple moments in chronological order, multi-dimensional temporal features are obtained.

[0008] Optionally, the bidirectional temporal dependency features are fused based on a generalized ordered weighted log average fusion mechanism to obtain fused features, including: Bidirectional time-dependent features are sorted in descending order of magnitude. The elements in the set are sorted to obtain an ordered feature set. ; ; Represents an ordered set of features The first in One element, , Represents bidirectional temporal dependency features The total number of elements in the vector, where each element represents a bidirectional temporally dependent feature vector component; Using logarithmic transformation operators on ordered feature sets Each element in Compression processing is performed to obtain compressed bidirectional temporal dependency features. Among them, the logarithmic transformation operator compresses vibration noise based on the saturation characteristics of the logarithmic function to preserve the gradient of the weak radar reflection signal of the passenger; Normalized weight vectors generated by reinforcement learning agents Compressed bidirectional temporal dependency features Perform a dot product operation to obtain the fused features; among them, , Represents the weight vector The Each weight component , express The first in Each element.

[0009] Optionally, logarithmic transformation operators can be used on ordered feature sets. Each element in Compression processing is performed to obtain compressed bidirectional temporal dependency features. ,include: For ordered feature sets Each element in Through calculation formula

[0010] Obtain compressed bidirectional temporal dependency features .

[0011] Optionally, the cloud-based reinforcement learning agent adaptively optimizes itself based on feedback from the performance of abnormal state recognition, including: Construct a bias state evaluation function, taking the weight of the fusion mechanism at the current moment and the classification confusion matrix as the environment state; Define a weighted F1 incremental reward function, assigning a higher reward weight to F1 scores in abnormal states compared to normal states; the expression for the weighted F1 incremental reward function is:

[0012] in, The F1 score represents the abnormal state. The reward weighting for F1 scores representing abnormal states. This represents the reward weight in the normal state. , This indicates the accuracy of classification for the normal safety category; Based on the propensity state evaluation function and the weighted F1 incremental reward function, the weights of the fusion mechanism in the next time step are optimized using the policy gradient method.

[0013] Optionally, based on the propensity state evaluation function and the weighted F1 incremental reward function, the weights of the fusion mechanism in the next time step are optimized using the policy gradient method, including: Through calculation formula

[0014] Obtain the weights of the fusion mechanism in the next time step. ;in, This indicates the weight of the fusion mechanism at the current moment. Indicates the current environmental state Below, the agent selects weights The probability, The direction of the policy gradient indicates how to adjust the parameters to increase the probability of the current action. As an evaluation signal, it determines the magnitude and direction of the update, This represents the dynamically adjusted learning rate, ensuring that the model can update the weights more significantly when performance stagnates, thereby effectively escaping local optima.

[0015] Optional adaptive learning rates include: Real-time monitoring of the performance in identifying abnormal states; the performance is measured by calculating the change in the F1 score of abnormal states on the validation set. When a stagnation or degradation in recognition performance is detected, a learning rate expansion operation is performed, forcing the system to break out of the local convergence state by increasing the learning rate of reinforcement learning; at this time... ,in, This represents the gradient descent step size expansion value; When an improvement in recognition performance is detected, a learning rate decay operation is performed to reduce the learning rate in order to converge within the current effective feature space; at this time... ,in, This represents the gradient descent step size decay value.

[0016] Optionally, when inputting the fused features into the fully connected layer and classifier to obtain passenger safety monitoring results, the method introduces a vibration rejection mechanism based on vehicle chassis dynamics. This vibration rejection mechanism includes: Real-time reception of vertical axis acceleration collected by the vehicle's inertial measurement unit (IMU). and pitch angular velocity ; Through calculation formula Obtaining impact energy from the chassis ;in, Represents gravitational acceleration. Indicates the weighting coefficient; When the calculated chassis impact energy When the interference threshold is exceeded, the decision window freeze logic is triggered, suspending passenger safety monitoring at the current moment until the chassis impact energy is released. Less than or equal to the interference threshold.

[0017] Optional, the vibration veto mechanism also includes: When the vertical axis acceleration When the collision threshold is exceeded, it is determined to be a car accident scenario, and a safety warning response is triggered directly.

[0018] Secondly, the present invention provides a passenger safety monitoring device for unmanned vehicles, comprising: The signal acquisition module is used to collect radar reflection signals from passengers through radar sensors deployed in the autonomous vehicle; The feature engineering module is used to perform feature engineering on radar reflection signals and extract multi-dimensional temporal features of passengers; the multi-dimensional temporal features include the passenger status at multiple times. The feature processing module is used to input multi-dimensional temporal features into the bidirectional long short-term memory network model to obtain the bidirectional temporal dependency features of passenger state; the bidirectional temporal dependency features characterize the temporal dependency relationship between the passenger state at each time step and the passenger state at the previous and next time steps. The feature fusion module is used to fuse bidirectional temporal dependent features based on a generalized ordered weighted logarithmic average fusion mechanism to obtain fused features. The fusion mechanism includes sorting the bidirectional temporal dependent features by amplitude, logarithmic transformation to compress noise, and weighted aggregation based on weights to enhance the response to weak radar reflection signals from passengers. The weight values ​​are adaptively optimized by the cloud-based reinforcement learning agent based on feedback from the performance of abnormal state recognition and then sent to the vehicle. The safety monitoring module is used to input fused features into the fully connected layer and classifier to obtain passenger safety monitoring results; the safety monitoring results are used to trigger safety warning responses for autonomous vehicles.

[0019] The present invention has at least the following beneficial effects: By collecting passenger radar reflection signals using non-contact radar sensors, the privacy issues associated with video surveillance are avoided. Simultaneously, feature engineering is used to extract multi-dimensional temporal features, ensuring the physical meaning and interpretability of the data, thus improving the accuracy of passenger safety status monitoring. A bidirectional long short-term memory network model is employed to capture temporal dependencies, enhancing the accuracy of modeling dynamic passenger states. A generalized ordered weighted logarithmic average fusion mechanism is introduced, balancing the ratio of strong noise to weak signals through sorting, logarithmic transformation, and weight aggregation, improving the responsiveness to weak physiological signals and further enhancing the accuracy of passenger safety status monitoring. A cloud-based reinforcement learning agent adaptively optimizes weights based on performance feedback from abnormal states, resolving recognition bias caused by class imbalance and ensuring sensitive detection of abnormal states, thus improving the accuracy of passenger safety status monitoring. Finally, the safety monitoring results trigger early warning responses, achieving real-time and reliable passenger safety monitoring. Attached Figure Description

[0020] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0021] Figure 1 This is a schematic diagram of the edge-cloud collaborative architecture in the embodiments of this application; Figure 2 This is a flowchart of a method for monitoring passenger safety in an unmanned vehicle according to one embodiment of this application; Figure 3 This is a schematic diagram of the structure of an unmanned vehicle passenger safety monitoring device in one embodiment of this application. Detailed Implementation

[0022] The technical solution of the present invention will now be described in detail and completely with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0023] In the description of this invention, it should be noted that the terms "upper", "lower", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0024] To overcome the shortcomings of existing technologies, this invention provides a method and device for passenger safety monitoring in autonomous vehicles. This method, while fully protecting passenger privacy, utilizes innovative signal processing, feature fusion, model optimization, and anti-interference mechanisms to achieve continuous, accurate, and robust monitoring of passenger vital signs (breathing, heartbeat) and behavioral states (sitting, active, falling, unconscious, etc.). It also triggers tiered safety warnings upon detecting anomalies, thus filling the technological gap in automated passenger safety monitoring in autonomous driving scenarios.

[0025] Example 1 In this embodiment of the invention, the passenger safety monitoring method for autonomous vehicles provided by the present invention is used in an autonomous taxi scenario, which employs the following... Figure 1The diagram illustrates an edge-cloud collaborative architecture. The vehicle-mounted hardware primarily includes: 77GHz FMCW millimeter-wave radar arrays (at least three to achieve cabin coverage) deployed on the A-pillars, B-pillars, and roof of the vehicle; an onboard edge controller (such as NVIDIA Jetson AGX Orin) responsible for data aggregation and edge computing; an inertial measurement unit for acquiring vehicle motion data; and an onboard T-Box for communication. The cloud-based architecture includes a data lake, a model training server equipped with a GPU cluster, a model version management platform, and a remote emergency response center. The vehicle and cloud form two collaborative loops: Vehicle-side real-time monitoring loop: continuously performing "signal acquisition → preprocessing → model inference → local decision → immediate response".

[0026] Cloud-based asynchronous optimization loop: Collect anonymous data from the vehicle on a timed or event-driven basis, perform reinforcement learning training, generate optimized model weights, and securely distribute them to the vehicle to update the local model.

[0027] The following is combined Figure 2 The flowchart shown provides a detailed explanation of the passenger safety monitoring method for unmanned vehicles provided by this invention.

[0028] like Figure 2 As shown, the passenger safety monitoring method for unmanned vehicles provided by the present invention includes steps 21 to 25: Step 21: Collect the radar reflection signals of passengers using radar sensors deployed inside the driverless vehicle.

[0029] Specifically, multiple FMCW (Frequency Modulated Continuous Wave) millimeter-wave radar sensors are deployed inside the A, B, and C pillars of the driverless taxi, forming an array to cover the passenger area inside the vehicle. Each sensor transmits a linear frequency modulated signal at a center frequency of 77 GHz, a bandwidth of 4 GHz, and a frame rate of 100 ms. The antenna array receives reflected echoes caused by the passenger's subtle body movements (such as breathing, heartbeat, and limb movements). The reflected signals include Doppler frequency shift (caused by motion speed) and distance information (calculated from phase difference), which are physical quantities representing the radar reflection characteristics of the passenger's physiological and behavioral states.

[0030] In one feasible implementation, a precise hardware trigger signal can synchronize the sampling times of all radar sensors and simultaneously acquire IMU data (including vertical acceleration) from the vehicle's CAN bus. and pitch angular velocity This provides a time alignment basis for subsequent multi-sensor fusion.

[0031] It is worth mentioning that step 21 of this invention enables non-contact, non-visual signal acquisition, which reduces privacy risks compared to video surveillance; the physical characteristics of radar signals (such as the Doppler effect) directly reflect the vital signs of passengers, providing a reliable input data foundation and improving the robustness of subsequent processing.

[0032] Step 22: Perform feature engineering on the radar reflection signal to extract multi-dimensional temporal features of passengers.

[0033] In this embodiment of the invention, the multi-dimensional temporal features include passenger status at multiple times.

[0034] Specifically, step 22 includes steps 22.1 to 22.5.

[0035] Step 22.1: Perform range fast Fourier transform and Doppler fast Fourier transform on the radar reflection signal to construct the first range Doppler image.

[0036] Specifically, range fast Fourier transform (R-FFT) and Doppler fast Fourier transform (D-FFT) are performed on the original radar reflection signal to construct the first range-Doppler map (RDM), which is a two-dimensional matrix. The rows represent the range gate (resolution 0.0375m) and the columns represent the Doppler velocity gate (resolution 0.05m / s). The intensity of each pixel reflects the reflection energy of a target with a specific radial velocity at that location.

[0037] Step 22.2: Use the mean cancellation method to remove static clutter from the first range Doppler image to obtain the second range Doppler image.

[0038] Specifically, the average value of the RDM (Relative Distance Model) for the previous N frames (e.g., N=10) is calculated as a static background estimate, and then this background estimate is subtracted from the current RDM. This effectively eliminates strong reflection clutter generated by static objects such as seats and doors, while preserving the signals of dynamic targets such as passengers.

[0039] Step 22.3: The constant false alarm rate (CFAR) detection algorithm is used to detect passenger reflection points from the second range Doppler image and generate a two-dimensional radar point cloud.

[0040] Specifically, on the clutter-suppressed RDM (second range-Doppler map), a two-dimensional CFAR detection algorithm adaptively determines the energy threshold of each range-Doppler cell, detecting points exceeding the threshold as potential passenger reflection points. Combining the detection results from multiple radars and MIMO angle measurement technology, these points are mapped to a three-dimensional space in the vehicle coordinate system, forming a 3D radar point cloud for each frame.

[0041] Step 22.4: Extract micro-Doppler signals from the two-dimensional radar point cloud and construct the passenger status at each time step based on the micro-Doppler signals.

[0042] The passenger's status includes breathing rate, breathing amplitude, heart rate, limb movement energy, center of gravity height, and duration of stillness.

[0043] Specifically, spatial clustering based on point clouds (such as the DBSCAN algorithm) is used to separate the areas belonging to passengers. For all points within this area, the time-varying Doppler velocity sequence, i.e., the micro-Doppler signal, is extracted. This signal finely encodes vital activity information such as the rise and fall of the chest and abdomen during breathing and the subtle tremors of the limbs.

[0044] Step 22.5: Combine the passenger status at multiple times in chronological order to obtain multi-dimensional temporal features.

[0045] Specifically, in each processing cycle (e.g., 1 second), a set of k-dimensional feature vectors is calculated based on the current micro-Doppler signal and historical point cloud, for example: ;in, The respiratory rate is represented by the main peak frequency extracted from the range of 0.1Hz to 0.5Hz through spectral analysis of the micro-Doppler signal in the thoracic region. Indicates the amplitude of breathing, expressed as The spectral amplitude corresponding to the main peak frequency. The heart rate is represented by extracting spectral components within the range of 1.0Hz to 2.0Hz (which need to be separated from respiratory harmonics through signal processing). The energy representing limb movement is obtained by calculating the variance of the point cloud velocities belonging to all passengers. The height of the center of gravity is represented by the coordinates of the centroid of the passenger point cloud in the vertical direction. The duration of stillness is represented by counting how many consecutive frames have not detected limb motion energy exceeding a threshold.

[0046] The k-dimensional feature vectors from T consecutive time steps (e.g., T=100) are stacked in chronological order to form a T×k matrix. Each column of this matrix (i.e., each feature dimension) is independently Z-score standardized to have a mean of 0 and a standard deviation of 1, thus eliminating the influence of differences in feature dimensions on the model. This yields the multi-dimensional temporal features.

[0047] It is worth mentioning that step 22 of this invention, through a professional radar signal processing chain, refines the original electromagnetic wave reflection signal into a pure, structured, and normalized temporal feature matrix. This includes both the passenger's macroscopic posture (center of gravity, stationary) and subtle vital signs (breathing, heartbeat), providing high-quality input for subsequent deep learning models. Step 23: Input the multi-dimensional temporal features into the bidirectional long short-term memory network model to obtain the bidirectional temporal dependency features of passenger status.

[0048] In this embodiment of the invention, the bidirectional temporal dependency characteristic represents the temporal dependency relationship between the passenger state at each time step and the passenger states at previous and subsequent time steps.

[0049] Specifically, the standardized T×k matrix (multi-dimensional temporal features) obtained in step 22 is input into a two-layer bidirectional LSTM network. In one feasible implementation, the network includes an input layer (receiving 6-dimensional temporal features, sequence length 10), two Bi-LSTM layers (a first Bi-LSTM layer and a second Bi-LSTM layer, each with 64 units, using the tanh activation function, with a forget gate bias of 1.0 to retain long-term memory), two Dropout layers (randomly dropping neurons with a probability of 0.2 to prevent overfitting), and a fully connected layer (outputting 128-dimensional features). The forward LSTM learns dependencies from the beginning to the end of the sequence, and the backward LSTM learns from the end to the beginning. Backpropagation uses the Adam optimizer with a learning rate of 0.001 and a batch size of 32.

[0050] In this process, the first Bi-LSTM layer uses forward and backward LSTMs to read temporal features from the past and future directions, respectively, and outputs a hidden state sequence that incorporates bidirectional information. The second Bi-LSTM layer further abstracts the features and finally takes the bidirectional hidden state at the last time step, concatenates them to form a high-dimensional feature vector. , This high-dimensional feature vector It not only contains the passenger's current state information, but also encodes the contextual dependencies of state evolution over a previous period. For example, a feature of "sudden drop in center of gravity" might be a normal behavior of sitting down if it occurs after a period of "high limb movement energy," but if it occurs after a long period of stillness, the risk of fainting increases significantly. Bidirectional LSTM is designed to capture this complex contextual dependency.

[0051] Step 24: Based on the generalized ordered weighted log average fusion mechanism, the bidirectional temporal dependency features are fused to obtain the fused features.

[0052] In this embodiment of the invention, the fusion mechanism includes sorting the bidirectional temporal-dependent features by amplitude, compressing noise through logarithmic transformation, and performing weighted aggregation based on weights, to enhance the response to weak radar reflection signals from passengers. Furthermore, the weight values ​​are adaptively optimized by a cloud-based reinforcement learning agent based on feedback from abnormal state recognition performance and then distributed to the vehicle.

[0053] Specifically, step 24 includes steps 24.1 to 24.3.

[0054] Step 24.1: Process the bidirectional temporal dependency features in descending order of amplitude. The elements in the set are sorted to obtain an ordered feature set. .

[0055] in, . Represents an ordered set of features The first in One element, , Represents bidirectional temporal dependency features The total number of elements in the sequence represents the bidirectional time-dependent feature vector components. This operation separates high-energy noise features, which may be caused by severe turbulence, from low-energy signal features representing breathing and heartbeat at their sequence positions.

[0056] Step 24.2, apply the logarithmic transformation operator to the ordered feature set Each element in Compression processing is performed to obtain compressed bidirectional temporal dependency features. .

[0057] Among them, the logarithmic transformation operator compresses vibration noise based on the saturation characteristics of the logarithmic function to preserve the gradient of the weak radar reflection signal of the passenger.

[0058] Specifically, for ordered feature sets Each element in Through calculation formula

[0059] Obtain compressed bidirectional temporal dependency features .in The function retains the sign (positive / negative) of the original characteristics. It is the natural logarithm. The "saturation property" of the logarithmic function is key: for large values... (noise), Growth was much slower than This resulted in a strong compression effect; for small amplitude values... (life signal), Approximate to The gradient is well preserved. This achieves dynamic range compression at the feature level, preventing the subsequent weighted summation from being dominated by a few high-energy noise points.

[0060] Step 24.3: Utilize the normalized weight vector generated by the reinforcement learning agent. Compressed bidirectional temporal dependency features Perform a dot product operation to obtain the fused features.

[0061] in, , Represents the weight vector The Each weight component , express The first in Each element.

[0062] Specifically, through the calculation formula

[0063] Obtain fusion features The goal of cloud-based reinforcement learning is precisely to learn a weight distribution. , making It is most sensitive to abnormal states. For example, it may learn to assign higher weights to feature dimensions that, after logarithmic compression, are significantly different from normal samples in syncope samples. .

[0064] It should be noted that existing methods for addressing the technical problem of "weak vital signals being submerged by strong noise in dynamic vehicle environments" typically employ frequency domain filtering (such as bandpass filters to separate breathing bands) or fixed threshold noise reduction. However, during vehicle operation, the noise spectrum generated by road bumps overlaps with vital signals and has an energy several orders of magnitude higher, limiting the effectiveness of traditional filtering methods. This invention creatively proposes a generalized ordered weighted logarithmic average fusion mechanism. The difference lies in the fact that existing feature fusion methods mostly employ average pooling, max pooling, or attention mechanisms, none of which are specifically designed for the physical characteristic of "extracting weak signals from high dynamic range signals." This invention separates noise and signal in the feature sequence by sorting by amplitude; it utilizes the saturation property of the logarithmic function to strongly compress the sorted high-amplitude components (noise) while linearly preserving the gradient of low-amplitude components (vital signals), thus achieving dynamic range normalization in the feature domain; and it introduces learnable weights for aggregation. This series of operations is based on a dedicated mathematical transformation designed with a deep understanding of the physical characteristics of radar signals, and is not a conventional approach in the field of deep learning. Its effect is to significantly improve the signal-to-noise ratio of the system when the vehicle is bumpy, so that weak vital signs can be highlighted at the feature level, thereby achieving high-sensitivity detection of abnormal states in harsh noise environments.

[0065] Specifically, the cloud-based reinforcement learning agent adaptively optimizes itself based on feedback from the performance of identifying abnormal states, including steps 24.3.1 to 24.3.3.

[0066] Step 24.3.1: Construct a bias state evaluation function, taking the weights of the fusion mechanism at the current moment and the classification confusion matrix as the environment state; Step 24.3.2, define the weighted F1 incremental reward function, setting a higher reward weight for the F1 score in abnormal states than for normal states; the expression of the weighted F1 incremental reward function is:

[0067] in, The F1 score represents the abnormal state. The reward weighting for F1 scores representing abnormal states. This represents the reward weight in the normal state. , This represents the classification accuracy for the normal and safe category. This term is introduced to maximize the anomaly recall rate while constraining the model's false positive rate, preventing the model from losing its ability to identify ordinary daily behaviors due to excessive focus on rare anomalies.

[0068] Step 24.3.3: Based on the propensity state evaluation function and the weighted F1 incremental reward function, optimize the weights of the fusion mechanism in the next time step using the policy gradient method.

[0069] Specifically, through the calculation formula

[0070] Obtain the weights of the fusion mechanism in the next time step. ;in, This indicates the weight of the fusion mechanism at the current moment. Indicates the current environmental state Below, the agent selects weights The probability of . This formula represents the update of model parameters using the stochastic gradient ascent method. Where, The direction of the policy gradient indicates how to adjust the parameters to increase the probability of the current action; As an evaluation signal, it determines the magnitude and direction of the update (a positive reward is encouragement, and a negative reward is punishment); and This is the learning rate dynamically adjusted through the 'reverse exploration mechanism', ensuring that the model can update the weights at a greater rate when performance stagnates, thereby effectively escaping local optima.

[0071] It should be noted that, in this embodiment of the invention, the adaptive learning rate includes: Real-time monitoring of the performance in identifying abnormal states; the performance is measured by calculating the change in the F1 score of abnormal states on the validation set. When a stagnation or degradation in recognition performance is detected, a learning rate expansion operation is performed, forcing the system to break out of the local convergence state by increasing the learning rate of reinforcement learning; at this time... ,in, This represents the gradient descent step size expansion value; When an improvement in recognition performance is detected, a learning rate decay operation is performed to reduce the learning rate in order to converge within the current effective feature space; at this time... ,in, This represents the gradient descent step size decay value.

[0072] It should be noted that, regarding the technical problem of "high false negative rate due to extreme scarcity of outlier samples," existing machine learning methods often address class imbalance by resampling, cost-sensitive learning, or adjusting classification thresholds. However, these are static and passive adjustments that cannot adapt to online operations and may even compromise overall accuracy. This invention creatively introduces a reinforcement learning framework and designs a dedicated reward function and learning rate strategy focused on "improving outlier class recognition performance." The difference lies in: firstly, the reward function does not optimize the overall accuracy but rather applies a weighted F1 increment and assigns it to a small number of outlier classes. Far higher than ordinary categories First, it forcibly reverses the model's bias towards the majority class from the optimization objective. Second, it proposes a "reverse exploration" learning rate strategy, which actively increases the learning rate to escape local optima when the model's performance in identifying outliers stagnates, contradicting the common sense that the learning rate should only decrease in traditional training. Applying the "exploration-exploitation" idea of ​​reinforcement learning to solve the class imbalance problem in static datasets, and designing corresponding targeted rewards and aggressive exploration strategies, is not something easily conceived by those skilled in the art.

[0073] Step 25: Input the fused features into the fully connected layer and classifier to obtain the passenger safety monitoring results.

[0074] In this embodiment of the invention, the safety monitoring results are used to trigger a safety warning response for the autonomous vehicle.

[0075] Specifically, the fully connected layer (256 neurons, ReLU activation) and the Softmax classifier are fused together to output the probability distribution of passenger status. Based on the probability distribution results and preset thresholds, a graded response (L1 level, L2 level, L3 level) is triggered.

[0076] For example, Level 1 (Observation): If prolonged stillness or minor abnormalities are detected (probability 60% to 80%), the vehicle's voice system will ask, "Do you need help?".

[0077] Level L2 (Manual Intervention): If persistent shortness of breath or abnormal heart rate is detected (probability > 80%), the data packet is uploaded to the cloud, and a human customer service representative connects to the in-vehicle microphone / camera for confirmation.

[0078] Level 3 (Emergency Rescue): If vital signs are detected to be absent or a car accident is confirmed (probability > 95% or collision threshold is triggered), an emergency signal will be automatically sent to the emergency center, and the vehicle will be pulled over and the doors will be unlocked.

[0079] In another feasible implementation, when inputting the fused features into the fully connected layer and classifier to obtain the passenger safety monitoring results, this invention also introduces a vibration rejection mechanism based on vehicle chassis dynamics to address vehicle vibration interference. This vibration rejection mechanism includes: Step 1: Receive the vertical axis acceleration collected in real time by the vehicle's inertial measurement unit (IMU). and pitch angular velocity .

[0080] Step II, through calculation formula Obtaining impact energy from the chassis .

[0081] in, Represents gravitational acceleration. This represents the weighting coefficient.

[0082] Step III, when the calculated chassis impact energy When the interference threshold is exceeded, the decision window freeze logic is triggered, suspending passenger safety monitoring at the current moment until the chassis impact energy is released. Less than or equal to the interference threshold.

[0083] When the calculated chassis impact energy When the interference threshold is exceeded, it means that the vehicle is experiencing significant bumps.

[0084] Furthermore, when the vertical axis acceleration When the collision threshold (e.g., 5g) is exceeded, a Level 3 response is triggered directly. This means that a car accident has occurred.

[0085] It is worth mentioning that this mechanism can effectively distinguish between vehicle vibrations and actual passenger movements, significantly reducing false alarms caused by road condition interference.

[0086] Example 2 like Figure 3 As shown, in this embodiment of the invention, a passenger safety monitoring device for an unmanned vehicle is also provided. The device 300 includes: The signal acquisition module 301 is used to acquire the radar reflection signals of passengers through radar sensors deployed in the driverless vehicle; The feature engineering module 302 is used to perform feature engineering on the radar reflection signal and extract multi-dimensional temporal features of passengers; the multi-dimensional temporal features include the passenger status at multiple times. The feature processing module 303 is used to input multi-dimensional temporal features into the bidirectional long short-term memory network model to obtain the bidirectional temporal dependency features of passenger state; the bidirectional temporal dependency features characterize the temporal dependency relationship between the passenger state at each time step and the passenger state at the previous and next time steps. The feature fusion module 304 is used to fuse bidirectional temporal dependent features based on a generalized ordered weighted logarithmic average fusion mechanism to obtain fused features. The fusion mechanism includes sorting the bidirectional temporal dependent features by amplitude, logarithmic transformation to compress noise, and weighted aggregation based on weights to enhance the response to weak radar reflection signals from passengers. The weight values ​​are adaptively optimized by the cloud-based reinforcement learning agent based on feedback from the recognition performance of abnormal states and then sent to the vehicle. The safety monitoring module 305 is used to input the fused features into the fully connected layer and the classifier to obtain the passenger safety monitoring results; the safety monitoring results are used to trigger the safety warning response of the autonomous vehicle.

[0087] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. Their specific functions and technical effects can be found in the method embodiments section, and will not be repeated here. Those skilled in the art will understand that, for ease of description and brevity, the division of the above-mentioned functional units and modules is only used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0088] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0089] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

Claims

1. A method for monitoring passenger safety in an unmanned vehicle, characterized in that, include: Radar reflection signals from passengers are collected by radar sensors deployed inside the driverless vehicle; Feature engineering is performed on the radar reflection signal to extract multi-dimensional temporal features of the passenger; the multi-dimensional temporal features include the passenger's state at multiple times. The multi-dimensional temporal features are input into a bidirectional long short-term memory network model to obtain the bidirectional temporal dependency features of passenger states; the bidirectional temporal dependency features characterize the temporal dependency relationship between the passenger state at each time step and the passenger states at previous and subsequent time steps. The bidirectional temporal dependency features are fused using a generalized ordered weighted logarithmic average fusion mechanism to obtain fused features. The fusion mechanism includes sorting the bidirectional temporal dependency features by amplitude, performing logarithmic transformation to compress noise, and weighted aggregation based on weights to enhance the response to weak radar reflection signals from passengers. The weight values ​​are adaptively optimized by a cloud-based reinforcement learning agent based on feedback from the performance of abnormal state recognition and then sent to the vehicle. The fused features are input into a fully connected layer and a classifier to obtain the passenger's safety monitoring results; the safety monitoring results are used to trigger a safety warning response for the autonomous vehicle.

2. The method for monitoring passenger safety in unmanned vehicles according to claim 1, characterized in that, The step of performing feature engineering on the radar reflection signal to extract the multi-dimensional temporal features of the passenger includes: Perform range fast Fourier transform and Doppler fast Fourier transform on the radar reflection signal to construct a first range Doppler image; Static clutter in the first range Doppler image is removed using the mean cancellation method to obtain the second range Doppler image; A constant false alarm rate (CFAR) detection algorithm is used to detect passenger reflection points from the second range Doppler image and generate a two-dimensional radar point cloud. Micro-Doppler signals are extracted from the two-dimensional radar point cloud, and the passenger status at each moment is constructed based on the micro-Doppler signals; the passenger status includes respiratory rate, respiratory amplitude, heart rate, limb movement energy, center of gravity height, and duration of stillness; The passenger status at the multiple times is combined in chronological order to obtain the multi-dimensional temporal features.

3. The method for monitoring passenger safety in unmanned vehicles according to claim 2, characterized in that, The generalized ordered weighted logarithmic average fusion mechanism is used to fuse the bidirectional temporal dependency features to obtain fused features, including: The bidirectional temporal dependency features are ordered from largest to smallest amplitude. The elements in the set are sorted to obtain an ordered feature set. ; ; Represents an ordered set of features The first in One element, , This represents the bidirectional temporal dependency feature. The total number of elements in the middle, wherein the elements represent bidirectional time-dependent feature vector components; Using the logarithmic transformation operator on the ordered feature set Each element in Compression processing is performed to obtain compressed bidirectional temporal dependency features. The logarithmic transformation operator compresses vibration noise based on the saturation characteristics of the logarithmic function to preserve the gradient of the passenger's weak radar reflection signal. The normalized weight vector generated by the reinforcement learning agent Compressed bidirectional temporal dependency features Perform a dot product operation to obtain the fused features; wherein, , Represents the weight vector The Each weight component , express The first in Each element.

4. The method for monitoring passenger safety in unmanned vehicles according to claim 3, characterized in that, The logarithmic transformation operator is used on the ordered feature set Each element in Compression processing is performed to obtain compressed bidirectional temporal dependency features. ,include: For ordered feature sets Each element in Through calculation formula Obtain compressed bidirectional temporal dependency features .

5. The method for monitoring passenger safety in unmanned vehicles according to claim 4, characterized in that, The cloud-based reinforcement learning agent adaptively optimizes itself based on feedback from the performance of abnormal state recognition, including: Construct a biased state evaluation function, taking the weights of the fusion mechanism and the classification confusion matrix at the current moment as the environment state; Define a weighted F1 incremental reward function, assigning a higher reward weight to the F1 score in abnormal states compared to normal states; the expression for the weighted F1 incremental reward function is: in, The F1 score represents the abnormal state. The reward weighting for F1 scores representing abnormal states. This represents the reward weight in the normal state. , This indicates the accuracy of classification for the normal safety category; Based on the propensity state evaluation function and the weighted F1 incremental reward function, the weights of the fusion mechanism at the next time step are optimized using the policy gradient method.

6. The method for monitoring passenger safety in unmanned vehicles according to claim 5, characterized in that, The step of optimizing the weights of the fusion mechanism for the next time step using the policy gradient method based on the propensity state evaluation function and the weighted F1 incremental reward function includes: Through calculation formula Obtain the weights of the fusion mechanism in the next time step. ;in, This indicates the weight of the fusion mechanism at the current moment. Indicates the current environmental state Below, the agent selects weights The probability, The direction of the policy gradient indicates how to adjust the parameters to increase the probability of the current action. As an evaluation signal, it determines the magnitude and direction of the update. This represents the learning rate, ensuring that the model can update the weights more significantly when performance stagnates, thus effectively escaping local optima.

7. The method for monitoring passenger safety in unmanned vehicles according to claim 6, characterized in that, The adaptive learning rate includes: Real-time monitoring of the performance in identifying abnormal states; the performance in identifying abnormal states is measured by calculating the change in the F1 score of abnormal states on the validation set. When a stagnation or degradation in recognition performance is detected, a learning rate expansion operation is performed, forcing the system to break out of the local convergence state by increasing the learning rate of reinforcement learning; at this time... ,in, This represents the gradient descent step size expansion value; When the improved recognition performance is detected, a learning rate decay operation is performed to reduce the learning rate in order to converge within the current effective feature space; at this time... ,in, This represents the gradient descent step size decay value.

8. The method for monitoring passenger safety in an unmanned vehicle according to claim 7, characterized in that, The method, when inputting the fused features into a fully connected layer and a classifier to obtain the passenger's safety monitoring results, introduces a vibration rejection mechanism based on vehicle chassis dynamics. This vibration rejection mechanism includes: Real-time reception of vertical axis acceleration collected by the vehicle's inertial measurement unit (IMU). and pitch angular velocity ; Through calculation formula Obtaining impact energy from the chassis ;in, Represents gravitational acceleration. Indicates the weighting coefficient; When the calculated chassis impact energy When the interference threshold is exceeded, the decision window freeze logic is triggered, suspending passenger safety monitoring at the current moment until the chassis impact energy is reached. Less than or equal to the interference threshold.

9. The method for monitoring passenger safety in an unmanned vehicle according to claim 8, characterized in that, The vibration veto mechanism also includes: When the vertical axis acceleration When the collision threshold is exceeded, it is determined to be a car accident scenario, and the safety warning response is directly triggered.

10. A passenger safety monitoring device for an unmanned vehicle, characterized in that, include: The signal acquisition module is used to collect radar reflection signals from passengers through radar sensors deployed in the autonomous vehicle; The feature engineering module is used to perform feature engineering on the radar reflection signal and extract multi-dimensional temporal features of the passenger; the multi-dimensional temporal features include the passenger's state at multiple times. The feature processing module is used to input the multi-dimensional temporal features into the bidirectional long short-term memory network model to obtain the bidirectional temporal dependency features of the passenger state; the bidirectional temporal dependency features represent the temporal dependency relationship between the passenger state at each time step and the passenger state at the previous and next time steps. The feature fusion module is used to fuse the bidirectional temporal dependency features based on a generalized ordered weighted logarithmic average fusion mechanism to obtain fused features. The fusion mechanism includes sorting the bidirectional temporal dependency features by amplitude, performing logarithmic transformation to compress noise, and weighted aggregation based on weights to enhance the response to weak radar reflection signals from passengers. The values ​​of the weights are adaptively optimized by a cloud-based reinforcement learning agent based on feedback from the performance of abnormal state recognition and then sent to the vehicle. The safety monitoring module is used to input the fused features into a fully connected layer and a classifier to obtain the safety monitoring results of the passenger; the safety monitoring results are used to trigger a safety warning response for the autonomous vehicle.

Images