Passenger health monitoring method and system
By fusing visual and radar sensor data through a dynamic gating coding network and combining it with a medical information database, the problem of performance degradation of traditional methods in dynamic environments has been solved, achieving highly accurate and reliable passenger health monitoring.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-04
- Publication Date
- 2026-03-24
AI Technical Summary
Traditional passenger health monitoring methods degrade in dynamic, uncontrolled environments and lack clinical reasoning capabilities, resulting in a sudden drop in data quality and low practical value.
A dynamic gating coding network is used for multimodal fusion. Feature vectors are acquired through visual and radar sensors, and gating fusion and medical information database retrieval are performed to generate health monitoring results that conform to clinical standards.
It maintains robustness in complex and dynamic environments, improves the accuracy and reliability of health monitoring, and provides interpretable health judgments.
Smart Images
Figure CN121714248A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of health monitoring, in particular to a passenger health monitoring method and system. BACKGROUND
[0002] The vehicle cabin environment is a typical dynamic uncontrolled environment. Light mutation, partial occlusion (such as passenger adjusting posture), sensor noise, etc. will all cause the data quality of a single modality to drop instantaneously. The performance of traditional data fusion methods will deteriorate sharply in such scenarios.
[0003] And there is a huge semantic gap from sensor data to health conclusions with clinical significance. Traditional classification models can usually only output "normal / abnormal" labels or isolated parameters, and have low practical value. SUMMARY
[0004] To solve the above technical problems, the present application provides a passenger health monitoring method and system which can dynamically adapt to complex real environments and has clinical reasoning ability.
[0005] Specifically, the present application provides a passenger health monitoring method, comprising the following steps: obtaining a passenger real-time image and a radar echo signal to obtain an image feature vector and a radar feature vector respectively; performing gated fusion based on the image feature vector and the radar feature vector to obtain a fusion feature; and obtaining a passenger preliminary health description according to the fusion feature, and retrieving a medical description in a medical information database based on the passenger preliminary health description to obtain a health monitoring result by combining the passenger preliminary health description and the medical description.
[0006] In the above technical solution, a dynamic gated encoding network is used. Through this method, an attention weight can be calculated for each modality in parallel, thereby realizing modality redundancy and intelligent switching. Even if a certain modality temporarily fails, the system will not crash, but will rely on another modality to maintain basic monitoring ability, ensuring the robustness of the system in complex dynamic environments. And instead of relying solely on inherent knowledge, the generated "passenger preliminary health description" is used as a query to map low-level features to health monitoring results that conform to clinical standards, improving the accuracy of health monitoring.
[0007] Further, obtaining the image feature vector comprises: dividing the passenger real-time image into a plurality of image blocks, projecting each image block into a block vector, and adding position encoding to obtain the image feature vector.
[0008] In the above technical solution, compared to pixel-by-pixel processing, block processing greatly reduces the sequence length and reduces the computational complexity. Adding position encoding informs the model of the absolute or relative position of each image block in the original image, making up for the loss of two-dimensional spatial structure information after block projection.
[0009] Further, the radar feature vector is obtained by: performing fast Fourier transform on the radar echo signal in the fast time dimension to obtain a frequency domain signal; generating a range profile based on the frequency domain signal and a radar transmission signal; performing a second fast Fourier transform on the radar echo signal in the slow time dimension to obtain a Doppler profile in combination with the radar echo signal; and constructing a range-Doppler matrix based on the range profile and the Doppler profile to obtain a radar feature map from the range-Doppler matrix.
[0010] In the above technical solution, the original radar echo signal is converted into a range-Doppler matrix with rich space-time information, providing high-quality input features for subsequent deep learning models.
[0011] Further, obtaining the radar feature vector further comprises: encoding the radar feature map through a convolutional neural network to obtain the radar feature vector.
[0012] In the above technical solution, the high-dimensional feature map is compressed into a fixed-length feature vector to facilitate subsequent multi-modal fusion.
[0013] Further, the image feature vector and the radar feature vector are fused based on the image feature vector and the radar feature vector to obtain a fusion feature, comprising: mapping the image feature vector and the radar feature vector to the same dimension space to obtain visual unified features and radar unified features respectively; and weighting and fusing the visual unified features and the radar unified features based on dynamic weights to obtain the fusion feature.
[0014] In the above technical solution, the two modal features are mapped to a unified dimension space, eliminating the difference in feature dimensions; the weights are dynamically adjusted based on the environment, improving the adaptability of complex real scenes.
[0015] Further, the passenger preliminary health description is obtained by: decoding the fusion feature into a passenger preliminary health description through a preset model, the passenger preliminary health description including at least an external performance description and a physiological index description.
[0016] In the above technical solution, the key conversion from data to information is completed, providing high-quality text input for subsequent knowledge retrieval.
[0017] Further, a plurality of medical entries are generated based on medical literature in advance and stored in a medical information database; and a medical description is obtained by: converting the passenger preliminary health description into a query statement, and vectorizing the query statement to extract a group of most similar medical entries as the medical description based on the vectorized query statement in the medical information database.
[0018] In the technical solution, the medical information database is searched to provide an explainable, traceable and clinically aligned context for the subsequent model, thereby improving the accuracy and reliability of health judgment.
[0019] Further, the health monitoring result is obtained by merging the medical description and the preliminary health description of the passenger to generate a prompt word, and decoding the prompt word by a preset model to obtain the final health monitoring result.
[0020] In the technical solution, the information alignment and ambiguity elimination are realized by merging to generate a prompt word, thereby improving the accuracy of the monitoring result; and the professional report is generated by the preset model.
[0021] Based on the same concept, the application further provides a passenger health monitoring system, which comprises a sensor for acquiring real-time images and radar echo signals of a passenger, a domain controller for outputting a final health monitoring result to a display based on the acquisition result of the sensor, and the display for displaying and / or broadcasting the health monitoring result.
[0022] Further, the domain controller comprises at least one processor and at least one memory for storing a computer program, and the at least one processor is configured to execute the computer program stored in the memory, so that the domain controller executes the passenger health monitoring method.
[0023] Compared with the prior art, the application has the following advantages: The application adopts a dynamic gating coding network, which can calculate an attention weight for each modality in parallel, thereby realizing modality redundancy and intelligent switching, and ensuring the robustness of the system in a complex dynamic environment; and the low-level features can be mapped to a health monitoring result conforming to clinical specifications, thereby improving the accuracy of health monitoring. BRIEF DESCRIPTION OF DRAWINGS
[0024] Figure 1 A flowchart of the passenger health monitoring method according to the application.
[0025] Figure 2 A timing diagram of the passenger health monitoring method according to the application.
[0026] Figure 3 A framework diagram of the passenger health monitoring system according to the application.
[0027] Figure 4 A framework diagram of the domain controller according to the application. DETAILED DESCRIPTION
[0028] The passenger health monitoring method and system of the present application will be further described in detail below in combination with specific embodiments and drawings.
[0029] Please refer to Figures 1-2 The present application provides a passenger health monitoring method, comprising the following steps S100-S300.
[0030] In a feasible implementation, the visual sensor collects passenger facial images in real time and transmits data to the domain controller, and the pre-deployed visual feature encoder module (based on the ViT model) extracts image feature vectors; the millimeter wave radar sensor synchronously collects chest micro-motion signals, and through the DSP, fast Fourier transform is performed to generate a distance Doppler matrix, and then the radar feature encoder module (based on CNN) extracts radar feature vectors; further, the dynamic gating fusion network runs in the domain controller, and the above two features are adaptively weighted and fused to output fusion features, which are input into the health state decoder module (lightweight LLM based on LoRA fine-tuning) running in the domain controller to generate a preliminary health description of the passenger; then, the medical retrieval module retrieves relevant medical items in the medical information database, merges them with the preliminary health description of the passenger to construct enhanced prompt words, and generates a final structured health monitoring result by the same LLM; the final result is displayed in real time by the vehicle-mounted display and can be broadcast through the voice unit.
[0031] Next, the specific implementation of steps S100-S300 will be described in detail.
[0032] Step S100: Obtain passenger real-time images and radar echo signals to obtain image feature vectors and radar feature vectors, respectively.
[0033] Among them, obtaining the image feature vector comprises: dividing the passenger real-time image into a plurality of image patches, projecting each image patch into a patch vector, and adding position encoding to obtain the image feature vector.
[0034] In some embodiments, the visual sensor collects passenger real-time images (such as facial images) at a rate of, for example, 30 frames per second, and the actual real-time processing is at a rate of 1-2 frames per second, and then the passenger real-time image is transmitted to the domain controller for the following processing: Image preprocessing: for example, adjusting the resolution of the passenger real-time image and performing normalization; it should be noted that those skilled in the art can also perform other preprocessing operations on the image according to actual application requirements, which is only an example here and is not limited to this.
[0035] Patch and projection: the image is divided into, for example, 14x14 16x16 pixel patches (i.e. image patches), and each patch is mapped to, for example, a 768-dimensional vector (i.e. patch vector) through a linear projection layer.
[0036] Location encoding: Add a learnable location encoding vector to each patch.
[0037] ViT (Vision Transformer) encoding: After adding the position encoding vector, the block vector is input to, for example, a 12-layer Transformer encoder, and the output image feature vector is generated.
[0038] Among them, visual sensors refer to optical devices installed in the vehicle cabin to collect images of passengers, such as: Infrared camera: It can work in low light or nighttime conditions and forms images by detecting infrared light emitted by the human body.
[0039] Visible light camera: captures color or grayscale images under normal lighting conditions.
[0040] Or RGB-IR camera: Supports both visible light and infrared imaging, adaptable to all weather conditions.
[0041] In this embodiment, the visual sensor is used to extract visual features such as passenger facial expressions, posture, and skin color.
[0042] Furthermore, ViT is a visual feature extraction model based on the Transformer architecture. Its core idea is to divide the image into multiple small patches and model global dependencies through a self-attention mechanism.
[0043] In the above technical solution, compared with pixel-by-pixel processing, block processing greatly reduces the sequence length and computational complexity. Adding position encoding tells the model the absolute or relative position of each image block in the original image, making up for the two-dimensional spatial structure information lost after block projection.
[0044] Furthermore, obtaining the radar feature vector includes: performing a fast Fourier transform on the radar echo signal in the fast time dimension to obtain a frequency domain signal; generating a range spectrum based on the frequency domain signal and the radar transmitted signal; performing a second fast Fourier transform on the radar echo signal in the slow time dimension to obtain a Doppler spectrum by combining the radar echo signal; and constructing a range-Doppler matrix based on the range spectrum and the Doppler spectrum to obtain a radar feature map based on the range-Doppler matrix.
[0045] In some embodiments, a Fast Fourier Transform (FFT) is performed on the sampled data (i.e., the fast time dimension) within each linear frequency modulated waveform (Chirp signal) cycle to convert the time-domain signal (i.e., the radar echo signal) to a frequency-domain signal. For example, an FFT is performed on 256 sample points for each Chirp cycle. Then, the frequency difference between the frequency-domain signal and the radar transmitted signal is calculated to obtain the beat frequency, generating a range spectrum, where the target distance r is: ; in, For beat frequency, This represents the frequency modulation slope.
[0046] Then, a second FFT is performed on the complex data (slow time dimension) of the same distance cell over multiple consecutive Chirp cycles. Utilizing the Doppler effect, the radial velocity of the target is detected through phase changes. The velocity calculation formula is as follows: ; in, For Doppler frequency shift, λ is the wavelength.
[0047] Furthermore, distance information and speed information Two-dimensional combination is performed to construct a range-Doppler matrix. The matrix is then normalized by modulo and logarithmic operations to enhance feature contrast and form the final RDI feature map (i.e., radar feature map).
[0048] It should be noted that this embodiment utilizes the electromagnetic wave penetration capability of a millimeter-wave radar sensor to detect subtle movements in the passenger's chest cavity, thereby directly measuring respiratory and heart rate, unaffected by light or obstruction; for example, a millimeter-wave radar preferably in the 60GHz band is preferred, which uses frequency modulated continuous wave (FMCW) mode to detect the target's distance, speed, and angle information by transmitting a linear frequency modulated signal and receiving its echo.
[0049] Furthermore, RDI is a two-dimensional radar signal representation, with the horizontal axis representing range and the vertical axis representing velocity. Each pixel value represents the signal strength at that range and velocity.
[0050] In the above technical solution, the original radar echo signal is transformed into a range-Doppler matrix with rich spatiotemporal information, providing high-quality input features for subsequent deep learning models.
[0051] Furthermore, obtaining the radar feature vector also includes: performing feature encoding on the radar feature map using a convolutional neural network to obtain the radar feature vector.
[0052] In some embodiments, a convolutional neural network (CNN) is used to encode the radar RDI feature map. Specifically, the encoder contains four consecutive convolutional feature layers, each consisting of a convolution operation, a non-linear activation function, and a downsampling operation, forming an end-to-end feature encoding. Each convolutional layer employs max pooling to achieve a 2x spatial downsampling. Finally, a flattening operation is performed to compress the high-dimensional feature map into a fixed-length feature vector. Specifically, after four convolutions + pooling and the final flattening operation, the input RDI feature map is encoded into a feature vector (i.e., the radar feature vector).
[0053] In the above technical solution, the high-dimensional feature map is compressed into a fixed-length feature vector to facilitate subsequent multimodal fusion.
[0054] Step S200: Perform gated fusion based on the image feature vector and radar feature vector to obtain fused features.
[0055] Specifically, the gated fusion based on the image feature vector and radar feature vector to obtain fused features includes: mapping the image feature vector and radar feature vector to the same dimensional space to obtain visual unified features and radar unified features respectively; and weighting the visual unified features and radar unified features based on dynamic weights to obtain fused features.
[0056] In some embodiments, the dynamic gated coding network first maps the features of the two modalities to a unified dimensional space through linear projection, eliminating the difference in feature dimensions and obtaining visually unified features. Uniform characteristics with radar Visual and radar unified features are input into the confidence assessment module to obtain the dynamic weights of the visual and radar signals. Specifically, the visual signal weights... for: ;in, Accordingly, the dynamic weights of radar features are: It is a multilayer perceptron network.
[0057] Therefore, the output fusion characteristics of the dynamic gated coding network for: .
[0058] Obtained fusion features This will be further input into a preset model (such as an LLM model).
[0059] It should be noted that, for example, when passengers are wearing masks, which limits image feature extraction, the system can automatically reduce visual weight and increase radar weight to ensure the output of respiratory rate and heart rate data.
[0060] In the above technical solution, the two modal features are mapped to a unified dimensional space to eliminate the difference in feature dimensions; the weights are dynamically adjusted based on the environment, which improves the adaptability to complex real-world scenarios.
[0061] Step S300: Obtain a preliminary health description of the passenger based on the fusion features, and retrieve a medical description from the medical information database based on the preliminary health description of the passenger, so as to combine the preliminary health description of the passenger and the medical description to obtain a health monitoring result.
[0062] Obtaining a preliminary health description of a passenger includes: decoding the fused features into a preliminary health description of the passenger using a preset model, wherein the preliminary health description of the passenger includes at least a description of external manifestations and a description of physiological indicators.
[0063] In some embodiments, fusion features After being converted into a text embedding sequence by a projection layer, it is input into a pre-trained LLM (Large Language Model). The LoRA (Low-Rank Adaptation) fine-tuning strategy is adopted to output a natural language description, such as "The passenger's breathing rate is 22 breaths / minute, heart rate is 98 beats / minute, face is slightly flushed, and body posture is leaning forward."
[0064] LoRA is a lightweight fine-tuning method that adds a low-rank matrix to the model weights, allowing it to adapt to new tasks with only a small number of parameters trained, significantly reducing computational and storage overhead. The number of training parameters is only 0.1% of the original model.
[0065] The above technical solution completes the key transformation from data to information, providing high-quality text input for subsequent knowledge retrieval.
[0066] Furthermore, several sets of medical entries are generated in advance based on medical literature and stored in a medical information database; obtaining a medical description includes: converting the passenger's preliminary health description into a query statement, and vectorizing the query statement, so as to extract the most similar set of medical entries in the medical information database based on the vectorized query statement as a medical description.
[0067] In some embodiments, the medical information database includes “symptom-diagnosis-treatment” triplets extracted from authoritative medical literature such as existing clinical medical guidelines, and formatted as entries. Each entry contains a paragraph of natural language describing the phenomenon, cause, and treatment recommendations of the disease symptoms.
[0068] The process involves converting passengers' initial health descriptions into natural language queries, encoding the queries using the same medical embedding model as the database to ensure semantic alignment, and finally obtaining the output results (i.e., medical descriptions) through retrieval and reordering, while providing corresponding information retrieval feedback.
[0069] In the above technical solution, retrieving medical information databases provides interpretable, traceable, and clinically aligned context for subsequent models, thereby improving the accuracy and reliability of health judgments.
[0070] Furthermore, obtaining health monitoring results includes: merging the medical description with the passenger's preliminary health description to generate a prompt word; and decoding the prompt word using a preset model to obtain the final health monitoring result.
[0071] In some embodiments, the query results are combined with the passenger's initial health description to synthesize a new comprehensive prompt word to enhance the original output information. Then, the same LLM model is used to decode the comprehensive prompt word and finally generate professional health monitoring results, such as "According to the medical knowledge base, facial flushing accompanied by shortness of breath may indicate mild hypoxia or allergic reaction. It is recommended to check the air quality in the vehicle and keep it ventilated. A high heart rate may be related to anxiety. It is recommended to observe the passenger's emotional state."
[0072] In addition, the text of the monitoring results can be displayed on the vehicle's screen, and the corresponding results can also be broadcast via voice. If an emergency is detected, an alarm can be automatically triggered and navigation to the nearest hospital can be suggested. The above functions can be selected and set by those skilled in the art, and are not limited to these.
[0073] In the above technical solution, information alignment and ambiguity are achieved by merging generated prompt words, thereby improving the accuracy of monitoring results; and the preset model helps to generate professional reports.
[0074] In summary, the passenger health monitoring method described in this application employs a dynamic gating coding network. This method can compute an attention weight in parallel for each modality, thereby achieving modal redundancy and intelligent switching. Even if a certain modality temporarily fails, the system will not crash, but will rely on another modality to maintain basic monitoring capabilities, ensuring the robustness of the system in complex dynamic environments. Furthermore, it does not rely solely on inherent knowledge, but uses the generated "preliminary passenger health description" as a query to map low-level features into health monitoring results that conform to clinical standards, thus improving the accuracy of health monitoring.
[0075] Based on the same concept, please refer to Figure 3This application also provides a passenger health monitoring system, the system comprising: a sensor for acquiring real-time images and radar echo signals of passengers; a domain controller for outputting final health monitoring results to a display based on the acquisition results of the sensor; and the display for displaying and / or broadcasting the health monitoring results.
[0076] In some embodiments, the sensor includes a visual sensor and a millimeter-wave radar sensor; the visual sensor is, for example, an infrared camera, a visible light camera, or an RGB-IR camera; the millimeter-wave radar sensor is preferably a 60 GHz band millimeter-wave radar.
[0077] Furthermore, the domain controller employs an automotive-grade SoC; the display is configured to present an interface and / or other visual information to vehicle occupants (e.g., operators, passengers). In some examples, the display includes a head-up display, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a flat panel display, a solid-state display, and / or any other type of display configured to present an interface and / or other visual information to vehicle occupants. Additionally, in this embodiment, the display is also equipped with a speaker to announce currently obtained health monitoring results.
[0078] For further details, please see Figure 4 The domain controller includes at least one processor and at least one memory, the at least one memory being used to store a computer program, and the at least one processor being used to execute the computer program stored in the memory, causing the domain controller to perform the passenger health monitoring method.
[0079] The processor can be any suitable processing device or set of processing devices, such as, but not limited to, a microprocessor, a microcontroller-based platform, an integrated circuit, one or more field-programmable gate arrays (FPGAs) and / or one or more application-specific integrated circuits (ASICs); the memory can be volatile memory (e.g., RAM including non-volatile RAM, magnetic RAM, ferroelectric RAM, etc.), non-volatile memory (e.g., disk storage, flash memory, EPROM, EEPROM, memristor-based non-volatile solid-state memory, etc.), immutable memory (e.g., EPROM), read-only memory and / or high-capacity storage devices (e.g., hard disk drives, solid-state drives, etc.).
[0080] To further explain, memory is a computer-readable medium on which one or more instruction sets (such as software for operating the methods disclosed herein) may be embedded. Instructions may embody one or more of the methods or logic described herein. For example, instructions may reside wholly or at least partially in memory, computer-readable medium, and / or within a processor during execution.
[0081] The memory includes multiple functional modules, such as a visual feature encoder module, a radar signal processing and encoding module, a dynamic gating fusion network, and a health status decoder and medical retrieval module; specifically: Visual feature encoder module: It uses a pre-trained ViT model to convert real-time images of passengers collected by visual sensors into visual codes (i.e., image feature vectors).
[0082] Radar signal processing and encoding module: adopts a CNN model; in which, the radar echo signal collected by the millimeter-wave radar sensor is converted into a range Doppler matrix with rich spatiotemporal information, and a convolutional neural network is used to encode the range Doppler matrix. Specifically, the convolutional layer uses max pooling to achieve 2x spatial downsampling, and finally, through flattening operation, the high-dimensional feature map (i.e., the range Doppler matrix) is compressed into a fixed-length feature vector (radar feature vector).
[0083] Dynamic Gated Fusion Network: Implemented using a lightweight MLP; modal weights are dynamically adjusted according to environmental conditions to achieve optimal fusion of visual and radar features to obtain fused features.
[0084] The health status decoder and medical retrieval module integrate a lightweight LLM and vector retrieval engine. The health status decoder uses a general large language model (LLM) as its main structure. After fine-tuning with LoRA, it transforms the input fused features into a description containing passenger status, respiratory rate, and heart rate data, serving as the passenger's preliminary health description. In the medical retrieval module, the passenger's preliminary health description is first converted into a query vector through text embedding. Then, vector similarity is calculated in the vector space of the medical information database. The most similar medical description in the vector space is obtained through a general vector search technique using RAG (Retrieval-Augmented Generation). The retrieval result is combined with the passenger's preliminary health description into a comprehensive prompt word. This comprehensive prompt word is then fed into a result generation module composed of a general large oracle model to generate the final health monitoring result.
[0085] It should be noted that RAG is a technical framework that combines retrieval and generation. By incorporating external knowledge base retrieval results during the generation process, it improves the accuracy and interpretability of the output.
[0086] In addition, the domain controller also includes an input / output interface, which can be the input / output circuit in the domain controller, to output the information processed by the processor to the display, and to receive the signals acquired by the sensor and send them to the processor for corresponding processing.
[0087] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.
[0088] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0089] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.
[0090] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.
[0091] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0092] Although the description of this application has been made in conjunction with the specific embodiments described above, it will be apparent to those skilled in the art that many substitutions, modifications, and variations can be made based on the foregoing. Therefore, all such substitutions, modifications, and variations are included within the spirit and scope of the appended claims.
Claims
1. A method for monitoring passenger health, characterized in that, Includes the following steps: Real-time images of passengers and radar echo signals are acquired to obtain image feature vectors and radar feature vectors, respectively. Gated fusion is performed based on the image feature vector and radar feature vector to obtain fused features; Furthermore, a preliminary health description of the passenger is obtained based on the fusion features, and a medical description is retrieved from a medical information database based on the preliminary health description of the passenger, so as to obtain a health monitoring result by combining the preliminary health description of the passenger and the medical description.
2. The passenger health monitoring method according to claim 1, characterized in that, Obtaining image feature vectors includes: The real-time passenger image is divided into multiple image blocks, each image block is projected into a block vector, and position encoding is added to obtain the image feature vector.
3. The passenger health monitoring method according to claim 1, characterized in that, Obtaining radar feature vectors includes: Fast Fourier Transform is performed on radar echo signals in the fast time dimension to obtain frequency domain signals; Range spectrum is generated based on frequency domain signals and radar transmitted signals; A second fast Fourier transform is performed on the radar echo signal with a slow time dimension to obtain the Doppler spectrum by combining the radar echo signal. Furthermore, a range-Doppler matrix is constructed based on the range spectrum and Doppler spectrum to obtain a radar feature map according to the range-Doppler matrix.
4. The passenger health monitoring method according to claim 3, characterized in that, Obtaining radar feature vectors also includes: The radar feature map is encoded using a convolutional neural network to obtain a radar feature vector.
5. The passenger health monitoring method according to claim 1, characterized in that, Gated fusion is performed based on the image feature vector and radar feature vector to obtain fused features, including: The image feature vector and radar feature vector are mapped to the same dimensional space to obtain visual unified features and radar unified features respectively; Furthermore, the visual unified features and radar unified features are weighted and fused based on dynamic weights to obtain fused features.
6. The passenger health monitoring method according to claim 5, characterized in that, Obtain a preliminary health description of the passenger, including: The fused features are decoded into a preliminary health description of the passenger using a preset model. The preliminary health description of the passenger includes at least a description of external manifestations and a description of physiological indicators.
7. The passenger health monitoring method according to claim 6, characterized in that, Several sets of medical entries are generated in advance based on medical literature and stored in a medical information database; Obtain a medical description, including: The passenger's preliminary health description is converted into a query statement, and the query statement is vectorized. Based on the vectorized query statement, the most similar set of medical entries in the medical information database is extracted as a medical description.
8. The passenger health monitoring method according to claim 7, characterized in that, Obtain health monitoring results, including: The medical description is combined with the passenger's preliminary health description to generate prompt words; Furthermore, the prompt words are decoded using a preset model to obtain the final health monitoring results.
9. A passenger health monitoring system, characterized in that, The system includes: Sensors are used to acquire real-time images and radar echo signals of passengers; A domain controller is used to output the final health monitoring results to a display based on the acquisition results of the sensors; The display is used to display and / or broadcast the health monitoring results.
10. The passenger health monitoring system according to claim 9, characterized in that, The domain controller includes at least one processor and at least one memory, the at least one memory being used to store a computer program, and the at least one processor being used to execute the computer program stored in the memory, such that the domain controller performs the passenger health monitoring method as described in any one of claims 1-8.