Intelligent triage classification AR glasses system based on multi-modal data

Abstract

The invention specifically discloses an intelligent triage classification AR glasses system based on multi-modal data, and relates to the technical field of medical risk management. The system comprises an AR glasses hardware terminal module, a multi-modal data acquisition module, an intelligent analysis processing module, an AR man-machine interaction module, a data storage and transmission module and a peripheral linkage module. Wherein the AR glasses hardware terminal is in lightweight design, the multi-modal data acquisition module acquires and optimizes video, voice, physiology and other data of the wounded, the intelligent analysis processing module outputs red, yellow, green and black triage results through space-time alignment, feature fusion and AI analysis, the AR man-machine interaction module is used for optimizing interaction, and the data storage and transmission module guarantees data safety. And the peripheral linkage module realizes label printing and NFC writing. The system is short in single-time triage time, triage accuracy and environmental adaptability can be improved, information tracing is achieved, two hands of medical care are liberated, and rescue efficiency and the survival rate of wounded personnel are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of medical risk management technology, and in particular to an intelligent triage and classification AR glasses system based on multimodal data. Background Technology

[0002] Triage, a core component of disaster medicine and emergency medicine, aims to rapidly classify injured individuals (typically into four levels: red, yellow, green, and black) based on the urgency of their injuries and treatment priorities in situations with limited medical resources, thereby maximizing rescue efficiency and survival rates. However, traditional triage methods rely excessively on the experience and judgment of medical personnel, are significantly influenced by subjective factors, and are prone to misjudgment due to fatigue and stress in high-intensity rescue environments. Furthermore, the fragmented nature of manually recorded information makes it difficult to achieve full traceability and efficient handover of injured personnel data.

[0003] Meanwhile, existing technologies have multiple limitations in triage scenarios: poor environmental adaptability, with visual observation and traditional equipment prone to failure in complex conditions such as nighttime, smoke, rain, and snow; significant efficiency bottlenecks, as manual inspection and recording of each casualty in large-scale casualty incidents is insufficient to handle massive numbers of injured; and limited data dimensions, with most electronic triage devices only collecting single types of data such as vital signs or visual information, lacking multi-dimensional cross-validation, thus limiting the accuracy of judgments. These shortcomings may delay the best treatment opportunity in emergency rescue, affecting the overall rescue outcome.

[0004] Some AR glasses-based technologies have been attempted to be applied in the field of triage, such as AR devices that integrate physiological parameter acquisition. However, there are still significant shortcomings in key technologies: the ability to deeply integrate multimodal data is weak, and it is impossible to achieve collaborative analysis of visual, voice, and physiological signals; the level of intelligent automatic analysis is limited, making it difficult to replace professionals in completing complex injury assessments; the ability to work stably in harsh environments is insufficient, and a solution covering all scenarios has not yet been formed, and the core pain points of traditional triage methods have not been systematically addressed. Summary of the Invention

[0005] The purpose of this invention is to propose an intelligent triage AR glasses system based on multimodal data to achieve rapid, objective, accurate, and all-weather automated triage, thereby improving rescue efficiency and patient survival rate.

[0006] To achieve the above objectives, this invention proposes an intelligent damage detection and classification AR glasses system based on multimodal data, comprising: an augmented reality AR glasses hardware terminal, a multimodal data acquisition module, an intelligent analysis and processing module, an AR human-computer interaction module, a data storage and transmission module, and a peripheral linkage module; The AR glasses hardware terminal adopts a lightweight, ergonomic glasses-like design and incorporates a micro-computing unit, battery, optical display system, camera module, microphone array, ambient light sensor, NFC reader and writer, and communication module. The multimodal data acquisition module is used to collect video streams, image data, voice data, physiological data and environmental data of the wounded, and to perform dynamic range compression and local sharpening, edge preservation and noise reduction and edge enhancement on the collected visual data, and to perform quantitative analysis of bleeding volume. The intelligent analysis and processing module is deployed locally on the glasses or in collaboration with edge computing and the cloud to perform spatiotemporal alignment, feature-level fusion, dedicated artificial intelligence (AI) analysis, and intelligent defect classification decision-making on multimodal data. The AR human-computer interaction module is used to overlay the analysis results onto the real field of vision in the form of virtual labels. It supports voice command control and voice-to-text conversion, and guides medical staff to complete the standardized triage process through AR arrows and highlighted prompts. The data storage and transmission module is used to encrypt and store data locally, and transmit the data to the command center or cloud platform in real time via 5G or 4G network. It uses MQTT over SSL / TLS protocol for data transmission and establishes a secure data queue locally to achieve data buffering and retransmission. The peripheral linkage module is used to control a portable printer to print injury identification wristbands or tags via Bluetooth commands, and to encrypt and write detailed injury data into the near-field communication (NFC) chip inside the tag.

[0007] Preferably, the multimodal data acquisition module includes a visual acquisition unit, a voice acquisition unit, a physiological data acquisition unit, and an environmental perception unit; The visual acquisition unit uses an RGB camera with digital image stabilization and low-light enhancement, along with an infrared camera, to acquire video streams and image data of the injured. The voice acquisition unit uses a beamforming microphone array to collect voice conversations between medical staff and the injured, ambient sounds, and the injured's groans and breathing sounds. The physiological data acquisition unit initially monitors vital signs through contact points on the temples of glasses or miniature non-contact sensors, and wirelessly connects to professional portable monitoring devices via Bluetooth or NFC to read data on blood pressure, body temperature, heart rate, and blood oxygen saturation. The environmental sensing unit is used to collect ambient light intensity and temperature information to enable the system to adaptively adjust.

[0008] Preferably, the images acquired by the vision acquisition unit are processed, including: A multi-scale brightness adaptive algorithm based on the Retinex theory of the human visual system is used to perform dynamic range compression and local sharpening on the fused image to ensure that the bright areas are not overexposed and the dark areas retain details. Nonlocal mean (NLM) denoising algorithm is used for edge-preserving denoising, combined with a phase congruency-based edge detection algorithm for edge enhancement. The essential structural edges of the wound are extracted, and the degree and shape of the tear are determined. The formula is as follows: ; in, For the pixels to be denoised, For other pixels within the search window, and For pixels , grayscale value, These are the filter parameters. The normalization constant is For p and q The squared Gaussian-weighted Euclidean distance between the central pixel blocks. For Gaussian kernel, For the search window, This represents the denoising value for pixel p in image L; Normalization constant The formula is as follows: ; Within the segmented trauma area, the pixel values ​​of the blood region under RGB and NIR are analyzed. Combined with the depth information provided by the depth camera, the bleeding area is estimated, and the bleeding volume is assessed in a three-level semi-quantitative manner, including: oozing bleeding, active bleeding, and spurting bleeding.

[0009] Preferably, the intelligent analysis and processing module includes a multimodal data fusion engine, a dedicated AI analysis model set, and an intelligent damage detection classification decision model; The multimodal data fusion engine adopts a feature-level fusion model based on deep learning. It achieves time alignment of multimodal data by combining linear interpolation and dynamic time warping (DTW). It achieves spatial alignment by combining the visual inertial odometry (VIO) module of AR glasses with target detection. Then, it performs feature-level fusion of the feature vectors of each modality through a cross-modal encoder based on the deep learning Transformer architecture, and outputs a global feature vector F with a fixed dimension. The dedicated AI analysis model set includes a bleeding recognition model, a fracture probability judgment model, a facial color recognition model, a breathing sound or groaning sound recognition model, and a text recognition OCR model, which are used to identify the injury site and bleeding volume, analyze the probability of limb fracture, judge the patient's facial color status, identify respiratory rate and abnormal breathing sounds, and read digital information displayed by medical devices, respectively. The intelligent triage classification decision model adopts a hybrid expert system architecture, combining a rule engine based on the triage classification protocol START / JumpSTART with a deep learning model based on a multilayer perceptron (MLP) structure. It outputs triage classification results and confidence levels through a decision fusion and arbitration module.

[0010] The preferred formula for calculating confidence level is as follows: ; in, For the confidence level of the final result, For rule confidence, These are the probability values ​​for the corresponding levels output by the deep learning model. Score the data quality. These are adjustable weighting parameters.

[0011] Preferably, the bleeding recognition model performs bilateral filtering and adaptive histogram equalization (CLAHE) preprocessing on the input RGB image of the wound site, obtains a pixel-level segmentation mask through the semantic segmentation model U-Net++, outputs the bleeding level through bleeding region calculation and bleeding volume classification logic, and estimates the depth of the wound region by combining the depth map. The fracture probability judgment model takes an RGB image and depth map of the limb as input, and outputs the fracture result by the fracture classification branch of the multi-task learning convolutional neural network MT-CNN, the fracture probability regression branch outputs the fracture probability, and the joint localization branch outputs the joint coordinates. The facial color recognition model performs automatic white balance (AWB) and color space conversion on the input facial region of interest (ROI) image, calculates the mean and standard deviation of color statistical features, and outputs facial classification results and ROI localization through a lightweight spiking neural network (SNN). The breathing sound or groaning sound recognition model generates a Mel spectrogram of the preprocessed audio signal, extracts local frequency domain patterns through a 2D convolutional layer, and after a pooling layer, outputs the breathing frequency by detecting peak values ​​in the spectrogram and captures temporal dependencies through a gated recurrent unit (GRU) and outputs event classification results by combining them with a fully connected layer. The text recognition OCR model uses a CNN-based detector to detect text in the input device screen image. In text recognition, feature sequences are first extracted through convolutional layers, and then decoded sequences are generated using recurrent layers or Transformers. Character sequences are output through connectionist temporal classification CTC loss or attention mechanism, and finally structured data is output.

[0012] Preferably, the rule engine based on the damage classification protocol START / JumpSTART outputs classification suggestions through a hard-coded deterministic decision tree. The deep learning model takes a global feature vector F as input and outputs a four-dimensional probability vector. The decision fusion and arbitration module follows the rule priority principle, directly adopting the results judged as red or black by the rule engine, and performing fuzzy arbitration on the results of yellow or green combined with the prediction results of the deep learning model.

[0013] Preferably, when the peripheral linkage module controls the printer, Bluetooth Low Energy connection is used, and the instruction set adopts Zebra Programming Language (ZPL), Citizen Printer Command Language (CPCL), or the simplified printer control code ESC / POS instruction set; The unique ID format of the printed injury identification wristband or tag is event code-date-serial number, and the QR code content is a short URL and a unique ID; the top of the tag has a striking color block, the middle has a large font category level and a unique ID, and the bottom has a QR code; The instructions include status query commands. When the printer runs out of paper, the ribbon runs out, or communication is interrupted, the AR glasses notify the operator through flashing AR interface icons and voice prompts.

[0014] Preferably, the data written to the NFC chip by the peripheral linkage module includes text records, Uniform Resource Identifier (URI) records, and custom external records; Text records store basic injury information in plain text; URI records store the same URL as the QR code; and custom external records store complete JSON-formatted injury data encrypted using the AES-128-GCM algorithm. The AR glasses have a built-in unique device key. The system has a pre-set root key for deriving the device key when activating the device. Through AR visual prompts and voice guidance, the user keeps the writing terminal at the optimal read / write distance of 4cm. After writing, the data is read back for verification immediately. If it fails, it will automatically retry, up to 3 times. When the writing fails, the operator will be prompted to rely on the QR code.

[0015] Preferably, the data storage and transmission module and the peripheral linkage module meet the requirements of data minimization, encryption, access control security and privacy. The QR code and URI records only contain IDs. Detailed data is accessed through a secure interface with controlled permissions. Access to detailed data through the cloud API interface requires authentication through the OAuth 2.0 protocol and all access logs are recorded.

[0016] Therefore, this invention proposes an intelligent damage detection and classification AR glasses system based on multimodal data, which has the following advantages: (1) This invention realizes the automated acquisition of visual, speech and physiological data through the multimodal data acquisition module. Combined with the AI ​​model set and decision algorithm of the intelligent analysis and processing module, it will significantly break through the efficiency bottleneck of manual triage. At the same time, the system relies on the deep learning model to objectively identify injury features such as bleeding, fracture, and facial color. Combined with multimodal data cross-validation, it effectively avoids the problems of misjudgment and omission caused by medical staff due to fatigue, experience differences or environmental pressure. The triage classification accuracy rate is far higher than that of traditional manual methods.

[0017] (2) This invention utilizes the local encrypted storage and 5G / 4G real-time transmission function of the data storage and transmission module to synchronize structured data such as audio and video, physiological parameters, and AI analysis results during the triage process to the rear command center. At the same time, the color identification labels (including unique IDs and QR codes) printed by the peripheral linkage module and the detailed injury data encrypted and stored by the NFC chip open up the information handover channel from the rescue site to the rear hospital, ensuring that the information of the injured is traceable in all stages of transfer and treatment, avoiding the problems of easy loss and information disconnect of traditional paper records, and providing complete data support for subsequent diagnosis and treatment.

[0018] (3) The AR human-computer interaction module in this invention presents information such as triage level, vital signs, and treatment suggestions directly in the field of vision of medical staff through the "reality superimposed on virtual" method. At the same time, it supports voice command control (such as "start triage" and "print label"), which completely frees the hands of medical staff, allowing them to complete information collection and recording while performing emergency measures such as hemostasis and fixation, greatly optimizing the workflow. In addition, the judgment criteria output by the AI ​​model can assist medical staff in making quick decisions and can also provide training support for novice medical staff, improving the overall rescue capability of the team.

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall structure of an intelligent damage detection and classification AR glasses system based on multimodal data according to the present invention; Figure 2 This is a schematic diagram of the intelligent analysis and processing module in this invention; Figure 3 This is a schematic diagram of the bleeding recognition model in this invention; Figure 4 This is a schematic diagram of the fracture probability judgment model in this invention; Figure 5 This is a schematic diagram of the facial color recognition model in this invention; Figure 6 This is a schematic diagram of the structure of the breathing sound or groaning sound recognition model in this invention; Figure 7 This is a schematic diagram of the structure of the Chinese character recognition model in this invention; Figure 8 This is a schematic diagram of the intelligent damage detection classification decision model in this invention; Figure 9 This is a schematic diagram of the external linkage module in this invention; Figure 10 This is a flowchart of NFC information writing in this invention. Detailed Implementation

[0021] To make the technical solutions, advantages, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below. The described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0022] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0023] Example like Figure 1 As shown, the present invention provides an intelligent damage detection and classification AR glasses system based on multimodal data, including: an augmented reality AR glasses hardware terminal, a multimodal data acquisition module, an intelligent analysis and processing module, an AR human-computer interaction module, a data storage and transmission module, and a peripheral linkage module; The AR glasses hardware terminal adopts a lightweight, ergonomic glasses-like design and incorporates a micro-computing unit, battery, optical display system (waveguide technology), camera module (RGB camera with red, green and blue color model, infrared camera), microphone array, ambient light sensor, near-field communication NFC reader and writer, and communication module (5G / 4G, WiFi, Bluetooth). The multimodal data acquisition module includes a visual acquisition unit, a voice acquisition unit, a physiological data acquisition unit, and an environmental perception unit; The visual acquisition unit uses an RGB camera and an infrared camera with digital image stabilization and low-light enhancement to acquire video streams and image data of the injured. The voice acquisition unit uses a beamforming microphone array to collect voice conversations between medical staff and the injured, ambient sounds, and the injured's groans and breathing sounds. The physiological data acquisition unit initially monitors vital signs through contact points on the temples of glasses or miniature non-contact sensors, and wirelessly connects to professional portable monitoring devices via Bluetooth or NFC to read data on blood pressure, body temperature, heart rate, and blood oxygen saturation. The environmental sensing unit is used to collect ambient light intensity and temperature information to enable the system to adaptively adjust.

[0024] The trauma area in the image acquired by the visual acquisition unit is processed as follows: A multi-scale brightness adaptive algorithm based on the Retinex theory of the human visual system is used to perform dynamic range compression and local sharpening on the fused image to ensure that the bright areas are not overexposed and the dark areas retain details. Nonlocal means (NLM) denoising algorithm is used for edge-preserving denoising, which effectively removes noise while better protecting key details such as wound edges and foreign object contours. Edge enhancement is then performed using a phase congruency-based edge detection algorithm to extract the essential structural edges of the wound and determine the degree and shape of the tear. The formula is as follows: ; in, For the pixels to be denoised, For other pixels within the search window, and For pixels , grayscale value, These are the filter parameters. The normalization constant is For p and q The squared Gaussian-weighted Euclidean distance between the central pixel blocks. For Gaussian kernel, For the search window, This represents the denoising value for pixel p in image L; Normalization constant The formula is as follows: ; Within the segmented trauma area, the pixel values ​​of the blood region under RGB and NIR are analyzed. Combined with the depth information provided by the depth camera, the bleeding area is estimated, and the bleeding volume is assessed in a three-level semi-quantitative manner, including: oozing bleeding, active bleeding, and spurting bleeding.

[0025] The intelligent analysis and processing module is deployed locally on the glasses or in collaboration with edge computing and the cloud. It is used to perform spatiotemporal alignment, feature-level fusion, dedicated artificial intelligence (AI) analysis, and intelligent defect classification decision-making on multimodal data. It includes a multimodal data fusion engine, a dedicated AI analysis model set, and an intelligent defect classification decision-making model. like Figure 2As shown, the multimodal data fusion engine employs a deep learning-based feature-level fusion model to perform spatiotemporal alignment and joint analysis of visual, speech, and physiological data. The aim is to align and fuse data from different sources, formats, and time frequencies to generate a comprehensive and unified representation of the injured person's condition. This includes spatiotemporal alignment and feature-level fusion.

[0026] For time alignment, a combination of linear interpolation and dynamic time warping (DTW) algorithms is used. For well-ordered sequences such as physiological data, interpolation is used to generate data at a unified time point. For audio event sequences, DTW is used to non-linearly align them with video event sequences to find the best matching path. The input is a raw data stream with timestamps (video frames, audio buffers, physiological data packets), and the output is that all data streams are synchronized to the same master clock with a time deviation of less than 10ms. For spatial alignment, the visual inertial odometry (VIO) module of the AR glasses is used to calculate the glasses' pose in space in real time. Combined with target detection (such as YOLO), the position of the injured person in the image is determined. The recognition results (such as "head wound") are mapped to the real-world coordinate system through coordinate transformation. Input video frames, IMU data, and depth maps, output all recognition results and bind them to specific injured person IDs and body parts.

[0027] In terms of feature-level fusion, a cross-modal encoder based on deep learning Transformer architecture is used to perform feature-level fusion of feature vectors of each modality. All recognition results are bound to specific wounded IDs and body parts, and a fixed-dimensional global feature vector F that integrates all available information is output.

[0028] The dedicated AI analysis model set includes a bleeding recognition model, a fracture probability judgment model, a facial color recognition model, a breathing sound or groaning sound recognition model, and a text recognition OCR model, which are used to identify the injury site and bleeding volume, analyze the probability of limb fracture, judge the patient's facial color status, identify respiratory rate and abnormal breathing sounds, and read digital information displayed by medical devices, respectively. like Figure 3 As shown, the bleeding recognition model performs bilateral filtering and adaptive histogram equalization (CLAHE) preprocessing on the input RGB image of the wound site, obtains a pixel-level segmentation mask through the semantic segmentation model U-Net++, outputs the bleeding level through bleeding region calculation and bleeding volume classification logic, and estimates the depth of the wound region by combining the depth map. like Figure 4 As shown, the fracture probability judgment model takes an RGB image and depth map of the limb as input, and outputs the fracture status result through the fracture classification branch of the multi-task learning convolutional neural network MT-CNN, the fracture probability regression branch outputs the fracture probability, and the joint localization branch outputs the joint coordinates. like Figure 5As shown, the facial color recognition model performs automatic white balance (AWB) and color space conversion on the input facial ROI image, calculates the mean and standard deviation of color statistical features, and outputs facial classification results and ROI localization through a lightweight spiking neural network (SNN). like Figure 6 As shown, the breathing sound or groaning sound recognition model generates a Mel spectrogram for the preprocessed audio signal, extracts local frequency domain patterns through a 2D convolutional layer, and after a pooling layer, outputs the breathing frequency by detecting peak values ​​in the spectrogram and captures temporal dependencies through a gated recurrent unit (GRU) on the other hand, and outputs the event classification results by combining them with a fully connected layer. like Figure 7 As shown, the text recognition OCR model uses a CNN-based detector to detect text in the input device screen image. In text recognition, feature sequences are first extracted through convolutional layers, then decoded sequences are generated using recurrent layers or Transformers, and character sequences are output through connectionist temporal classification CTC loss or attention mechanism, finally outputting structured data.

[0029] like Figure 8 As shown, the intelligent triage classification decision model adopts a hybrid expert system architecture, combining a rule engine based on the triage classification protocol START / JumpSTART with a deep learning model based on a multilayer perceptron (MLP) structure. The model outputs triage classification results and confidence levels through a decision fusion and arbitration module.

[0030] The rule engine takes preprocessed and calibrated, explicit, and quantifiable indicators as input, including: can_walk (ability to walk), respiratory_rate (breathing rate), capillary_refill_time (capillary refill time), pulse_rate (pulse), and avpu_score (level of consciousness). It outputs classification suggestions through a hard-coded deterministic decision tree, with extremely high priority in its judgment logic, especially for the most critical cases, outputting a clear and standard-compliant classification suggestion rule_triage_category (red, yellow, green, black).

[0031] The deep learning model uses a simple multilayer perceptron (MLP) model. The input is a high-dimensional global feature vector F from the multimodal fusion engine, and the output is a four-dimensional probability vector dl_probabilities = [P(Red), P(Yellow), P(Green), P(Black)]; where P(Red) is the probability of red, P(Yellow) is the probability of yellow, P(Green) is the probability of green, and P(Black) is the probability of black.

[0032] The rule priority principle of the decision fusion and arbitration module: If the rule engine determines the situation as red (critical) or black (death), the system immediately adopts that result; for situations where the rule engine determines the situation as yellow (emergency) or green (minor injury), the system compares the rule result with the prediction result of the deep learning model, as follows: If the rule results are consistent with the prediction results of the deep learning model, the final result has a high degree of confidence.

[0033] If a deep learning model gives a more severe classification with a high probability (e.g., the rule classifies it as "green," but the model classifies it as "yellow" with a high probability), the system will prioritize the more severe classification and significantly lower the confidence level, forcing manual review. This can effectively prevent "missed classifications."

[0034] If a deep learning model provides a lighter classification, the rule-based results are usually maintained.

[0035] The confidence level is calculated using the following formula: ; in, For the confidence level of the final result, For rule confidence, These are the probability values ​​for the corresponding levels output by the deep learning model. Score the data quality. These are adjustable weighting parameters.

[0036] The AR human-computer interaction module is used to overlay analysis results onto the real field of vision in the form of virtual labels. It supports voice command control (such as "start triage", "record voice notes", "print labels") and voice-to-text conversion, and guides medical staff to complete the standardized triage process through AR arrows and highlighted prompts.

[0037] The data storage and transmission module is used to encrypt and store data locally, and transmit the data to the command center or cloud platform in real time via 5G or 4G network. It uses MQTT over SSL / TLS protocol for data transmission and establishes a secure data queue locally to achieve data buffering and retransmission.

[0038] like Figures 9-10 As shown, the peripheral linkage module is used to control the portable printer to print injury identification wristbands or tags via Bluetooth commands, and to encrypt and write detailed injury data into the NFC chip inside the tag.

[0039] When the peripheral linkage module controls the printer, it uses Bluetooth Low Energy connection and the instruction set adopts Zebra Programming Language ZPL, Compact Printer Control Language CPCL, or simplified printer control code ESC / POS instruction set. The unique ID format of the printed injury identification wristband or tag is event code-date-serial number, and the QR code content is a short URL and a unique ID; the top of the tag has a striking color block, the middle has a large font category level and a unique ID, and the bottom has a QR code; The instructions include status query commands. When the printer runs out of paper, the ribbon runs out, or communication is interrupted, the AR glasses notify the operator through flashing AR interface icons and voice prompts.

[0040] The data written to the NFC chip by the peripheral linkage module includes text records, URI records, and custom external records; Text records store basic injury information in plain text; URI records store the same URL as the QR code; and custom external records store complete JSON-formatted injury data encrypted using the AES-128-GCM algorithm. Meanwhile, the AR glasses have a built-in unique device key, and the system has a pre-set root key for deriving the device key when activating the device. Through AR visual prompts and voice guidance, users are guided to keep the writing terminal at the optimal read and write distance of 4cm. After writing, the verification is read back immediately. If it fails, it will automatically retry, with a maximum of 3 retries. When the writing finally fails, the operator is prompted to rely on the QR code.

[0041] The data storage and transmission module and peripheral linkage module meet the requirements of data minimization, encryption, access control security and privacy. The QR code and URI records only contain IDs. Detailed data is accessed through a secure interface with controlled permissions. Access to detailed data through the cloud API interface requires authentication via the OAuth 2.0 protocol and all access logs are recorded.

[0042] The invention will be further illustrated below through specific implementation examples.

[0043] A rescuer wearing AR glasses enters the disaster site and, upon discovering an injured person, the specific rescue procedure is as follows: Rescuers looked at the injured and gave the voice command "Begin triage".

[0044] The glasses' camera automatically turns on, recording video and capturing images. The visual analysis model then begins to work: it identifies active bleeding in the injured person's left thigh (bleeding recognition model), pale face (facial color model), and abnormal shape of the right lower leg (fracture probability model indicates high risk).

[0045] Rescuers attached the Bluetooth blood pressure cuff to the injured person's arm. After measuring the data, the glasses automatically recognized the reading on the blood pressure monitor screen (systolic pressure 85 mmHg, diastolic pressure 50 mmHg) through OCR and obtained the pulse oximeter data (blood oxygen saturation 92%) via Bluetooth.

[0046] The microphone picked up the faint groans of the wounded, and the audio analysis model calculated the respiratory rate to be 35 breaths per minute.

[0047] The multimodal data fusion engine integrates and analyzes all information (visual hemorrhage, pallor, low blood pressure, low blood oxygen, and rapid breathing).

[0048] Based on the START standard, the intelligent triage decision-making model determines the injured person to be classified as red (critical, priority treatment), and highlights the injured person with a striking red border on the AR display screen, while also displaying key indicators and conclusions.

[0049] After the rescuers confirmed the injury, they issued the instruction to "print a red label." The portable printer immediately printed a red QR code wristband, and at the same time, more detailed injury data was encrypted and written into the wristband's NFC chip.

[0050] Throughout the process, audio and video recordings, physiological data, and analysis results were encrypted, stored, and uploaded to the command center.

[0051] The entire process is completed within 1 minute, and is efficient, accurate, and provides complete information.

[0052] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0053] Therefore, this invention provides an intelligent triage AR glasses system based on multimodal data. Through the deep integration of multimodal data fusion, AI intelligent analysis and AR interaction technology, it significantly improves the efficiency and accuracy of triage, realizes full traceability of casualty information, enhances adaptability in complex environments, and optimizes the operation process of medical staff. It effectively solves the pain points of traditional triage methods, such as strong subjectivity, fragmented information and poor environmental adaptability, and comprehensively improves the efficiency of rescue coordination and the survival rate of casualties.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An intelligent triage classification AR glasses system based on multi-modal data, characterized in that, The application relates to an AR (Augmented Reality) glasses terminal for first-aid treatment, which comprises the following modules: an AR glasses hardware terminal, a multi-modal data acquisition module, an intelligent analysis processing module, an AR human-computer interaction module, a data storage and transmission module, and a peripheral linkage module. The AR glasses hardware terminal adopts a lightweight and ergonomic glasses form design, and is internally provided with a micro computing unit, a battery, an optical display system, a camera module, a microphone array, an ambient light sensor, an NFC reader and a communication module. The multi-modal data acquisition module is used for acquiring video streams, image data, voice data, physiological data and environmental data of the wounded, and performing dynamic range compression and local sharpening, edge-preserving denoising and edge enhancement on the acquired visual data, and performing bleeding volume quantization analysis. The intelligent analysis processing module is disposed in the glasses locally or cooperates with edge computing and a cloud end, and is used for performing space-time alignment, feature-level fusion, special AI analysis and intelligent injury classification decision on the multi-modal data. The AR human-computer interaction module is used for superimposing the analysis result in the form of a virtual label on the real field of view, supporting voice instruction control and voice-to-text conversion, and guiding medical staff to complete a standardized injury treatment process through AR arrows and highlight prompts. The data storage and transmission module is used for locally storing data in an encrypted manner, and transmitting the data to a rear command center or a cloud platform in a real-time and encrypted manner through a 5G or 4G network, adopts an MQTT over SSL / TLS protocol for data transmission, and establishes a secure data queue locally to realize data buffering and retransmission. The peripheral linkage module is used for controlling a portable printer to print an injury condition identification bracelet or label through a Bluetooth instruction, and writing detailed injury condition data into an NFC chip in the label in an encrypted manner. The multi-modal data acquisition module comprises a visual acquisition unit, a voice acquisition unit, a physiological data acquisition unit and an environmental perception unit.

2. The intelligent triage classification AR glasses system based on multi-modal data according to claim 1, characterized in that, The visual acquisition unit adopts a red-green-blue color model (RGB) camera and an infrared camera with digital anti-shake and low-light enhancement functions, and is used for acquiring video streams and image data of the wounded. The voice acquisition unit adopts a beamforming microphone array, and is used for acquiring voice conversations of medical staff and the wounded, environmental sounds and groans and breathing sounds of the wounded. The physiological data acquisition unit preliminarily monitors vital signs through a glasses leg contact point or a micro non-contact sensor, and reads blood pressure, body temperature, heart rate and blood oxygen saturation data through a Bluetooth or NFC wireless access professional portable monitoring device. The environmental perception unit is used for acquiring ambient light intensity and temperature information, and realizes self-adaptive adjustment of the system. The image acquired by the visual acquisition unit is processed, and the processing comprises the following steps:

3. The intelligent triage classification AR glasses system based on multi-modal data according to claim 2, characterized in that, A multi-scale brightness adaptive algorithm based on a human visual system (Retinex) theory is adopted to perform dynamic range compression and local sharpening on the fused image, so that the bright part is not overexposed, and the dark part has details. An NLM (non-local mean) denoising algorithm is adopted to perform edge-preserving denoising processing, an edge detection algorithm based on phase congruency is adopted to perform edge enhancement, essential structure edges of the wound are extracted, and the tearing degree and shape of the wound are judged, and the formula is as follows: ​ ； wherein, is a pixel point to be de-noised, is another pixel point within a search window, and is a pixel point , a gray value of is a filter parameter, is a normalization constant, is a Gaussian weighted Euclidean distance square between pixel blocks centered at p and q is a Gaussian kernel, is a search window, is a de-noised value for image L pixel p;​ Normalization constant The formula is as follows: ； In the segmented wound area, the pixel values of the blood area under RGB and near-infrared NIR are analyzed, and the blood area is estimated by combining the depth information provided by the depth camera to perform a three-level semi-quantitative evaluation of the amount of bleeding, including: oozing, active bleeding and spouting bleeding.

4. The intelligent triage classification AR glasses system based on multi-modal data of claim 1, wherein, The intelligent analysis processing module includes a multi-modal data fusion engine, a special AI analysis model set, and an intelligent triage classification decision model. The multi-modal data fusion engine adopts a feature-level fusion model based on deep learning, realizes multi-modal data time alignment through linear interpolation and dynamic time warping DTW combination, realizes spatial alignment by combining the visual inertial odometry VIO module of the AR glasses with target detection, and then realizes feature-level fusion of each modal feature vector through a cross-modal encoder based on the deep learning Transformer architecture, and outputs a fixed-dimensional global feature vector F. The special AI analysis model set includes a bleeding recognition model, a fracture probability judgment model, a face color recognition model, a respiratory sound or moaning sound recognition model, and an optical character recognition OCR model, which are respectively used to identify the wound site and the amount of bleeding, analyze the probability of limb fracture, judge the face color state of the wounded, identify the respiratory frequency and abnormal respiratory sound, and read the digital information displayed by the medical equipment. The intelligent triage classification decision model adopts a hybrid expert system Hybrid Expert System architecture, combines a rule engine based on the triage classification protocol START / JumpSTART and a deep learning model with a multi-layer perceptron MLP structure, and outputs the triage classification result and confidence through a decision fusion and arbitration module.

5. The intelligent triage classification AR glasses system based on multi-modal data according to claim 4, characterized in that, The confidence calculation formula is as follows: ; wherein, is the final result confidence, is the rule confidence, is the probability value of the corresponding grade output by the deep learning model, is the data quality score, is an adjustable weight parameter.

6. The intelligent triage classification AR glasses system based on multi-modal data according to claim 4, wherein, The bleeding recognition model performs bilateral filtering and adaptive histogram equalization CLAHE preprocessing on the input wound site RGB image, obtains a pixel-level segmentation mask through a semantic segmentation model U-Net++, and outputs the bleeding grade through bleeding area calculation and bleeding amount grading logic, and estimates the depth of the wound area combined with the depth map. The fracture probability judgment model inputs the limb RGB image and depth map, outputs the fracture result through the fracture classification branch of the multi-task learning convolutional neural network MT-CNN, outputs the fracture probability through the fracture probability regression branch, and outputs the joint coordinates through the joint positioning branch. The face color recognition model performs automatic white balance AWB and color space conversion on the input face region of interest ROI image, calculates the mean and standard deviation of the color statistical features, and outputs the face classification result and ROI positioning through the lightweight spiking neural network SNN. The respiratory sound or moaning sound recognition model generates a mel-frequency spectrogram for the preprocessed audio signal, extracts local frequency domain patterns through a 2D convolution layer, and after the pooling layer, on the one hand, outputs the respiratory frequency through spectral peak detection, and on the other hand, captures the time sequence dependence through a gated recurrent unit GRU, and outputs the event classification result through a fully connected layer. The text recognition OCR model uses a CNN-based detector for text detection on the input device screen image. In text recognition, the feature sequence is first extracted through the convolutional layer, and then the decoding sequence is generated using the recurrent layer or Transformer. The character sequence is output through the connectionist temporal classification (CTC) loss or attention mechanism (Attention). Finally, the structured data is output.

7. The intelligent triage classification AR glasses system based on multi-modal data of claim 4, wherein, The rule engine based on the START / JumpSTART injury classification protocol outputs classification suggestions through a hard-coded deterministic decision tree. The deep learning model inputs a global feature vector F and outputs a four-dimensional probability vector. The decision fusion and arbitration module follows the rule priority principle, directly adopts the results determined by the rule engine as red or black, and performs fuzzy arbitration on the yellow or green results combined with the deep learning model prediction results.

8. The intelligent triage classification AR glasses system based on multi-modal data of claim 1, wherein, When the peripheral linkage module controls the printer, it uses Bluetooth low energy connection, and the instruction set uses Zebra Programming Language (ZPL), Compact Printer Control Language (CPCL), or Reduced Printer Control Code (ESC / POS) instruction set. The unique ID format of the printed injury condition identification bracelet or label is event code-date-sequence number, and the two-dimensional code content is a short URL and a unique ID. The top of the label is a prominent color block, the middle is a large font classification level and a unique ID, and the bottom is a two-dimensional code. The command contains a status query command. When the printer is out of paper, the ribbon is used up, or the communication is interrupted, the AR glasses notify the operator through AR interface icon flashing and voice prompt.

9. The intelligent triage classification AR glasses system based on multi-modal data of claim 1, wherein, The data written to the NFC chip by the peripheral linkage module includes text records, uniform resource identifier (URI) records, and custom external records. The text record stores basic injury information in plain text, the URI record stores the same URL as the two-dimensional code, and the custom external record stores the complete JSON format injury data encrypted using the AES-128-GCM algorithm. The AR glasses have a unique device key. The system preloads a root key to derive the device key when the device is activated. Through AR visual prompts and voice guidance, the user is prompted to hold the terminal at a distance of 4 cm for optimal reading and writing. After writing, it immediately reads back for verification. If it fails, it will automatically retry, up to 3 times. If the final writing fails, the operator is prompted to rely on the two-dimensional code.

10. The intelligent triage classification AR glasses system based on multi-modal data of claim 1, wherein, The data storage and transmission module and the peripheral linkage module meet the requirements of data minimization, encryption, access control security and privacy. The two-dimensional code and URI record only contain the ID, the detailed data is accessed through a secure interface and the access is controlled, and the cloud API interface accesses the detailed data through the OAuth 2.0 protocol authentication and records all access logs.

Images