Method and system for evaluating survival state of wounded person on basis of incomplete parameters

Abstract

A method and system for evaluating the survival state of a wounded person on the basis of incomplete parameters. The method comprises: acquiring visual information, audio information and thermal imaging multi-modal data of a target; inputting the multi-modal data into a feature extraction network, so as to obtain multi-modal feature data; and combining the multi-modal feature data with physiological indexes, so as to form a multi-modal data set, and on the basis of a trained survival index model, obtaining an evaluation result for the survival state of the target. The present invention provides a more comprehensive and more accurate evaluation result for the survival state of a wounded person, so as to implement rapid and accurate triage of wounded persons at a disaster site.

Description

A method and system for evaluating the survival status of wounded people based on incomplete parameters Technical Field

[0001] The present invention relates to the field of information processing technology, and more particularly to a method and system for evaluating the survival status of a wounded person based on incomplete parameters. Background Art

[0002] In the field of emergency medical rescue, especially in disaster-prone environments, medical resources are often limited, and complete medical information on the injured is difficult to obtain. In such situations, the use of efficient and adaptable triage methods is crucial. The goal of such methods is to rapidly assess and triage the injured in a short period of time to identify those most in need of emergency medical assistance. Currently, the main triage methods and strategies include Simplified Emergency Triage (START), JumpSTART, color-coding systems, and rapid on-site assessment, which help to quickly determine treatment priorities in emergency situations. However, these triage methods are relatively crude, can only classify a few categories, and the classification results are less precise.

[0003] Currently, some related research is analyzing the survival status of injured people through visual image analysis. Facial feature recognition is the process of analyzing the tissue and structural features of a patient's face to understand their health status. Eyes and lips are among the most important facial features, and their information can reveal a wealth of information about a patient's health and physiological condition. By identifying and analyzing information about eyes and lips, it can help predict patient survival and disease progression, providing a more accurate and reliable method for patient survival prediction. Researchers have combined machine learning and deep learning to efficiently process and analyze large-scale clinical data, discovering the correlation between facial features and survival time, and constructing survival analysis models to provide valuable reference information for clinicians.

[0004] However, in the existing technology, a single clinical physiological dataset is usually used for patient survival analysis, which has the following defects: a single clinical physiological dataset cannot fully reflect the patient's overall condition. If only relying on the physiological dataset, other important predictors may be ignored, resulting in inaccurate analysis results. The generalization ability of the survival analysis results is poor and cannot be applied to other patient groups or situations; the clinical physiological dataset may have problems such as missing values, outliers or erroneous data. Such data quality issues affect the accuracy of the classification results.

[0005] In summary, traditional survival analysis methods for casualties usually rely on a single physiological indicator dataset. However, this single information is sometimes difficult to fully describe the patient's disease status and survival status. In addition, due to the limitations of medical resources and testing capabilities at disaster rescue sites, it is difficult to obtain complete physiological parameters, which brings great difficulties to treatment decisions based on physiological indicators.

[0006] Summary of the Invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method and system for assessing the survival status of the wounded based on incomplete parameters, aiming to solve the problem of wounded triage under the conditions of limited medical resources, limited detection capabilities and limited data at disaster rescue sites such as earthquakes and fires.

[0008] According to a first aspect of the present invention, a method for assessing the survival status of a wounded person based on incomplete parameters is provided. The method comprises the following steps:

[0009] Acquire the target's visual information, audio information, and thermal imaging multimodal data;

[0010] Inputting the multimodal data into a feature extraction network to obtain multimodal feature data;

[0011] The multimodal feature data and physiological indicators are combined to form a multimodal data set, and based on the trained survival index model, an assessment result of the target survival status is obtained.

[0012] According to a second aspect of the present invention, a system for assessing the survival status of a wounded person based on incomplete parameters is provided. The system comprises:

[0013] Data acquisition module: used to obtain the target's visual information, audio information and thermal imaging multimodal data;

[0014] Feature extraction module: used to input the multimodal data into the feature extraction network to obtain multimodal feature data;

[0015] Evaluation module: used to combine the multimodal feature data and physiological indicators to form a multimodal data set, and obtain an evaluation result of the target survival status based on the trained survival index model.

[0016] Compared with the existing technology, the advantage of the present invention is that the proposed scheme for assessing the survival status of the wounded based on incomplete parameters combines the incomplete physiological indicators collected from the disaster rescue site environment, and combines the disaster rescue site environment with the multimodal data characteristics of the wounded's vision, voice, and thermal imaging to fuse and construct a wounded survival index, providing a more comprehensive and accurate assessment result of the survival status of the wounded, thereby realizing rapid and accurate triage and classification of the wounded at the disaster site.

[0017] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0019] FIG1 is a flow chart of a method for assessing the survival status of a wounded person based on incomplete parameters according to an embodiment of the present invention;

[0020] FIG2 is a diagram of a backbone network with deep residual cross connections in a multimodal feature extraction network according to one embodiment of the present invention;

[0021] FIG3 is a schematic diagram of a structure of an enhanced perception long short-term memory network according to an embodiment of the present invention;

[0022] In the figure, Convolutional layer-convolution layer; Batch Normalization layer-batch normalization layer; Rectified Linear layer-rectified linear unit; Global Average Pooling-global average pooling layer; Input layer-input layer; Output layer-output layer; Hidden layer-hidden layer. DETAILED DESCRIPTION

[0023] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0024] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0025] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0026] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0027] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0028] In the present invention, a method for assessing the survival status of the wounded based on incomplete parameters is proposed. A survival index based on a cascade neural network and enhanced perceptual long short-term memory network (MultiCNN-APLSTM) with multimodal features is constructed. A multimodal dataset combining facial visual features, audio features, thermal imaging features, and physiological indicators of patients collected at disaster rescue sites is used to perform triage and classification of the survival status of the wounded. In general, the method includes feature extraction and analysis of the patient's visual, audio, and thermal imaging multimodal data, using a generative adversarial network to balance the incomplete dataset, and predicting and classifying the multimodal dataset based on incomplete parameters. In the facial visual analysis, audio, and thermal imaging analysis model MultiCNN (cascade neural network), a multimodal feature lightweight network (MRegnet) is proposed to present the main feature identifiers of vision, audio, and thermal imaging. In the backbone of the MRegnet network, a residual stem (Rstem) is proposed to pair features to enhance the feature representation of the network. When predicting multimodal data sets with incomplete parameter properties, an enhanced perceptual long short-term memory network (APLSTM) with an adaptive gated weighted adjustment mechanism is proposed as a classification survival analysis model. Then, a survival index model is constructed on the incomplete data set balanced by a generative adversarial network to achieve accurate classification of the wounded.

[0029] In general, the CNN model takes as input the patient's multimodal data analysis of visual, audio, and thermal images, and outputs the results of the multimodal data analysis. Furthermore, the facial visual, audio, and thermal imaging analysis results are combined with clinical physiological indicators to form a multimodal dataset, which is then used by an APLSTM (Enhanced Perceptual Long Short-Term Memory) network for prediction and classification. Furthermore, based on the CNN, a lightweight network architecture, MRegnet, is proposed as a module for extracting multimodal features from visual, audio, and thermal images. A backbone (Rstem) with a residual structure is used as the initial extraction component of MRegnet. Rstem is the initial processing component of the network. The application of the residual structure is proposed to enable smoother training of feature information within the network, effectively alleviating the problems of vanishing gradients and slow convergence, and preventing model overfitting during training. Furthermore, the APLSTM module is proposed as a model for predicting and classifying tabular data. By combining a fully connected network and an activation function with the LSTM in sequence classification tasks, it can effectively handle item dependencies in long input sequences and improve the model's classification performance.

[0030] Specifically, as shown in FIG1 , the provided method for assessing the survival status of a wounded person based on incomplete parameters includes the following steps:

[0031] Step S110: Collect a multimodal dataset.

[0032] Multimodal datasets include visual information, audio, and thermal imaging multimodal data of patients, and can be further combined with physiological data.

[0033] Comprehensive survival analysis based on multimodal patient data offers numerous advantages. The richness of multimodal data, combined with clinical and physiological data, facial visual features, audio, and thermal imaging, provides multiple perspectives, offering a more comprehensive understanding of a patient's health status. Clinical and physiological data provides information on the patient's physiological function, facial visual features provide information on their appearance and expression, thermal imaging reveals their surface temperature distribution, and audio data provides information on their emotions. This richness of multimodal data provides more comprehensive and accurate information, helping to better predict patient survival. By combining this information, we can better understand the patient's physiological status, thereby improving the accuracy of survival analysis.

[0034] Step S120: Design a convolutional neural network to extract and analyze multimodal data features.

[0035] In one embodiment, as shown in Figure 2, a lightweight network structure, MRegnet, is proposed as a module for extracting multimodal feature information from vision, audio, and thermal imaging. Figure 2(a) shows FRegnet, Figure 2(b) shows the main body, Figure 2(c) shows the stage, and Figure 2(d) shows Rstem. A backbone (Rstem) with a residual structure serves as the initial extraction component of MRegnet. The deep residual structure is a special neural network architecture whose fundamental concept is to introduce a cross-level mapping structure into the backbone network, enabling direct transmission of feature information within the network and avoiding information loss after multiple nonlinear transformations. In the Rstem structure, the feature information after convolution operations at one convolutional layer serves as the input of the residual structure. The input information is then split into two parts: one part is connected through cross-level connections, and the other part undergoes deeper feature extraction operations under cross-level connections to obtain the output information. The input and output information are summed to obtain the output information of the residual structure. A global average pooling operation then reduces the output information and average pools the feature values ​​of each channel in the output information to obtain a global feature vector. As a smooth nonlinear activation function, Softplus maps input values ​​to a smooth nonlinear space, which can replace the traditional RELU activation function, improve the network's expressiveness, avoid the gradient vanishing problem of the RELU activation function, and reduce the number of model parameters and computational complexity. After the global average pooling function and the softplus activation function are operated, the model's calculation speed is accelerated, the model's nonlinear transformation ability is effectively increased, and the model's expressiveness and performance are improved. For example, this process can be expressed by the following formula:

[0036] Among them, G i It is the output result of the backbone of the deep residual structure, Q i is the output result of the entire residual structure, Relu and Softplus are activation functions, k (including k ci and k di ) is the convolutional layer, b i is the bias result, k GAPi is the global average pooling operation, x is the information of the input features, and N is the number of convolutional layers.

[0037] Step S130: Using a generative adversarial network to balance the incomplete multimodal dataset.

[0038] To improve the data distribution quality of incomplete datasets, a preferred embodiment uses a generative adversarial network (GAN) to balance incomplete datasets. This approach includes the following steps: First, the incomplete dataset is understood and prepared to understand its category distribution and imbalance, and to determine which categories are minority and which are majority. This will help guide subsequent processing steps. The GAN network consists of a generator and a discriminator. The generator is responsible for generating new samples, while the discriminator is responsible for distinguishing between generated samples and real samples. To balance the dataset, more samples can be generated based on the minority category in the dataset.

[0039] Specifically, the augmentation process for an incomplete dataset involves the following: the generator accepts a random noise vector and converts it into fake data through a neural network. In the case of a one-dimensional dataset, the generator's output is one-dimensional. The discriminator accepts real data or fake data and uses the neural network to determine whether it is real data. The training goal of a GAN is to minimize the loss function, which indicates that the discriminator wants to maximize its ability to distinguish between real and fake data, while the generator wants to minimize the discriminator's ability to identify the fake data it generates. By alternately training the generator and discriminator, the GAN network can learn to generate new data that is similar to the real dataset.

[0040] Step S140: Design an enhanced perception long short-term memory network as a survival index model to evaluate the survival status of the target.

[0041] When building a predictive classification model, an APLSTM module was proposed. The APLSTM module consists of an LSTM, a gating mechanism, an adaptive weighting mechanism, and a residual structure, as shown in Figure 3. The LSTM structure consists of an input gate, a forget gate, an output gate, and an internal state unit, and is designed to process sequential data. The gating mechanism consists of a fully connected layer with a sigmoid activation function, which controls the weight of the output state. The output passes through the gating layer to obtain a gate vector, Gate, which is used to adjust the importance of the output state. The adaptive weighting mechanism is implemented by the attention layer, which performs a weighted summation of the output at each time step. The attention layer consists of a series of fully connected layers, a tanh activation function, and a softmax function. It calculates the attention weight for each time step and multiplies the output element-wise by the attention weight to obtain a weighted output. The residual structure performs nonlinear transformation and dimensionality reduction on the original output information. The weighted output is added to the original output information to form a residual connection to generate the predicted output information.

[0042] Accordingly, the present invention also provides a system for assessing the survival status of casualties based on incomplete parameters, for implementing one or more aspects of the aforementioned method. For example, the system includes: a data acquisition module for acquiring multimodal data from a target, including visual, audio, and thermal imaging; a feature extraction module for inputting this multimodal data into a feature extraction network to obtain multimodal feature data; and an assessment module for combining this multimodal feature data with physiological indicators to form a multimodal dataset, and obtaining an assessment of the target's survival status based on a trained survival index model. Each module can be implemented using a general-purpose processor, a dedicated processor, or an FPGA.

[0043] In summary, compared with the prior art, the present invention has the following advantages:

[0044] 1) This paper proposes a method for assessing the survival status of the wounded based on incomplete parameters. It constructs a survival index based on a cascade neural network and enhanced perceptual long short-term memory network (MultiCNN-APLSTM) with multimodal features. It uses a multimodal dataset that combines facial visual features, audio features, thermal imaging features, and physiological indicators of patients collected at disaster rescue sites to perform injury classification on the survival status of the wounded.

[0045] 2) This paper uses generative adversarial networks to balance incomplete datasets and predict and classify multimodal datasets based on incomplete parameters. Furthermore, in facial visual analysis, audio, and thermal imaging analysis, a multimodal feature lightweight network (MRegnet) is proposed to present the main features of visual, audio, and thermal imaging. Using a deep residual structure as the initial part of the MRegnet structure alleviates the slow convergence and overfitting problems during feature extraction, enhances the network's ability to detect detailed information, improves the model's ability to change nonlinearly, and better extracts polymorphic data features.

[0046] 3) When predicting multimodal datasets with incomplete parameters, this paper proposes an enhanced perceptual long short-term memory network (APLSTM) with an adaptive gated weighted adjustment mechanism as a classification survival analysis model. Based on the incomplete dataset balanced using a generative adversarial network, a survival index model is constructed to accurately classify the injured. When the proposed APLSTM is used as a predictive classification model, the adaptive gated weighted adjustment mechanism allows the prediction model to adjust the weights of sequence information in different dimensions and at different times. This allows the model to easily capture key information and long-term dependencies in the sequence information, improving the model's ability to process high-dimensional sequence feature information.

[0047] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0048] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0049] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0050] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0051] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0052] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0053] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0054] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0055] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for evaluating the survival status of the wounded based on incomplete parameters, comprising the following steps: Obtain visual information, audio information, and thermal imaging multimodal data of the target; Input the multimodal data into a feature extraction network to obtain multimodal feature data; Combine the multimodal feature data with physiological indicators to form a multimodal data set, and based on a trained survival index model, obtain an evaluation result of the target survival status.

2. The method according to claim 1, wherein, The feature extraction module includes a backbone structure with a residual structure, a global average pooling layer, and an activation layer. In the backbone structure, the feature information after convolution operation of one convolutional layer is used as the input information of the residual structure. After respectively performing cross-level connection and extracting deep features under the cross-level connection, the output information of the residual structure is obtained; the global average pooling layer is used to reduce the output information and perform average pooling on the feature values of each channel in the output information to obtain a global feature vector; The activation layer uses a Softplus non-linear activation function to map the input value to a non-linear space.

3. The method according to claim 2, wherein, The feature extraction module is represented as: Among them, G i is the output result of the backbone structure of the residual structure, Q i is the output result of the entire residual structure, Relu and Softplus are activation functions, k ci and k di are the weights of the convolutional layer, b i is the bias result, k GAPi is the global average pooling operation, x is the information of the input feature, x i is the feature of convolutional layer i, and N is the number of convolutional layers.

4. The method according to claim 1, wherein, The survival index model is an enhanced perception long short-term memory network, including an LSTM module, a gating mechanism module, an adaptive weighting mechanism module, and a residual structure. The LSTM module is used to process sequential data, and the gating mechanism module is used to control the weight of the output state; the adaptive weighting mechanism module includes an attention layer, which is used to perform weighted summation on the output of each time step. The attention layer is used to calculate the attention weight of each time step, multiply the output element by element with the attention weight to obtain a weighted output; the residual structure performs non-linear transformation and dimensionality reduction operations on the original output information, and realizes residual connection by adding the weighted output and the original output information to generate predicted output information.

5. The method according to claim 1, wherein, The multimodal data set for training the survival index model is obtained by augmenting the original multimodal data set using a generative adversarial network. The generative adversarial network includes a generator and a discriminator. During the augmentation process, the generator receives a random noise vector and converts it into fake data through a first neural network. The discriminator receives real data or fake data and judges whether it is real data through a second neural network. The training objective of the generative adversarial network is to minimize a set loss function, which represents that the discriminator hopes to maximize the ability to identify real data and fake data, while the generator hopes to minimize the ability of the discriminator to identify the fake data it generates.

6. The method according to claim 4, wherein, The gating mechanism module includes a fully connected layer and uses a sigmoid activation function.

7. The method according to claim 4, wherein, The attention layer includes a series of fully connected layers, a tanh activation function, and a softmax.

8. A system for evaluating the survival status of the wounded based on incomplete parameters, comprising: Data acquisition module: used to obtain visual information, audio information and thermal imaging multimodal data of the target; Feature extraction module: used to input the multimodal data into a feature extraction network to obtain multimodal feature data; Evaluation module: used to combine the multimodal feature data with physiological indicators to form a multimodal data set, and based on a trained survival index model, obtain an evaluation result of the target survival state.

9. A computer-readable storage medium, on which a computer program is stored, wherein, when the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer device, including a memory and a processor, and a computer program capable of running on the processor is stored on the memory, characterized in that, when the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images