Patient crisis state monitoring and early warning system for a radiotherapy procedure

Abstract

The application relates to the technical field of image segmentation, in particular to a patient crisis state monitoring and early warning system for a radiotherapy process, which comprises an image information acquisition device, video information of a patient's face is acquired, video frames with a definition lower than a preset threshold are removed from the video information based on fast Fourier transform to obtain an image group of the patient's face; the application realizes high-precision and robust identification of the facial feature contour of the patient under the radiotherapy environment by fusing visible light video and three-dimensional point cloud information and utilizing frame screening based on fast Fourier transform, depth information mask generation and a double-path depth feature extraction network.

Description

Technical Field

[0001] This application relates to the field of radiotherapy technology, and more specifically, to a patient crisis monitoring and early warning system for use in the process of radiotherapy. Background Technology

[0002] The content in this section provides only background information related to this application and may not constitute prior art.

[0003] During radiotherapy, to ensure high-precision irradiation, patients are typically placed within a specially designed, personalized body membrane (or vacuum pad, thermoplastic membrane). This membrane closely conforms to the contours of the patient's chest and abdomen, and its main function is to minimize chest and abdominal displacement caused by breathing or unconscious movement during treatment. Through this physical constraint, the membrane effectively reduces the deviation of the treatment target area (such as a tumor) from the planned location, thereby improving the spatial accuracy and treatment efficiency of high-energy radiation (such as X-rays, proton beams).

[0004] For cancer patients with weakened physical conditions (such as those in advanced stages, elderly patients, those with underlying lung disease, or those with low performance status), immobilization with a body membrane may pose a risk of respiratory restriction. The pressure of the body membrane on the chest and abdomen, especially when maintained in a fixed position for extended periods (usually 10 minutes or longer), can cause chest tightness, difficulty breathing, and in severe cases, even life-threatening conditions such as suffocation. Currently, in clinical practice, identifying such risks primarily relies on patients actively calling for help. Healthcare workers are informed of discomfort through communication systems outside the treatment room (such as intercoms) or by the patient pressing the emergency call button.

[0005] Some patients (especially those who are extremely weak, confused, or sedated) may completely lose the ability to call for help due to exhaustion, slow reaction, or communication difficulties. During radiation therapy, patients may be unable to clearly express themselves or find the call device in time due to panic or increased discomfort. Radiation therapy is usually a lengthy process, and the treatment room is noisy and oppressive, making it difficult for medical staff to continuously and directly observe the patient's subtle facial expressions or physical reactions. This passive reliance on the patient's cries for help can easily lead to delayed detection or even complete omission of critical conditions such as respiratory distress when the patient is unable to effectively signal for help. Summary of the Invention

[0006] The summary section of this application is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this application propose a patient crisis monitoring and early warning system for radiotherapy procedures to address the technical problems mentioned in the background section above.

[0008] As a first aspect of this application, some embodiments of this application provide a patient critical condition monitoring and early warning system for radiotherapy procedures, including:

[0009] Includes the following steps:

[0010] The image information acquisition device acquires video information of the patient's face, and filters the video frames in the video information based on fast Fourier transform to remove video frames with a clarity lower than a preset threshold to obtain an image group of the patient's face.

[0011] A facial 3D coordinate acquisition device acquires 3D point cloud information of the patient's face, fuses the 3D point cloud information into the corresponding image group, and generates a point cloud fused image group of the patient's face.

[0012] The facial depth information generation device calculates the depth difference between each pixel in each image of the point cloud fusion image group and the other pixels in the preset window, generates a structured difference value, normalizes the structured difference value, and generates a depth information mask.

[0013] An emotion recognition device extracts video frames from the original video information in a point cloud fused image group and simultaneously inputs the video frames and depth information masks into a depth image feature extraction network.

[0014] The deep image feature extraction network extracts initial feature maps and hierarchical fusion feature maps from video frames and depth information masks. It then fuses the initial feature maps and hierarchical fusion feature maps and performs semantic segmentation to obtain weighted feature maps related to the patient's facial contours.

[0015] The weighted feature map is encoded and decoded to generate a patient's mood change score. When the mood change score exceeds a preset threshold, an alert is issued.

[0016] The emotional change score was positively correlated with the similarity to fear-related expressions.

[0017] This system constructs a depth feature map that integrates spatial and textural information by non-contact, real-time analysis of patient facial video information and combining it with 3D point cloud depth data. It then utilizes deep neural networks to accurately identify subtle emotional changes highly correlated with crisis states such as fear and suffocation. It proactively and objectively captures abnormal pain or panic signals on a patient's face when they are unable to effectively call for help due to weakness, confusion, or environmental constraints, generating a quantified emotional change score and triggering an alert. This overcomes the lag and passivity of traditional methods that rely on patients actively calling for help, significantly shortening the response time for medical staff to sudden respiratory distress and other emergencies during treatment. It effectively reduces the risk of serious complications such as asphyxiation in high-risk patients during prolonged, high-constraint radiotherapy, thereby improving the safety and controllability of the treatment process.

[0018] The image information acquisition device includes:

[0019] Fourier Transform Module: Acquires video information of the patient's face, performs Fast Fourier Transform on each video frame, and filters it to obtain the energy of the high-frequency components of each video frame.

[0020] Image filtering module: A pre-set sharpness threshold is used to delete video frame images in which the energy of high-frequency components in the video frame is lower than the preset sharpness threshold, so as to filter out the image group of the patient's face;

[0021] All video frames in the image group are arranged in chronological order.

[0022] This application uses Fast Fourier Transform (FFT) to quantify and analyze the high-frequency component energy of video frames, and performs dynamic frame filtering based on a preset sharpness threshold, effectively eliminating low-quality images caused by motion blur, defocus, or insufficient lighting. This preprocessing mechanism improves the overall sharpness of the input image group.

[0023] The facial 3D coordinate acquisition device includes:

[0024] Temporal alignment module: Acquires 3D point cloud information of the patient's face and aligns the 3D point cloud information with the video frames in the image group in the temporal dimension;

[0025] Coordinate calibration module: Obtains the 3D coordinates of each pixel in the 3D point cloud information and re-establishes a new 3D coordinate system with the horizontal plane as the reference plane;

[0026] Coordinate Update Module: Changes the original 3D coordinates of the 3D point cloud information to coordinates in the new 3D coordinate system.

[0027] This application ensures synchronization between 3D point clouds and video frames through precise time alignment and innovatively reconstructs a coordinate system based on a horizontal plane, converting the raw point cloud data into standardized depth information. This process effectively eliminates depth measurement biases caused by patient positioning (such as changes in supine angle) or differences in device perspective, laying a solid foundation for the subsequent generation of high-precision, spatially consistent depth information masks. It improves the detection sensitivity and robustness of subtle facial expressions (especially features related to respiratory distress and fear) and depth changes, thereby ensuring the quality and reliability of input data for the emotion recognition model.

[0028] The facial depth information generation device includes:

[0029] Depth information acquisition module: Extracts the depth coordinates of 3D point cloud information, generates a depth point cloud, and the pixel value of each pixel in the depth point cloud is the depth coordinate;

[0030] Image focusing module: Extracts the initial facial contour from the 2D image based on the difference information of the depth coordinates;

[0031] Image preprocessing module: Normalizes the depth values ​​of the initial facial contour to generate a depth information mask.

[0032] This application generates a depth point cloud map by directly extracting the depth coordinates of a 3D point cloud, and uses depth difference information to accurately focus on the facial contour region. After normalization, a standardized depth information mask is generated. This process effectively removes irrelevant background interference, enhances the contrast and recognizability of subtle depth changes in key facial areas (such as around the eyes, nose, and mouth), and provides high-purity, high-dynamic-range spatial structural feature input for subsequent emotion recognition models. This significantly improves the model's detection sensitivity and quantification accuracy for extremely subtle facial muscle contractions and skin deformations (such as nasal dilation, furrowed brows, and taut corners of the mouth) caused by breathing difficulties or panic.

[0033] The image focusing module uses the largest region in the depth point cloud where the pixel difference is less than a preset threshold as the initial facial contour.

[0034] Furthermore, the deep image feature extraction network includes:

[0035] The feature extraction module is used to extract initial feature maps and hierarchical fused feature maps from input video frames and depth information masks.

[0036] Features are fused into modules, and the initial feature map and the hierarchical fusion feature map are fused into a weighted feature map based on the self-attention mechanism;

[0037] The encoder compiles weighted feature maps into a one-dimensional sequence based on emotion influence factors;

[0038] The Transformer decoder generates a patient's mood change score through an autoregressive approach.

[0039] The alarm module monitors whether the patient's emotional change score exceeds a threshold and issues an alarm when the threshold is exceeded.

[0040] This application constructs a cross-frame facial contour dataset and first extracts the scale-invariant feature transform (SIFT) key points of each contour. Then, it calculates the overlap between contours based on the geometric positional relationship of similar feature points. This approach breaks through the limitations of traditional single-frame intra-contour redundant comparison, reducing computational complexity. At the same time, by utilizing the rotation and scale invariance of SIFT features, combined with spatial positional constraints, it achieves efficient and accurate contour matching, significantly improving the screening efficiency of large-scale facial contour sets while ensuring deduplication accuracy.

[0041] The feature extraction module includes:

[0042] The input layer is used to input video frames and depth information masks.

[0043] A convolutional network consists of five convolutional layers, with the number of convolutional kernels gradually increasing from top to bottom.

[0044] The cross network, consisting of five convolutional layers, corresponds to the convolutional network. It takes the element-wise multiplication of the current convolutional layer and the previous cross layer as input to output a hierarchical fused feature map.

[0045] The input layer is a 1*1 convolutional network, and the video frame and depth information mask are input separately to obtain the initial feature map and the mask feature map.

[0046] The initial feature map is input into the convolutional network, and the mask feature map is input into the cross network;

[0047] The initial layer of a convolutional network corresponds to the bottom layer of a cross-network.

[0048] This network precisely focuses on subtle dynamic facial changes strongly correlated with emotions (especially fear and feelings of suffocation) through multimodal (video texture + depth mask) feature extraction and hierarchical feature fusion based on a self-attention mechanism. Leveraging the powerful temporal modeling and contextual understanding capabilities of the Transformer architecture, it compiles the fused high-dimensional spatial features into a one-dimensional sequence reflecting the patient's real-time emotional state changes. Finally, it generates a quantified emotion change score through autoregressive decoding. This design improves the system's ability to capture and predict subtle, progressive emotional fluctuations (such as escalating panic due to breathing difficulties), enabling early, objective, and automated identification and warning of patient crisis states (such as near-suffocation).

[0049] Furthermore, the feature fusion module concatenates the initial feature map and the hierarchical fusion feature map along the feature dimension, generates attention scores based on the convolutional network, generates joint weights based on the attention scores, and fuses the initial feature map and the hierarchical fusion feature map according to the attention weights.

[0050] Furthermore, the encoder includes:

[0051] The information compilation block compiles the weighted feature map into a two-dimensional image matrix;

[0052] The positional encoding block adds positional encoding to the two-dimensional image matrix;

[0053] The attention module simultaneously receives the positional encoding and the two-dimensional image matrix to generate image features;

[0054] Feedforward neural networks nonlinearly transform image features to generate one-dimensional sequences.

[0055] This encoder compiles the fused weighted feature maps into a two-dimensional image matrix and incorporates positional encoding to accurately preserve the spatial structural information of key facial regions. Then, an attention mechanism dynamically focuses on the local features most relevant to emotional expression (such as subtle changes in the muscles around the eyes and nose), and a feedforward neural network performs nonlinear transformation and compression to generate a one-dimensional sequence rich in spatiotemporal semantic information. This design effectively solves the problems of spatial information loss and key feature weakening during the conversion of high-dimensional spatial features to temporal sequences. It provides a highly abstract, structured, and discriminative input for the subsequent Transformer decoder to accurately model the dynamic evolution and contextual dependencies of emotional states, improving the accuracy and robustness of emotional change score generation.

[0056] Furthermore, the Transformer decoder includes:

[0057] A standard Transformer decoding block for decoding one-dimensional sequences;

[0058] The temporal memory matrix block, based on multi-head attention and a feedforward neural network, guides the standard Transformer decoding block so that when decoding a one-dimensional sequence, it continuously regresses the one-dimensional sequence information of the previous time step to capture the continuous changes in facial expressions.

[0059] This decoder processes the one-dimensional sequence output by the encoder using a standard Transformer decoder block and introduces a temporal memory matrix block mechanism. This mechanism utilizes multi-head attention and a feedforward neural network to force the model to continuously regress and fuse sequence information from the previous time step while decoding the current time step. This design endows the decoder with powerful temporal memory and state inheritance capabilities, enabling it to accurately model the subtle, continuous, and gradual dynamic evolution of a patient's facial expressions during treatment (such as the cumulative process of fear from mild tension to intense panic), effectively capturing the trend of facial expression changes caused by breathing restriction. This improves the accuracy, temporal coherence, and early warning capabilities of the generated emotion change score in representing real, complex, and long-term emotional state fluctuations. Attached Figure Description

[0060] Figure 1 This is a flowchart of a patient crisis monitoring and early warning system used in radiotherapy procedures.

[0061] Figure 2 This is a flowchart of a depth image feature extraction network.

[0062] Figure 3 This is a flowchart of the feature extraction module.

[0063] Figure 4 This is a flowchart of the Transformer decoder. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments. The same reference numerals in the accompanying drawings represent the same components. It should be noted that the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the described embodiments of this application without creative effort are within the scope of protection of this application.

[0065] Compared to the embodiments shown in the accompanying drawings, feasible embodiments within the scope of this application may have fewer components, other components not shown in the drawings, different components, differently arranged components, or components with different connections, etc. Furthermore, two or more components in the drawings may be implemented in a single component, or a single component shown in the drawings may be implemented as multiple separate components.

[0066] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains. The terms “first,” “second,” and similar terms used in this specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not necessarily indicate a quantity limitation. Terms such as “upper” and “lower” are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes.

[0067] refer to Figure 1 Example 1: A patient crisis monitoring and early warning system for radiotherapy, comprising: an image information acquisition device, a facial three-dimensional coordinate acquisition device, a facial depth information generation device, an emotion recognition device, and an emotion recognition device.

[0068] The image information acquisition device acquires video information of the patient's face, and filters the video frames in the video information based on fast Fourier transform to remove video frames with a resolution lower than a preset threshold, thereby obtaining a group of images of the patient's face.

[0069] The image information acquisition device includes:

[0070] Fourier Transform Module: Acquires video information of the patient's face, performs Fast Fourier Transform on each video frame, and filters it to obtain the energy of the high-frequency components of each video frame.

[0071] Fourier transform is an existing technique, and the specific execution process will not be described here. The general approach is as follows: convert the color image of the video frame to grayscale, apply 2D FFT, extract high-frequency components, and calculate their energy. The energy is defined as the sum of the squares of the amplitudes of the high-frequency components, reflecting the intensity of details (such as edges and textures) in the image.

[0072] Image filtering module: A pre-set sharpness threshold is used to delete video frame images in which the energy of high-frequency components in the video frame is lower than the preset sharpness threshold, so as to filter out the image group of the patient's face;

[0073] All video frames in the image group are arranged in chronological order.

[0074] For example, after acquiring video information, the video frame sequence is evaluated one by one. Frame 3 (where the patient's rapid head turn causes slight motion blur) has a calculated high-frequency energy of 35, frame 7 has a high-frequency energy of 28, and frame 15 has a high-frequency energy of 42. Since the high-frequency energy of these frames is below the preset sharpness threshold of 50, the system classifies them as low-quality images and deletes them. Ultimately, only clear video frames with a high-frequency energy of 50 or higher (such as frames 1, 2, 4, 5, 6, 8...) are retained and arranged strictly according to their original timestamp order to form an "image group of the patient's face" for subsequent processing.

[0075] The facial 3D coordinate acquisition device acquires the 3D point cloud information of the patient's face, fuses the 3D point cloud information into the corresponding image group, and generates a point cloud fused image group of the patient's face.

[0076] Three-dimensional point cloud information is obtained by projecting structured light onto the surface of an object using a TOF sensor and capturing the deformation of the structured light on the surface using a dedicated sensor. The position of each point on the surface of the object in three-dimensional coordinates is then calculated, thus obtaining three-dimensional point cloud information.

[0077] Correspondingly, the TOF sensor and the camera capturing the video frames completely overlap in their fields of view. By overlaying the video frames captured at the same time point with the 3D point cloud information, the coordinates of each pixel in the video frame in 3D space can be determined. Correspondingly, the color channel information of each pixel in the 3D point cloud information can be obtained using the video frames.

[0078] The facial 3D coordinate acquisition device includes: a time alignment module, a coordinate calibration module, and a coordinate update module.

[0079] Temporal alignment module: Acquires 3D point cloud information of the patient's face and decomposes the 3D point cloud information into 3D image frames that are aligned with the video frames in the image group in the temporal dimension.

[0080] Specifically, video information captured at the same time point is paired with 3D point cloud information to form a 3D image frame, which is the 3D point cloud at the corresponding time point.

[0081] The image information acquisition device has reduced the video information. Therefore, when mapping the video information to the three-dimensional point cloud information, the image group of the patient's face that has been reduced is used to bind the three-dimensional point cloud information to the frame number in the time dimension.

[0082] Coordinate calibration module: Obtain the 3D coordinates of each pixel in the 3D image frame relative to the point cloud sensor, and re-establish a new 3D coordinate system directly below the point cloud sensor.

[0083] Generally, cameras and point cloud sensors are difficult to place directly above a patient's face (they need to avoid the trajectory of existing radiotherapy equipment, and placing them directly above the patient's face would result in obstruction by the equipment). They can only be placed diagonally above the patient's face to capture facial expressions from an angle. Therefore, the recorded 3D position uses the "point cloud sensor's own position" as the spatial zero point. However, this is not natural for observing a human face. Therefore, the system sets a new spatial zero point directly below the camera (e.g., a point on the floor of the passageway) and establishes a new coordinate system based on this point. The vertical direction (Z-axis) of this new coordinate system is perpendicular to the ground (positive upwards), while the horizontal and vertical directions (X and Y axes) are parallel to the ground.

[0084] Coordinate update module: Changes the original 3D coordinates in the 3D image frame to coordinates in the new 3D coordinate system, and combines all the 3D image frames with changed coordinates into a point cloud fusion image group.

[0085] In this way, the depth data obtained from the original tilted perspective and based on the camera is converted into height data from the more familiar "eye-level" perspective based on the ground. The converted depth information (height value) can more clearly and directly reflect facial features (such as the protrusion of the bridge of the nose and the depression of the eye sockets).

[0086] The coordinate transformation method in this solution is existing technology and will not be discussed further here.

[0087] The facial depth information generation device calculates the depth difference between each pixel in each image of the point cloud fusion image group and the other pixels in the preset window, generates a structured difference value, normalizes the structured difference value, and generates a depth information mask.

[0088] The facial depth information generation device includes: a depth information acquisition module, an image focusing module, and an image preprocessing module.

[0089] Depth information acquisition module: Extracts the depth coordinates of 3D point cloud information and generates a depth point cloud, where the pixel value of each pixel in the depth point cloud is the depth coordinate.

[0090] The depth information acquisition module extracts the depth information from the 3D point cloud information, namely "how far each point is from the ground to a certain preset plane", and thus obtains a depth map.

[0091] Image focusing module: Extracts the initial facial contour from a two-dimensional image based on the difference information of depth coordinates.

[0092] For example, a depth threshold (10cm) can be set, and then every pixel in the image can be scanned to find the largest continuous region where the depth threshold difference does not exceed 10cm. This can essentially determine the location of the face. Of course, in practice, the field of view acquired by the camera and time alignment module needs to be adjusted to reduce irrelevant areas.

[0093] Image preprocessing module: Normalizes the depth values ​​of the initial facial contour to generate a depth information mask.

[0094] Normalization is a current technology, and the specific normalization process will not be discussed here. The depth information mask provides depth information for the patient's facial region. Although it does not define the patient's facial contours, the changes in these depth areas can accurately reflect information such as facial contours and facial features.

[0095] The emotion recognition device extracts video frames from the original video information in the point cloud fused image group, and simultaneously inputs the video frames and depth information masks into the depth image feature extraction network.

[0096] The deep image feature extraction network extracts initial feature maps and hierarchical fusion feature maps from video frames and depth information masks. It then fuses the initial feature maps and hierarchical fusion feature maps and performs semantic segmentation to obtain weighted feature maps related to the patient's facial contours.

[0097] The weighted feature map is encoded and decoded to generate a patient's emotional change score;

[0098] When the emotion change score exceeds a preset threshold, an alert is issued;

[0099] The emotional change score was positively correlated with the similarity to fear-related expressions.

[0100] The deep image feature extraction network consists of two parts: one part is used to obtain a mood change score, and the other part is used to issue an alert based on the mood change score.

[0101] Therefore, the key to deep image feature extraction networks lies in how to identify the patient's emotions and whether those emotions contain elements of fear. If the emotional changes are drastic and clearly evolve towards fear and suffocation, it indicates that the patient is at greater risk. Conversely, if the patient's expression is calm and there is no struggling, fear, or panic, it indicates that the patient's current condition is relatively good.

[0102] Deep image feature extraction networks include:

[0103] The feature extraction module is used to extract initial feature maps and hierarchical fused feature maps from input video frames and depth information masks.

[0104] Features are fused into modules, and the initial feature map and the hierarchical fusion feature map are fused into a weighted feature map based on the self-attention mechanism;

[0105] The encoder compiles weighted feature maps into a one-dimensional sequence based on emotion influence factors;

[0106] The Transformer decoder generates a patient's mood change score through an autoregressive approach.

[0107] The alarm module monitors whether the patient's emotional change score exceeds a threshold and issues an alarm when the threshold is exceeded.

[0108] When obtaining emotion change scores, deep image feature extraction networks can be roughly divided into feature extraction, feature fusion, feature encoding, and feature decoding.

[0109] Feature extraction and feature fusion mainly involve using convolutional networks to highlight key features, which primarily refer to the patient's facial contours and facial features. After obtaining the weighted feature map, if the weighted feature map is input into a specific output layer and trained with the goal of obtaining more accurate facial contours, it is possible to obtain the patient's facial contours and facial features.

[0110] However, simply extracting and fusing features is insufficient to capture a patient's emotional changes. More specifically, emotions are difficult to judge simply by similarity. Each person's facial features distort differently when expressing suffocation or panic, and these features are also unique to each individual. Simply comparing a patient's current facial contour with samples of panic and suffocation in a database for training will fail to produce a stable neural network model because there are no directly related features between the two.

[0111] To this end, this application provides an encoder and a decoder. The encoder can further compress the weighted feature map, and during the compression process, it introduces positional encoding related to emotions such as fear and suffocation into the facial contours and features through positional encoding.

[0112] After the encoder compresses the information, the Transformer decoder can quickly process multiple information inputs. Thus, with the help of the time memory matrix block, it can continuously regress the one-dimensional sequence information from the previous time step when decoding a one-dimensional sequence, in order to capture the continuous changes in facial expressions.

[0113] Therefore, to address the issue of accurately recognizing expressions of fear and suffocation, continuous and dynamic facial monitoring can be used to capture the trends and directions of facial expression changes, thereby enabling a more accurate assessment of the patient's current emotional score.

[0114] like Figure 3As shown, Figure 3 The network structure of the feature extraction module is given: The feature extraction module includes:

[0115] The input layer is used to input video frames and depth information masks.

[0116] The input layer is a 1*1 convolutional network. The video frame and the depth information mask are input separately to obtain the initial feature map and the mask feature map. The initial feature map is input to the convolutional network, and the mask feature map is input to the cross network.

[0117] The input layer is a 1x1 convolutional network structure. It performs initial convolutions on video frames and a depth information mask to concentrate key information. After convolution in the input layer, the video frames are fed into the convolutional network, and the depth information mask is fed into the cross-network after convolution in the input layer.

[0118] In some embodiments, the output of the input layer needs to undergo max pooling before being input into the corresponding cross-network and convolutional network. In practice, this can be configured according to the model's accuracy requirements. The key to feature extraction in this application lies in the simultaneous fusion extraction using the cross-network and convolutional network, and the method of repeatedly applying the original feature map.

[0119] Specifically:

[0120] The convolutional network consists of five convolutional layers, with the number of convolutional kernels gradually increasing from top to bottom. The cross network consists of five convolutional layers, corresponding to the convolutional network. It takes the element-wise multiplication of the current convolutional layer and the previous cross layer as input to output a hierarchical fused feature map. The initial layer of the convolutional network corresponds to the bottom layer of the cross network.

[0121] Convolutional networks and cross networks are downsampling networks with the same structure; each layer of a convolutional network and a cross network is a convolutional layer.

[0122] In this scheme, video frames are processed into initial feature maps by the input layer, and then the initial feature maps are input to the five convolutional layers of the convolutional network. The output of each layer of the convolutional network is C1, C2, C3, C4, and C5. From C1 to C5, the semantic information becomes richer, but the detailed information becomes less.

[0123] Correspondingly, the mask feature map is input to the five convolutional layers of the cross network, and the output of the cross network is F1, F2, F3, F4, and F5. From F1 to F5, the semantic information becomes richer, but the detailed information becomes less.

[0124] Therefore, the initial feature map is input into the convolutional network, and then C1 is multiplied element-wise by F4 of the cross network and input into the last layer of the cross network to obtain F5;

[0125] The reason for this approach is primarily to balance the semantic information of the initial feature map and the mask feature map. The initial feature map has less semantic information but higher resolution. Therefore, inputting it into F5 maximizes its high resolution advantage, allowing the initial feature map with the highest resolution to match the F4 feature map with the richest semantic meaning, thus obtaining a weighted feature map with both superior semantic information and resolution.

[0126] Similarly, C5, as the video image with the lowest resolution and richest semantics output by the convolutional network, can correspond to the initial mask feature map. It can directly correspond the natural semantic information (texture and depth information) contained in the mask feature map to the compressed semantic information.

[0127] Thus, this approach of cross-referencing greatly increases the accuracy of information fusion in the weighted feature maps.

[0128] The feature fusion module concatenates the initial feature map and the hierarchical fusion feature map along the feature dimension, generates attention scores based on the convolutional network, generates joint weights based on the attention scores, and fuses the initial feature map and the hierarchical fusion feature map according to the attention weights.

[0129] The feature fusion module is an existing self-attention fusion module. The main method is to concatenate two feature maps to obtain attention scores, then obtain joint weights, and finally fuse them based on the joint weights.

[0130] The specific structure of feature fusion into modules is existing technology and will not be described further here.

[0131] The key to this solution lies in the weighted fusion of the weighted feature map and the initial feature map, for the following reason:

[0132] The initial feature map has high resolution and little semantic information, so it can be used as a query for the weighted feature map during the fusion process, realizing adaptive fusion and enhancement at the feature level, thereby revealing the inherent relationship of the input data and strengthening the expression of key features.

[0133] The encoder includes:

[0134] The information compilation block compiles the weighted feature map into a two-dimensional image matrix;

[0135] The positional encoding block adds positional encoding to the two-dimensional image matrix;

[0136] The attention module simultaneously receives the positional encoding and the two-dimensional image matrix to generate image features;

[0137] Feedforward neural networks nonlinearly transform image features to generate one-dimensional sequences.

[0138] The Transformer decoder includes:

[0139] A standard Transformer decoding block for decoding one-dimensional sequences;

[0140] The temporal memory matrix block, based on multi-head attention and a feedforward neural network, guides the standard Transformer decoding block so that when decoding a one-dimensional sequence, it continuously regresses the one-dimensional sequence information of the previous time step to capture the continuous changes in facial expressions.

[0141] The encoding blocks and Transformer decoders in this scheme are standard Transformer encoding blocks and standard Transformer decoders.

[0142] The key to its encoder lies in the design of positional encoding, while the design of the Transformer decoder lies in the temporal memory matrix block.

[0143] Specifically: The key to positional encoding is to enhance facial expression information. The positional encoding block has a built-in two-dimensional matrix that corresponds to the two-dimensional image matrix.

[0144] The elements in this two-dimensional matrix consist of two parts: a basic part representing location and a part enhancing expression. For example, the position of the mouth is enhanced using the location-representing part to facilitate its determination. The expression-enhancing part, based on the upward curve of the mouth, determines that it is a happy expression, thus amplifying the happy emotion. This is the principle of location-encoded blocks:

[0145] Specifically: the location coding block includes:

[0146] A context extractor is used to extract contextual information from a two-dimensional image matrix to obtain the types of facial expressions.

[0147] The context extractor includes convolutional layers and global pooling layers. The convolutional layers and global pooling layers sequentially convolve and pool the two-dimensional image matrix, compressing this information into a compact representation of context information, which can provide a general idea of ​​the types of expressions.

[0148] The memory stores a position encoding matrix and multiple expression encoding matrices; the expression encoding matrices include at least a happiness encoding matrix, a sadness encoding matrix, a fear encoding matrix, and a suffocation encoding matrix;

[0149] Feature enhancer: Selects the corresponding expression coding matrix according to the representation type, and enhances the expression coding matrix and position coding matrix into the two-dimensional image matrix.

[0150] ;

[0151] ;

[0152] ;

[0153] in, For the output of the feature enhancer, Let i be the eigenvalue of the i-th element in the two-dimensional image matrix. This is the compensation value of the feature enhancer for the i-th element in the two-dimensional image matrix. This represents the position compensation for the i-th element, used to compensate for facial position information. This represents the facial expression compensation for the i-th element, used to compensate for facial expression information. This represents the facial expression compensation weight at the i-th position. Represents the context vector. The feature vector at position i. This represents the learnable weight matrix. This represents a learnable weight vector. Represents the matrix transpose symbol. It is the sigmoid activation function. This represents the transpose of the learnable weight vector. This represents the hyperbolic tangent activation function.

[0154] Thus, this application uses positional encoding blocks to enhance facial expression information, enabling the network model to more accurately extract the positions of facial features and the corresponding emotional perception.

[0155] The structure of the standard Transformer decoding block will not be elaborated further here; for example, ... Figure 4 As shown, there are two normalization parts in the standard Transformer decoding block, and the time memory matrix block needs to process these two normalization parts.

[0156] like Figure 4 As shown, the output of the Transformer decoding block is actually the patient's facial expression type. The closer the patient's facial expression type is to panic and the faster the rate of emotional change, the higher the emotional change score.

[0157] Specifically, the processing procedure for the time memory matrix is ​​as follows:

[0158] S1: Obtain the output of the temporal memory matrix of the previous time step, and then concatenate it with the output of the current encoder.

[0159] S2: Perform multi-head attention feature fusion with the feature concatenation result and the output of the current encoder;

[0160] S3: Perform residual concatenation and normalization on the fusion result and the output of the current encoder;

[0161] S4: Perform feedforward neural network processing;

[0162] S5: Perform residual connection and normalization to obtain the current time step time memory matrix, and input the time step time memory matrix into the residual connection and normalization steps of the Transformer decoding block respectively.

[0163] The above describes the specific structure of a deep image feature extraction network;

[0164] The training process of the deep image feature extraction network is as follows: 500,000 labeled face images are obtained as the training set, and samples generated by 10,000 patients during radiotherapy are used as training data. End-to-end supervised training is performed using a loss function.

[0165] The loss function LOSS includes the loss of the feature extraction module, the loss of the positional encoding block in the encoder, and the loss of the Transformer decoder.

[0166] The loss function for the three parts can be either the cross-entropy loss function or other loss functions. The reason for choosing these three parts is that the network parameters within these three parts need to be learned and adjusted, while the internal parameters of multi-head attention are already fixed. The cross-entropy loss function is a current technology, and its specific expression will not be elaborated further here.

[0167] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A patient crisis monitoring and early warning system for radiotherapy procedures, characterized in that, Includes the following steps: The image information acquisition device acquires video information of the patient's face, and filters the video frames in the video information based on fast Fourier transform to remove video frames with a clarity lower than a preset threshold to obtain an image group of the patient's face. A facial 3D coordinate acquisition device acquires 3D point cloud information of the patient's face, fuses the 3D point cloud information into the corresponding image group, and generates a point cloud fused image group of the patient's face. The facial depth information generation device calculates the depth difference between each pixel in each image of the point cloud fusion image group and the other pixels in the preset window, generates a structured difference value, normalizes the structured difference value, and generates a depth information mask. An emotion recognition device extracts video frames from the original video information in a point cloud fused image group and simultaneously inputs the video frames and depth information masks into a depth image feature extraction network. The deep image feature extraction network extracts initial feature maps and hierarchical fusion feature maps from video frames and depth information masks. It then fuses the initial feature maps and hierarchical fusion feature maps and performs semantic segmentation to obtain weighted feature maps related to the patient's facial contours. The weighted feature map is encoded and decoded to generate a patient's emotional change score; When the emotion change score exceeds a preset threshold, an alert is issued; The emotional change score was positively correlated with the similarity to fear-related expressions; Deep image feature extraction networks include: The feature extraction module is used to extract initial feature maps and hierarchical fused feature maps from input video frames and depth information masks. The feature fusion module, based on a self-attention mechanism, fuses the initial feature map and the hierarchical fusion feature map into a weighted feature map; The encoder compiles weighted feature maps into a one-dimensional sequence based on emotion influence factors; The Transformer decoder generates a patient's mood change score through an autoregressive approach. The alarm module monitors whether the patient's mood change score exceeds a threshold and issues an alarm when the threshold is exceeded. The encoder includes: The information compilation block compiles the weighted feature map into a two-dimensional image matrix; The positional encoding block adds positional encoding to the two-dimensional image matrix; The attention module simultaneously receives the positional encoding and the two-dimensional image matrix to generate image features; Feedforward neural networks nonlinearly transform image features to generate one-dimensional sequences; The location-coded block includes: A context extractor is used to extract contextual information from a two-dimensional image matrix to obtain the types of facial expressions. The context extractor includes convolutional layers and global pooling layers. The convolutional layers and global pooling layers sequentially convolve and pool the two-dimensional image matrix, compressing this information into a compact representation of context information, which can provide the types of expressions. The memory stores a position encoding matrix and multiple expression encoding matrices; the expression encoding matrices include at least a happiness encoding matrix, a sadness encoding matrix, a fear encoding matrix, and a suffocation encoding matrix; Feature enhancer: Selects the corresponding expression coding matrix according to the representation type, and enhances the expression coding matrix and position coding matrix into the two-dimensional image matrix.

2. The patient crisis monitoring and early warning system for radiotherapy as described in claim 1, characterized in that, The image information acquisition device includes: Fourier Transform Module: Acquires video information of the patient's face, performs Fast Fourier Transform on each video frame, and filters it to obtain the energy of the high-frequency components of each video frame. Image filtering module: A pre-set sharpness threshold is used to delete video frame images in which the energy of high-frequency components in the video frame is lower than the preset sharpness threshold, so as to filter out the image group of the patient's face; All video frames in the image group are arranged in chronological order.

3. The patient crisis monitoring and early warning system for radiotherapy as described in claim 1, characterized in that, The facial 3D coordinate acquisition device includes: Temporal alignment module: Acquires 3D point cloud information of the patient's face and aligns the 3D point cloud information with video frames in the image group in the temporal dimension; Coordinate calibration module: Obtains the 3D coordinates of each pixel in the 3D point cloud information and re-establishes a new 3D coordinate system with the horizontal plane as the reference plane; Coordinate Update Module: Changes the original 3D coordinates of the 3D point cloud information to coordinates in the new 3D coordinate system.

4. The patient crisis monitoring and early warning system for radiotherapy as described in claim 1, characterized in that, The facial depth information generation device includes: Depth information acquisition module: Extracts the depth coordinates of 3D point cloud information, generates a depth point cloud, and the pixel value of each pixel in the depth point cloud is the depth coordinate; Image focusing module: Extracts the initial facial contour from the 2D image based on the difference information of the depth coordinates; Image preprocessing module: Normalizes the depth values ​​of the initial facial contour to generate a depth information mask.

5. The patient crisis monitoring and early warning system for radiotherapy according to claim 4, characterized in that, The image focusing module uses the largest region in the depth point cloud where the pixel difference is less than a preset threshold as the initial facial contour.

6. The patient crisis monitoring and early warning system for radiotherapy as described in claim 5, characterized in that, The feature extraction module includes: The input layer is used to input video frames and depth information masks. A convolutional network consists of five convolutional layers, with the number of convolutional kernels gradually increasing from top to bottom. The cross network, consisting of five convolutional layers, corresponds to the convolutional network. It takes the element-wise multiplication of the current convolutional layer and the previous cross layer as input to output a hierarchical fused feature map. The input layer is a 1*1 convolutional network. The video frame and the depth information mask are input to the video frame and the depth information mask, respectively, to obtain the initial feature map and the mask feature map. The initial feature map is input into the convolutional network, and the mask feature map is input into the cross network; The initial layer of a convolutional network corresponds to the bottom layer of a cross-network.

7. The patient crisis monitoring and early warning system for radiotherapy according to claim 6, characterized in that, The feature fusion module concatenates the initial feature map and the hierarchical fusion feature map along the feature dimension, generates attention scores based on the convolutional network, generates joint weights based on the attention scores, and fuses the initial feature map and the hierarchical fusion feature map according to the attention weights.

8. The patient crisis monitoring and early warning system for radiotherapy according to claim 7, characterized in that, The Transformer decoder includes: A standard Transformer decoding block for decoding one-dimensional sequences; The temporal memory matrix block, based on multi-head attention and a feedforward neural network, guides the standard Transformer decoding block so that when decoding a one-dimensional sequence, it continuously regresses the one-dimensional sequence information of the previous time step to capture the continuous changes in facial expressions.

Images