A respiratory recovery monitoring device for chest surgery

Abstract

The present invention relates to a respiratory recovery monitoring device after thoracic surgery. The device uses a visual sensor to capture images of the patient's respiratory status, preprocesses the images using a multi-scale template, and describes the respiratory status using a variety of high-dimensional feature variables. This allows the device to identify image features at multiple scales, improving the robustness of risk identification. The device is particularly suitable for accurately monitoring the respiratory status of patients in bed after thoracic surgery.

Description

Technical Field

[0001] The present invention belongs to the field of medical devices, and in particular, relates to a respiratory recovery monitoring device after thoracic surgery. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, intelligent algorithms are playing an increasingly important role in medical diagnosis. Thoracotomy is currently an important treatment for a wide range of conditions, including progressive intrapleural bleeding, tracheobronchial injury, esophageal rupture, combined thoracoabdominal injuries, cardiac and major vascular injuries, and severe lung lacerations. While thoracotomy can effectively alleviate financial problems, it is inherently invasive and can increase the risk of postoperative complications, such as edema and phlegm. Without timely and effective intervention, the patient's respiratory function and physical fitness can be significantly impacted. Thoracotomy is a common clinical treatment, widely used in the treatment of a variety of conditions, including heart and lung diseases. Analysis of the actual situation with thoracic surgery reveals that while improving the condition, the inherent trauma of the surgery can significantly impact the patient's respiratory function. Therefore, postoperative respiratory monitoring is a crucial clinical task. By monitoring the patient's respiratory status, the patient's treatment progress and surgical recovery can be assessed, and abnormal respiratory status can be automatically reported to facilitate timely medical intervention. However, postoperative patients are often bedridden, and their clothing differs significantly from their usual attire. In particular, some patients require observation in the ICU and must have various tubes inserted, such as tracheal and gastric tubes, which can interfere with judgment. Furthermore, postoperative patients often have weaker breathing and less body movement. These issues all limit the ability to accurately monitor a patient's breathing.

[0003] Medical monitoring methods and devices based on visual sensors have the advantages of being contactless, non-invasive, and easy to use. Therefore, existing technologies use image processing methods to detect respiration. However, this method generally requires the recognition of contour information, which makes the processing algorithm extremely complex and lacks accuracy. For example, a complex processing process is usually required, including image frame denoising, enhancement, contour feature extraction, and calculation of respiratory rate based on the temporal changes of contour features. This is not only time-consuming but also easily affected by interference from the surrounding environment and clothing. There are also existing methods that use radar and vision combined (for example, application number 202010486719). Although this can improve accuracy to a certain extent, it is also easily affected by interference from clothing. Based on this, the prior art has also proposed the use of neural networks for recognition. However, such learning models are relatively complex and sometimes even require data dimensionality reduction to reduce the amount of computation, which brings about the technical problem of limited recognition accuracy. In particular, for open-chest surgery, patients have weak breathing, long ICU stays, stiff body posture, various tubes, and other interferences. In particular, the correlation between chest contour changes and respiratory movement is reduced after thoracotomy. None of the above existing technologies can solve these problems.

[0004] To this end, a device dedicated to respiratory monitoring after thoracotomy is developed, which can make full use of various image information and predict the patient's respiratory risk efficiently, quickly and accurately. Summary of the Invention

[0005] This application proposes a post-thoracic respiratory recovery monitoring device. This device uses a visual sensor to monitor a patient's preoperative respiratory status, acquires normal preoperative respiratory status data, and builds a model to learn from this data. After thoracic surgery, the device is used to monitor the patient using the same monitoring device, collects post-operative respiratory status data, and evaluates this data based on the previously learned model, outputting a risk assessment estimate. This application also uses a multi-scale template to pre-process images during the pre-processing phase, enabling the identification of image features at multiple scales and improving the robustness of risk identification.

[0006] A respiratory recovery monitoring device after thoracic surgery, comprising two cameras and a server;

[0007] Two of the cameras are fixed on a fixed rod parallel to the patient's upper body and are installed in sequence along the direction of the patient's body; they are used to capture images of the patient's face and chest;

[0008] The processor in each camera pre-processes the captured image, where the pre-processing includes filtering the image using three templates A, B, and C of different sizes to obtain the filtered image of the first camera. and the filtered image from the second camera

[0009] Perform the following operations on the server:

[0010] For images Perform appearance feature filtering to obtain multi-scale appearance features For images and images After performing disparity feature filtering, multi-scale disparity features are obtained

[0011] The 6 high-dimensional feature variables Y s Through the activation function σ(x) and multi-scale appearance features and multi-scale disparity features associated;

[0012] in,

[0013] A learning model is established, and the model coefficients are determined based on the image samples collected in reality. The cost function in the learning process is:

[0014]

[0015] in is the estimated value of the risk assessment variable and z is the true value of the risk assessment variable, z = 0 or z = 1, corresponding to no respiratory risk and presence of respiratory risk in the current input time series image, respectively; θ is the control parameter; N represents the total number of different time series image sequences collected.

[0016] The server also performs the following operations: processing a number of images of the patient collected after surgery, and calculating a risk assessment result z.

[0017] When the value of z is greater than 0.6, it is considered that there is a risk of abnormal breathing in the patient, and the server reports the risk to the alarm device.

[0018] Both cameras can fully capture the patient's face and upper chest, and the parallax of the patient's face in the two cameras is greater than 1 / 20 of the imaging range.

[0019] The optical axes of the cameras remain parallel to each other, the parameters of the two camera lenses and imaging sensors remain completely consistent, and the two cameras synchronously capture images at a certain frame rate F;

[0020] Take F = 10 frames per second.

[0021] When there is a risk of abnormal breathing in the patient, the server writes the data into the electronic medical record.

[0022] The acquisition period T of the camera to acquire images satisfies T≥5F.

[0023] The alarm device is a handheld terminal or a display located in the ward.

[0024] The camera is connected to the server via a communication network.

[0025] Invention points and technical effects:

[0026] 1. A unique learning model is used to monitor abnormal breathing conditions of patients. The high-dimensional features used by the model include both time series features reflecting the patient's respiratory rate and information such as the patient's facial features. This allows for monitoring the patient's breathing status from a wider range of dimensions and more targeted identification of risks reflected by the patient's breathing status.

[0027] 2. The model uses images collected by visual sensors to automatically generate high-dimensional feature data that reflects the patient's respiratory status, including multi-scale appearance features, disparity features, and time series features of visual data. It also learns from this high-dimensional feature data to establish a patient respiratory status monitoring model.

[0028] 3. Utilizing a unique patient respiratory status monitoring model, the system conducts real-time assessments of the risks of the patient's respiratory status from multiple dimensions, including time and space. When risks occur, it reminds doctors to pay manual attention to and intervene in the patient, greatly reducing manual workload. It can promptly detect abnormal risks of the patient's respiratory status and improve the automation and response speed of respiratory recovery monitoring after thoracic surgery.

[0029] 4. Utilizing an optimized preprocessing template, image noise can be better filtered and local image continuity maintained, thereby reducing the burden on subsequent algorithms and improving feature recognition accuracy. Furthermore, specialized incentive and cost functions are designed to ensure the model's applicability to various medical environments, enabling efficient and accurate early warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the front-end acquisition equipment deployment. DETAILED DESCRIPTION

[0031] S1: Image acquisition of patient breathing

[0032] This involves installing a camera directly in front of the patient to be monitored, using the camera to capture time-series visual images, and transmitting the captured images to a backend server in real time. The specific method is described below.

[0033] S1.1 A method for capturing a frontal image of a patient using a binocular perspective. When the patient is in a basically stable state and breathing naturally, a set of two cameras is set facing the patient to fully capture the patient's face and upper chest. The two cameras are fixed on a fixed rod parallel to the patient's upper body and are installed in sequence along the direction of the patient's body ( Figure 1 ), about 10 cm apart. The actual installation distance can be fine-tuned according to the on-site environment and the patient's body shape, ensuring that both cameras can fully capture the patient's face and upper chest, and the parallax of the patient's face in the two cameras is greater than 1 / 20 of the imaging range. This can ensure image quality with a small amount of data, thus providing a basis for subsequent processing; the optical axes of the cameras remain parallel to each other, and the parameters of the two camera lenses and imaging sensors remain completely consistent. The two cameras are numbered as camera I and camera J. The two cameras synchronously capture images at a certain frame rate F, and the captured images are numbered in time sequence. As a preferred embodiment, F = 10 frames per second.

[0034] S1.2 When any camera captures a frame of image, it is preprocessed using the following method. After preprocessing, each image will generate three corresponding preprocessing results.

[0035] Define A, B, and C as three preset template windows, and

[0036]

[0037]

[0038]

[0039] Window A is 3*3 in size, window B is twice the size of window A, and window C is twice the size of window B. Each of these three windows is used to extract image features at different scales, thereby better identifying surface features in the image, from microscopic (small scale) to macroscopic (large scale), and thus improving the robustness of risk identification. The window values ​​are shown above, and the criteria are as follows: if a coordinate is within 1 / 3 of the window center in both directions, the weight is 1; if it is within 1 / 3 of the window center in one direction, the weight is 0.6; if it is at the edge of the window in one direction and does not meet any of the above conditions, the weight is 0.1; and otherwise, the weight is 0.4. As can be seen, the weights at different window coordinates are proportional to their distance from the window center, equivalent to an approximate Gaussian filter. However, the weights remain consistent within a certain range, further preserving the spatial characteristics of the image and helping to improve the accuracy of subsequent model learning.

[0040] For an image I captured by camera I at time t t , after filtering through the aforementioned A, B, and C windows, we get three images Similarly, for an image J captured by camera J at time t t , after filtering through the aforementioned A, B, and C windows, we get three images

[0041]

[0042] The two parameters in the brackets after the image identifier are the position coordinates of a pixel in the image, for example, I t (u+p,v+q) represents image I t The pixel coordinates at position u+p,v+q in the window A, A(p,q) represents the coordinates at position p,q in the window A, Representing an image The pixel coordinates at position u,v in the image are as follows.

[0043]

[0044]

[0045] In the above formula, B(p,q), I t (u+p,v+q),J t (u+p,v+q), The meaning of is similar to formula (1), is the intermediate value calculated based on the right side of equation (2), and max[] indicates that the four elements in the brackets take a maximum value. According to the above two equations, The meaning is to scale down the intermediate result image, and the reduction ratios are I t 、J t 1 / 2 of the original image, and the reduction method is to take the maximum value among the corresponding 4 pixels of the source image.

[0046]

[0047]

[0048] In the above formula, C(p,q), I t (u+p,v+q),J t (u+p,v+q), The meaning of is similar to formula (1), is the intermediate value calculated based on the right side of equation (4), and max[] indicates that the 16 elements in the brackets take a maximum value. According to the above two equations, The meaning is to scale down the intermediate result image, and the reduction ratios are I t 、J t1 / 4 of the original image, and the reduction method is to take the maximum value among the corresponding 16 pixels of the source image.

[0049] It can be understood that the above template is a template that has been compared with the classic Gaussian template and optimized through a large number of experiments. Compared with the classic template, it can better filter image noise and maintain local continuity of the image, thereby reducing the burden of subsequent algorithms and improving the accuracy of feature recognition.

[0050] S2: High-dimensional image feature generation

[0051] Based on the preprocessed time-series image sequence acquired in step 1, high-dimensional features reflecting the patient's respiratory condition are generated for establishing a respiratory monitoring model and risk identification.

[0052] The high-dimensional image features include multi-scale appearance features, parallax features, and temporal features of the visual image.

[0053] S2.1 In order to obtain the multi-scale appearance features, first collect the image I from one of the two cameras in step 1. t , preprocessed by (1)-(5), and the three filtered images are obtained

[0054]

[0055] in, The meaning is as mentioned above. α, β, and γ are multi-scale feature windows. p and q are the coordinates of two dimensions in the multi-scale feature window. The value range is as shown in formula (6). The multi-scale appearance features are α, β, and γ as defined above.

[0056] S2.2 In order to obtain the multi-scale disparity features, based on the above, the disparity image is calculated

[0057] Image I collected from one of the two cameras in step 1 t , preprocessed by (1)-(5), and the three filtered images are obtained After the above preprocessing, the image captured by another camera is obtained into three images:

[0058]

[0059] Among them, || represents the absolute value symbol, Respectively The value at the u,v coordinate.

[0060] definition:

[0061]

[0062] in, The meaning is as mentioned above, μ, π, ρ are multi-scale feature windows, p, q are the coordinates of two dimensions in the multi-scale feature window, and the value range is as shown in formula (8). The multi-scale disparity features are μ, π, and ρ as defined in the above formula (8).

[0063] S2.3 In order to obtain the multi-scale time series features, take T images I1, I2, ..., I1 that are continuously collected in time series. T , and calculate the corresponding filtered image according to the method described in step 1 Further calculate according to the method described in steps S2.1 and S2.2 In order to generate data with sufficient information, T≥5F; F is the image acquisition frame rate in step 1, preferably F=10, T=50, that is, the acquisition period is 5 seconds. For clarity, let:

[0064]

[0065] In the above formula, t represents the temporal coordinate, that is, the order of the acquired image frames. Moving t from the subscript on the right side of the equation to the independent variable on the left side of the equation is intended to make the logical relationship between different steps more intuitive and convenient to express.

[0066] Further, let:

[0067]

[0068] Where X1(u,v,t), X2(u,v,t),…,X6(u,v,t) are as described above, Φ, Ψ, and Ω are the three-dimensional multi-scale feature windows, and p, q, and r are the coordinates of the three dimensions in the multi-scale feature window, with the value ranges shown in Equation (9). Y1(u,v,t), Y2(u,v,t),…,Y6(u,v,t) represent the calculated results of the image time series after filtering through the three-dimensional multi-scale feature window. The multi-scale time series features are defined as Φ, Ψ, and Ω as defined above. b1, b2, and b3 are bias parameters.

[0069] Where σ(x) is a nonlinear function:

[0070]

[0071] arctanx represents the inverse tangent trigonometric function. The parameter δ creates a discontinuous breakpoint at x = 0, which helps improve the model's learning performance. The parameter ∈ is a control variable used to control the convergence rate of the nonlinear function during subsequent learning. ∈ = 11.5 and δ = 0.009 are preferred.

[0072] According to the above steps, the multi-scale appearance features α, β, γ, disparity features μ, π, ρ, and temporal features Φ, Ψ, Ω of the visual image are obtained.

[0073] The multi-scale appearance features are used to describe the static appearance features of patients in a real environment, such as facial expression and facial expressions. The parallax features are used to describe the distribution differences of the patient's static appearance observations at different viewing angles. This difference will produce a large change when the patient breathes and can be used to reflect the patient's respiratory status. The temporal features are used to describe the patient's breathing process and are an important reference for the patient's respiratory status. The present invention creatively proposes the above three types of features and combines them into high-dimensional features, and uses experimental data to prove the effectiveness of this method in monitoring patient respiratory status and risk assessment.

[0074] S3: Learning of patient respiratory monitoring model

[0075] It is used to establish a learning model and determine the value of the high-dimensional image features described in step 2 based on the data samples collected in reality.

[0076] The output of the patient respiratory status monitoring model is a risk assessment variable z, which has a value range of [0,1]. When the assessment variable tends to 0, it indicates that the patient's respiratory status tends to the normal range; when the assessment variable tends to 1, it indicates that the patient's respiratory status tends to the abnormal range, the risk level is high, and countermeasures should be taken.

[0077] definition:

[0078]

[0079] Where s∈{1,2,3,4,5,6} corresponds to the six outputs of equation (9) in step 2. The range of p,q,r varies depending on the value of s. When s=1 and s=4, the values ​​of p and q are equal to the corresponding dimensions of the acquired image. When s=2 and s=4, the values ​​of p and q are equal to 1 / 2 of the corresponding dimensions of the acquired image. When s=3 and s=6, the values ​​of p and q are equal to 1 / 4 of the corresponding dimensions of the acquired image. The range of r is [1,T], where T represents the time length of the acquired image sequence. Γ(s) is Y s The weighted sum of (p,q,r), whose corresponding weight is defined by w(p,q,r,s). s is the linear bias parameter corresponding to Γ(s). σ(x) is defined in the same way as in step 2 (10).

[0080] According to the definition, Γ(s) contains 6 components. Let the risk assessment variable z be:

[0081]

[0082] The patient respiratory status monitoring model is established. Where z is the weighted sum of the six components of Γ(s), the weight is defined by τ(s), and d is the linear bias parameter.

[0083] Existing methods such as backpropagation method are used to learn the respiratory state monitoring model.

[0084] According to step 1, several sets of time-series image sequences are collected before the patient's surgery as training samples, and whether the patient has the risk of respiratory abnormalities during the corresponding period is manually recorded; for the original collected images in the training samples, the corresponding filtered images can be obtained according to equations (1)-(4) and used as input data for step 2.

[0085] According to the input data obtained in step 2 and step 1, all features are assigned to 1, and the bias parameters b1, b2, and b3 are assigned to 0. According to equations (5)-(9), the input Y required for equation (11) in step 3 can be obtained. s (p,q,r);s∈{1,2,3,4,5,6}.

[0086] According to the input data Y obtained in step 3 and step 2 s (p,q,r), the bias parameter c s , d is assigned to 0, and the estimated value of the risk assessment variable is obtained by the respiratory state monitoring model established according to formulas (11) and (12):

[0087] Since the values ​​of the features and the bias parameters described in the previous steps are all arbitrarily assigned initial values, the estimated values There should be a large error between the true value of the risk assessment variable z and the actual value of the risk assessment variable. Learning is the process of reducing this error. Let:

[0088]

[0089] in, θ represents the sum of the estimated values ​​of all learning samples. The true value of a sample, z = 0 or z = 1, corresponds to the absence or presence of respiratory risk in the current input time-series image, respectively. θ is a control parameter. Based on the true distribution of the samples, 0 < θ < 1 is typically chosen. In this case, θ = 0.15 is the preferred value. N represents the total number of samples participating in the learning process, that is, the total number of different time-series image sequences collected.

[0090] Using the back propagation method, the goal is to make Minimize and iterate sample by sample to obtain the optimized values ​​of the multi-scale appearance features α, β, γ, disparity features μ, π, ρ, temporal features Φ, Ψ, Ω, and the aforementioned bias parameters in step 2, and the learning is completed.

[0091] S4: Model-based patient respiratory monitoring and risk identification

[0092] According to the patient breathing monitoring model learned in step 3, the risk probability corresponding to the input time series image sequence is calculated.

[0093] According to the method described in step 1, after the patient's surgery, the front end collects several images of the patient and transmits them to the back end server; the back end server composes a time-series image sequence in chronological order, pre-processes each image, and inputs the results into step 2.

[0094] According to the methods described in steps 2 and 3, and based on the optimized values ​​of the features and parameters learned in step 3, the risk assessment result z corresponding to the time series image sequence is obtained according to equations (5)-(12).

[0095] When the z value is greater than 0.6, it is considered that there is a risk of abnormal breathing in the patient and the risk is reported.

[0096] Through experiments on 437 case data in our hospital, we obtained the risk reporting accuracy of the method described in the present invention, the average response time to patients with abnormal breathing after adopting the method of the present invention, and a comparison with many traditional methods.

[0097] Comparison of methods with examples Accuracy (%) Response time to exceptions Conventional image processing methods 54.9％ 15 seconds General Neural Network Methods 88.2％ 89 seconds Artificial 68.4％ 357 seconds This application 91.1％ 17 seconds

[0098] As can be seen from the above, the method of the present invention can better identify the phenomenon of abnormal breathing in patients and greatly improve the response time for discovering risks. It can better balance accuracy and response time than many traditional methods, making monitoring postoperative respiratory recovery through camera image acquisition a clinical option.

[0099] The above method is implemented by a respiratory recovery monitoring device. The device includes two cameras and a server. Step S1 is completed by the two cameras, so the camera contains a processor that can pre-process the captured image. The camera sends the pre-processed image to the server through the communication network, and steps S2-S4 are all completed in the server. It can be understood that an alarm device can also be included. After the server processes the data to obtain the patient's risk information, it is sent to the alarm device, thereby prompting medical staff to take corresponding measures. Of course, the relevant data can be stored in the hospital's electronic information system as part of the electronic medical record. The alarm device can be a handheld terminal or a display located in the ward.

Claims

1. A respiratory recovery monitoring device after thoracic surgery, characterized in that: Includes two cameras and server; Two of the cameras are fixed on a fixed rod parallel to the patient's upper body and are installed in sequence along the direction of the patient's body; they are used to capture images of the patient's face and chest; The processor in each camera pre-processes the captured image, where the pre-processing includes filtering the image using three templates A, B, and C of different sizes to obtain the filtered image of the first camera. 、 、 and the filtered image from the second camera 、 、 ; Perform the following operations on the server: For images 、 、 Perform appearance feature filtering to obtain multi-scale appearance features 、 、 ; For images 、 、 and images 、 、 After performing disparity feature filtering, multi-scale disparity features are obtained 、 、 Specifically: ； in, Indicates the absolute value sign, 、 、 Respectively 、 、 The value at the u,v coordinate; 、 、 is a parallax image; ； in, 、 、 is a multi-scale feature window, p and q are the coordinates of two dimensions in the multi-scale feature window; The six high-dimensional feature variables Through the activation function Multi-scale appearance features 、 、 and multi-scale disparity features 、 、 Related: ； ； 、 、 is a three-dimensional multi-scale feature window, p, q, and r are the coordinates of the three dimensions in the multi-scale feature window. 、 、…、 It represents the calculation result of the time series of the image after filtering through the three-dimensional multi-scale feature window. 、 、 is the bias parameter; t represents the time series coordinate; in, ; in is a control variable; is a parameter; A learning model is established, and the model coefficients are determined based on the image samples collected in reality. The cost function in the learning process is: ； in is the estimated value of the risk assessment variable and z is the true value of the risk assessment variable, z = 0 or z = 1, corresponding to no respiratory risk and presence of respiratory risk in the current input time series image, respectively; is a control parameter; N represents the total number of different time-series image sequences acquired.

2. The device according to claim 1, wherein: The server also performs the following operations: processing a number of images of the patient collected after surgery, and calculating a risk assessment result z.

3. The device according to claim 1, wherein: Both cameras can fully capture the patient's face and upper chest, and the parallax of the patient's face in the two cameras is greater than 1 / 20 of the imaging range.

4. The device according to claim 3, wherein: The optical axes of the cameras remain parallel to each other, and the parameters of the two camera lenses and imaging sensors remain completely consistent.

5. The device according to claim 4, characterized in that: The two cameras capture images synchronously at a certain frame rate F.

6. The device according to claim 5, characterized in that: Take F=10 frames per second.

7. The device according to claim 1, wherein: When there is a risk of abnormal breathing in the patient, the server writes the data into the electronic medical record.

8. The device according to claim 6, wherein: The acquisition cycle T of the camera to collect images satisfies .

9. The device according to claim 2, wherein: The alarm device is a handheld terminal or a display located in the ward.

10. The device according to claim 1, wherein: When the value of z is greater than 0.6, it is considered that there is a risk of abnormal breathing in the patient, and the server reports the risk to the alarm device.

Images