A method for catheter inspection of patients in prone position based on multi-dimensional image data

Abstract

The present invention provides a method for inspecting a patient's catheter in a prone position based on multi-dimensional image data, which relates to the field of medical equipment inspection, including setting a reference reflective sticker in the area to be collected, collecting the initial position of the catheter, and collecting 2D and 3D images of the patient's face in real time; preprocessing the collected image and performing tilt correction; performing neural learning on the 2D correction image to identify the mouth area; performing catheter identification based on the mouth area, and when the catheter is not identified, a catheter fall-off alarm is issued; according to whether the coordinates of the mouth area and the coordinates of the catheter intersect, when the mouth area and the catheter do not intersect, a catheter fall-off alarm is issued; a displacement threshold is preset, and the spatial distance between the coordinates of the catheter and the coordinates of the initial position is calculated to obtain the position deviation, and when the position deviation is greater than or equal to the displacement threshold, a catheter displacement alarm is issued. The present invention can not only monitor in real time and automatically alarm, improve work efficiency, but also reduce the false alarm rate and has strong reliability.

Description

Technical Field

[0001] The invention belongs to the field of medical equipment inspection, and in particular relates to a method for inspecting a catheter of a prone patient based on multi-dimensional image data. Background Art

[0002] The prone surgical posture is often used in spine and neurosurgery surgeries. For prone surgeries that use general anesthesia or involve the respiratory system, chest or abdomen, it is usually necessary to insert a tracheal tube to ensure that the patient's airway is unobstructed so that the patient can breathe normally during the operation. In clinical practice, tape or inelastic cloth tape is often used to fix the tracheal tube inserted into the mouth of the surgical patient. There is a possibility that the tracheal tube may be displaced or even fall off due to factors such as loose fixation and accidental movement of the patient. Therefore, medical staff need to frequently observe the patient's facial condition during the operation to ensure that the tracheal tube has not been significantly displaced or fallen off.

[0003] Since the patient is placed face down on the operating table in a prone position, medical staff need to squat to enter under the operating table for observation, which is time-consuming, laborious and difficult to observe in real time. Using high-definition cameras and artificial intelligence technology to build a dynamic facial monitoring and early warning system for patients undergoing surgery in the prone position can provide great help to medical staff. As one of the important functions of the dynamic facial monitoring and early warning system for patients undergoing surgery in the prone position, the monitoring and early warning function of the tracheal tube of the surgical patient requires the system to be able to identify the tracheal tube and determine in real time whether it has been displaced or fallen off. When the displacement distance exceeds the preset threshold or falls off, the system can automatically alarm.

[0004] Currently, the key technology to realize the endotracheal tube monitoring and early warning function for patients undergoing surgery in the prone position is endotracheal tube recognition and displacement calculation. However, only ordinary planar images can be used to roughly determine the position of the endotracheal tube. It is difficult to accurately determine whether the patient actually puts the tube in his or her mouth, thus failing to ensure the patient's safety. Summary of the invention

[0005] In view of the above deficiencies in the prior art, an object of the present invention is to provide a method for catheter inspection of a patient in prone position based on multi-dimensional image data.

[0006] The present invention provides a method for inspecting a patient's catheter in a prone position based on multi-dimensional image data, comprising:

[0007] S1: Setting a reference reflective sticker in the area to be collected, collecting the initial position of the catheter based on the reference reflective sticker, and collecting 2D images and 3D images including the patient's face in real time based on the reference reflective sticker;

[0008] S2: preprocessing the 2D image and the 3D image collected in real time, and performing tilt correction on the preprocessed 2D image and the preprocessed 3D image respectively to obtain a 2D corrected image and a 3D corrected image;

[0009] S3: performing neural learning on the 2D corrected image to identify and obtain the mouth area;

[0010] S4: performing catheter recognition based on the mouth area, and issuing a catheter fall-off alarm when the catheter is not recognized;

[0011] S5: when the catheter is identified based on the mouth area, according to whether the coordinates of the mouth area and the coordinates of the catheter intersect, when the mouth area and the catheter do not intersect, a catheter fall-off alarm is issued;

[0012] S6: When the mouth area intersects with the catheter, a displacement threshold is preset, and the spatial distance between the coordinates of the catheter and the coordinates of the initial position is calculated by the Euclidean distance method to obtain the position deviation. When the position deviation is greater than or equal to the displacement threshold, a catheter displacement alarm is issued.

[0013] According to a method for catheter inspection of a patient in prone position based on multi-dimensional image data provided by the present invention, in step S2, preprocessing of the 2D image includes denoising, fusion and grayscale conversion.

[0014] According to a method for catheter inspection of a patient in prone position based on multi-dimensional image data provided by the present invention, in step S2, preprocessing of the 3D image includes denoising and histogram equalization.

[0015] According to a method for catheter inspection of a patient in prone position based on multi-dimensional image data provided by the present invention, in step S2, the tilt correction of the 2D image is achieved by perspective control.

[0016] According to a method for catheter inspection of a patient in prone position based on multi-dimensional image data provided by the present invention, in step S2, the step of performing tilt correction on the 3D image specifically includes:

[0017] S21: Acquire the center coordinates of multiple two-dimensional codes in the reference reflective sticker when collecting the 3D image;

[0018] S22: Obtaining a reference plane according to the center coordinates of multiple two-dimensional codes;

[0019] S23: Subtract the 3D image from the reference plane to obtain a 3D corrected image.

[0020] According to a method for catheter inspection of a patient in prone position based on multi-dimensional image data provided by the present invention, step S3 specifically comprises:

[0021] S31: performing neural learning on the 2D corrected image to detect the face area and obtain the face area;

[0022] S32: Obtaining the mouth area from the face area recognition by using a key point detection algorithm.

[0023] According to a method for catheter inspection of a patient in prone position based on multi-dimensional image data provided by the present invention, in step S31, neural learning is performed on the 2D corrected image through a convolutional neural network.

[0024] According to a method for inspecting a patient's catheter in a prone position based on multi-dimensional image data provided by the present invention, in step S6, the expression of the position deviation is:

[0025] ξ=(x1―x0) 2 +(y1―y0) 2 +(z1―z0) 2 ;

[0026] Where ξ is the calculated position deviation, (x1, y1, z1) represents the catheter position coordinates in the current 3D correction image, and (x0, y0, z0) is the catheter initial position coordinates.

[0027] According to a method for inspecting a patient's catheter in a prone position based on multi-dimensional image data provided by the present invention, the reference reflective sticker in step S1 is positioned by using a plurality of reference two-dimensional codes.

[0028] According to the method for inspecting a patient's catheter in a prone position based on multi-dimensional image data provided by the present invention, the 2D image and the 3D image in step S1 are both collected by a binocular stereo camera.

[0029] The beneficial effects of the present invention are as follows:

[0030] The present invention provides a method for inspecting the catheter of a prone patient based on multi-dimensional image data. Compared with the traditional manual inspection of the tracheal catheter of a prone surgical patient, the present invention can realize real-time monitoring and automatic alarm, reduce the workload of medical staff, and improve work efficiency. In addition, a method for using a two-dimensional code center to correct 2D and 3D images is proposed, so that the algorithm can adapt to the camera installation position deviation of different operating tables, thereby improving the reliability of the system. Secondly, compared with only using 2D images, the present invention can integrate the reliability of traditional 2D image algorithms and 3D image information, more accurately determine whether the tracheal tube has fallen off, and accurately determine whether the displacement distance of the tracheal tube exceeds a threshold, thereby reducing the false alarm rate and improving the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0032] Figure 1 It is a flow chart of a method for catheter inspection of a patient in prone position based on multi-dimensional image data provided by an embodiment of the present invention;

[0033] Figure 2 It is a flow chart of performing tilt correction on the 3D image in a method for catheter inspection of a patient in prone position based on multi-dimensional image data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0035] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0036] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0037] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0038] The embodiments of the present invention are described below with reference to the accompanying drawings.

[0039] like Figure 1 As shown, the present invention provides a method for inspecting a patient's catheter in a prone position based on multi-dimensional image data, comprising:

[0040] S1: Setting a reference reflective sticker in the area to be collected, collecting the initial position of the catheter based on the reference reflective sticker, and collecting 2D images and 3D images including the patient's face in real time based on the reference reflective sticker;

[0041] Wherein, the reference reflective sticker in step S1 is positioned by using a plurality of reference two-dimensional codes.

[0042] Wherein, the 2D image and the 3D image in step S1 are both collected by a binocular stereo camera.

[0043] In this stage, due to changes in the angle and distance of the camera installation, the actual collected images may be tilted, which brings great difficulties to the subsequent detection reliability and sample training. Therefore, we first make standard QR code reflective stickers and stick them on the operating table. Then, we use a binocular stereo camera placed under the prone operating table facing the patient's face to collect 2D image data and 3D image data of the patient's face.

[0044] S2: preprocessing the 2D image and the 3D image collected in real time, and performing tilt correction on the preprocessed 2D image and the preprocessed 3D image respectively to obtain a 2D corrected image and a 3D corrected image;

[0045] Since the reliability requirements for the tracheal tube inspection method for prone surgical patients are very high, neither underreporting nor too high a false alarm rate can be achieved. It is difficult to obtain valid samples in actual scenarios, especially samples that should be alarmed. In order to maximize the sample efficiency and improve the reliability of the entire system, 2D and 3D images must be dynamically corrected in real time.

[0046] The installation angle and distance of the camera may be different on different operating tables. In order to improve the versatility and accuracy of the detection algorithm, it is particularly important to correct the images of different cameras to a unified posture. Therefore, we correct the 2D images and 3D images in this stage.

[0047] Wherein, in step S2, the preprocessing of the 2D image includes denoising, fusion and grayscale conversion.

[0048] Wherein, in step S2, the preprocessing of the 3D image includes denoising and histogram equalization.

[0049] Wherein, in step S2, the tilt correction of the 2D image is achieved by perspective control.

[0050] Furthermore, the specific process is to first detect the four QR codes in the 2D image to obtain the center point of each QR code, and then use perspective control, that is, perspective correction to complete the image correction. After correction, the 2D image shows a front view of the face with uniform resolution, which creates good conditions for subsequent algorithm detection. Even if the position of the camera changes, the algorithm can still work reliably.

[0051] like Figure 2 As shown, in step S2, the step of performing tilt correction on the 3D image specifically includes:

[0052] S21: Acquire the center coordinates of multiple two-dimensional codes in the reference reflective sticker when collecting the 3D image;

[0053] S22: Obtaining a reference plane according to the center coordinates of multiple two-dimensional codes;

[0054] S23: Subtract the 3D image from the reference plane to obtain a 3D corrected image.

[0055] Furthermore, similar to 2D image correction, the reference plane of the 3D image is also tilted due to the camera installation angle and position. It is also very difficult to accurately determine whether the catheter is in the patient's mouth on a tilted 3D image. Therefore, the 3D image correction is also completed based on the QR code correction method. The height information of the centers of the four QR codes on the 3D image is obtained according to the center position of the QR code, and a reference plane P is fitted. The original 3D image is subtracted from the plane P to obtain the corrected face 3D image.

[0056] S3: performing neural learning on the 2D corrected image to identify and obtain the mouth area;

[0057] Wherein, step S3 specifically includes:

[0058] S31: performing neural learning on the 2D corrected image to detect the face area and obtain the face area;

[0059] S32: Obtaining the mouth area from the face area recognition by using a key point detection algorithm.

[0060] Wherein, in step S31, neural learning is performed on the 2D corrected image through a convolutional neural network.

[0061] In this stage, we first perform face detection and then perform mouth area recognition. Specifically, we build training sets and test sets through facial image data, use a deep learning face detection algorithm model based on a convolutional neural network (CNN) to detect the patient's face, and perform face detection on the patient. After the face is detected, the key point detection algorithm is used to determine the position of the mouth, providing information for subsequent algorithms.

[0062] S4: performing catheter recognition based on the mouth area, and issuing a catheter fall-off alarm when the catheter is not recognized;

[0063] Furthermore, according to the mouth area recognition result of S3, the 3D space coordinates of the corresponding area are obtained through the 2D coordinates of the mouth area. Then, at the mouth area position on the 3D image, the tracheal tube is recognized according to the depth information. If the tracheal tube cannot be recognized, an alarm for the tracheal tube falling off is issued.

[0064] S5: when the catheter is identified based on the mouth area, according to whether the coordinates of the mouth area and the coordinates of the catheter intersect, when the mouth area and the catheter do not intersect, a catheter fall-off alarm is issued;

[0065] Furthermore, after the tracheal tube is identified, the 3D spatial coordinates of the tracheal tube are obtained, and then calculations are performed based on the spatial coordinates of the mouth area and the spatial coordinates of the tracheal tube to determine whether the tracheal tube is located in the mouth area. When the spatial coordinates of the two do not intersect, it is determined that the tracheal tube has fallen off, and a tracheal tube falling off alarm is issued.

[0066] S6: When the mouth area intersects with the catheter, a displacement threshold is preset, and the spatial distance between the coordinates of the catheter and the coordinates of the initial position is calculated by the Euclidean distance method to obtain the position deviation. When the position deviation is greater than or equal to the displacement threshold, a catheter displacement alarm is issued.

[0067] Wherein, in step S6, the expression of the position deviation is:

[0068] ξ=(x1―x0) 2 +(y1―y0) 2 +(z1―z0) 2 ;

[0069] Where ξ is the calculated position deviation, (x1, y1, z1) represents the catheter position coordinates in the current 3D correction image, and (x0, y0, z0) is the catheter initial position coordinates.

[0070] Furthermore, when the two spatial coordinates intersect, the Euclidean distance formula is used to calculate the spatial distance between the spatial coordinate position of the tracheal tube and the initial spatial coordinate position of the tracheal tube, wherein the initial tracheal tube position is the tracheal tube position obtained for the first time.

[0071] According to the calculation formula shown above, the position deviation result can be obtained. Then, we compare the calculation result with the preset displacement alarm threshold to determine whether the tracheal tube displacement exceeds the threshold. When the tracheal tube displacement exceeds the threshold, it means that the tracheal tube in the patient's mouth has a large displacement but has not completely fallen off. At this time, there is also a certain risk, and a tracheal tube displacement alarm is issued.

[0072] A binocular stereo camera placed under the prone operating table facing the patient's face simultaneously collects 2D and 3D image data of the patient's face, and uses the 2D image data to identify the mouth area of ​​the tracheal tube. Based on the mouth area identified in the 2D image, the tracheal tube is identified, the position is calculated, and an alarm is issued at the corresponding 3D image position.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for catheter inspection of a patient in prone position based on multi-dimensional image data, characterized in that: include: S1: Setting a reference reflective sticker on the operating table, collecting the initial position of the catheter based on the reference reflective sticker, and collecting a 2D image and a 3D image including the patient's face in real time based on the reference reflective sticker; S2: preprocessing the 2D image and the 3D image collected in real time, and performing tilt correction on the preprocessed 2D image and the preprocessed 3D image respectively to obtain a 2D corrected image and a 3D corrected image; Wherein, the tilt correction of the 2D image is achieved by perspective control; S3: performing neural learning on the 2D corrected image to identify and obtain the mouth area; S4: performing catheter recognition based on the mouth area, and issuing a catheter fall-off alarm when the catheter is not recognized; Step S4 specifically includes: identifying the catheter according to the position of the mouth area in the 3D image based on the depth information, and issuing a catheter fall-off alarm when the catheter cannot be identified; S5: when the catheter is identified based on the mouth area, according to whether the coordinates of the mouth area and the coordinates of the catheter intersect, when the mouth area and the catheter do not intersect, a catheter fall-off alarm is issued; S6: When the mouth area intersects with the catheter, a displacement threshold is preset, and the spatial distance between the coordinates of the catheter and the coordinates of the initial position is calculated by the Euclidean distance method to obtain the position deviation. When the position deviation is greater than or equal to the displacement threshold, a catheter displacement alarm is issued.

2. A method for catheter inspection of a patient in prone position based on multi-dimensional image data according to claim 1, characterized in that: In step S2, the preprocessing of the 2D image includes denoising, fusion and grayscale conversion.

3. The method for catheter inspection of a patient in prone position based on multi-dimensional image data according to claim 1, characterized in that: In step S2, the preprocessing performed on the 3D image includes denoising and histogram equalization.

4. The method for catheter inspection of a patient in prone position based on multi-dimensional image data according to claim 1, characterized in that: In step S2, the step of performing tilt correction on the 3D image specifically includes: S21: Acquire the center coordinates of multiple two-dimensional codes in the reference reflective sticker when collecting the 3D image; S22: Obtaining a reference plane according to the center coordinates of multiple two-dimensional codes; S23: Subtract the 3D image from the reference plane to obtain a 3D corrected image.

5. The method for catheter inspection of a patient in prone position based on multi-dimensional image data according to claim 1, characterized in that: Step S3 specifically includes: S31: performing neural learning on the 2D corrected image to detect the face area and obtain the face area; S32: Obtaining the mouth area from the face area recognition by using a key point detection algorithm.

6. The method for catheter inspection of a patient in prone position based on multi-dimensional image data according to claim 5, characterized in that: In step S31, neural learning is performed on the 2D corrected image through a convolutional neural network.

7. The method for catheter inspection of a patient in prone position based on multi-dimensional image data according to claim 1, characterized in that: In step S6, the expression of the position deviation is: in, To calculate the position deviation obtained, Indicates the catheter position coordinates in the current 3D correction image, is the initial position coordinate of the catheter.

8. The method for catheter inspection of a patient in prone position based on multi-dimensional image data according to claim 1, characterized in that: The reference reflective sticker in step S1 is positioned by using a plurality of reference two-dimensional codes.

9. The method for catheter inspection of a patient in prone position based on multi-dimensional image data according to claim 1, characterized in that: The 2D image and the 3D image in step S1 are both collected by a binocular stereo camera.

Images