Sterile operation detection method based on deep learning algorithm

Abstract

The present invention provides a method for detecting aseptic operation based on a deep learning algorithm. The method uses a camera to continuously capture images in an operating room, identifies the aseptic area in the operating room, and obtains a segmentation mask of a movable target in the aseptic area; determines whether each pixel of the image is a moving pixel by a preset threshold; obtains a foreground area through a segmentation mask, and expands the foreground area outward to obtain a surrounding area, calculates the proportion of moving pixels in the foreground area and the surrounding area, and generates a moving area ratio matrix; constructs a touch prediction model, and uses the moving area ratio matrix to train the touch prediction model, and completes real-time aseptic operation detection through the trained touch prediction model. The beneficial effects of the present invention are: using deep learning computer vision technology to monitor the doctor's operation during surgery, and issuing early warnings for violations of aseptic principles.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition, and in particular relates to a sterile operation detection method based on a deep learning algorithm. Background Art

[0002] The heterogeneity of infection complicates the epidemiological study of SSIs: incidence rates vary widely across procedures, hospitals, surgeons, and patients. The increasing use of minimally invasive (laparoscopic) procedures has led to a decrease in the incidence of SSIs. For example, among patients undergoing cholecystectomy, the SSI rate after laparoscopic surgery was 1.1%, compared with 4% after open surgery. Similarly, among patients with acute appendicitis, the SSI rate after minimally invasive surgery was 2%, compared with 8% after open surgery. However, a considerable number of intraoperative infections are caused by inadvertent violations of aseptic principles by physicians or assistants. To avoid violations of aseptic techniques, hospitals often provide preoperative reminders to physicians, but this measure lacks real-time effectiveness, and thus intraoperative infections still occur. Summary of the Invention

[0003] In view of this, the present invention aims to propose a sterile operation detection method based on a deep learning algorithm, in order to solve at least one of the above-mentioned technical problems.

[0004] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0005] The first aspect of the present invention proposes a sterile operation detection method based on a deep learning algorithm, the method comprising:

[0006] Using a camera device to continuously capture images in the operating room, identify the sterile area in the operating room, and obtain a segmentation mask of the movable object in the sterile area;

[0007] Determine whether each pixel of the image is a motion pixel through a preset threshold;

[0008] The foreground area is obtained by segmenting the mask, and the foreground area is expanded outward to obtain the surrounding area. The proportion of motion pixels in the foreground area and the surrounding area is calculated, and a motion area ratio matrix is ​​generated.

[0009] A touch prediction model is constructed and trained using the motion area ratio matrix. The trained touch prediction model is used to complete real-time sterile operation detection.

[0010] Furthermore, the recognition accuracy of movable targets is improved by constructing an overcomplete dictionary. The construction process of the overcomplete dictionary is as follows:

[0011] The features of the movable target in the sterile area are used as a template dictionary, each template represents the features of the target in a certain state, and the features are the edges and optical flow patterns of the movable target;

[0012] The candidate target set is generated by Gaussian distribution sampling to form a dictionary matrix, and principal component analysis is used to perform dimensionality reduction projection on the dictionary atoms to reduce the computational complexity.

[0013] Furthermore, a sparse reconstruction algorithm is used to improve the recognition capability of movable targets in different environments, and the recognition threshold of movable targets is lowered when conditions are met. The sparse reconstruction algorithm is as follows:

[0014] Represent the target to be detected as a sparse linear combination of templates in the dictionary:

[0015] , ;

[0016] Among them, z t is the target to be detected in the current frame, For the template dictionary, c t is a sparse coefficient vector obtained by minimizing the error, and argmin means finding the c that minimizes the entire objective function. t , is the square of the L2 norm, λ is the regularization coefficient, is the L1 norm, which is the sum of the absolute values ​​of all elements.

[0017] Furthermore, by defining the coherence index between dictionary atoms, dictionary atoms with high correlation are screened to improve the efficiency of the sparse reconstruction algorithm.

[0018] Furthermore, when the characteristics of the movable target change, the template dictionary is dynamically updated. The updating process is as follows:

[0019] The template dictionary is updated every 10 frames. During the use of the sparse reconstruction algorithm, the sparse coefficient of each frame is recorded, and the dictionary atom with the smallest weight is selected based on the sparse coefficient;

[0020] Calculate the similarity between the target to be detected and the atoms in the dictionary:

[0021] ; Among them, z t is the target to be detected in the current frame, d gi For atoms in the dictionary, is the square of the L2 norm.

[0022] If the minimum similarity is lower than the preset threshold, the dictionary update is triggered, and the target to be detected is added to the dictionary to replace the dictionary atom with the smallest weight, and updated to a new template; otherwise, the dictionary remains unchanged.

[0023] Furthermore, the process of determining whether each pixel of the image is a motion pixel by using a preset threshold is as follows:

[0024] Motion sparsity S is introduced to eliminate background noise or local optical flow jitter:

[0025] S = number of moving pixels / total number of pixels in the neighborhood window;

[0026] Introduce historical consistency H to avoid occasional jitter:

[0027] ;

[0028] Set a binary mask for each pixel:

[0029] , where P is the threshold judgment parameter, , L is the optical flow value, L t is the streamer value at time t, L t-1 is the light flow value at the previous moment, w1, w2, w3 are weight parameters, σ is the judgment threshold for moving objects and non-moving objects, and n is the sum of the number of pixel frames within the specified time.

[0030] Furthermore, the foreground area gradually shrinks inward according to a certain pixel value:

[0031] ;

[0032] The surrounding area is gradually expanded outward according to a certain pixel value:

[0033] ;

[0034] Combining the images of the previous and next 10 frames, we get a motion area ratio matrix with p+q rows and 21 columns, where a r is the ratio of motion pixels in the foreground area, b r is the ratio of motion pixels in the surrounding area, a ij is the pixel in the foreground area, b ij is the pixel point in the surrounding area, m ij A binary mask to determine whether a pixel is a moving pixel.

[0035] Furthermore, the touch prediction model receives the image, motion region ratio matrix, and segmentation mask, uses independent convolutional encoders to extract features, and performs feature fusion in the channel dimension;

[0036] The fused features are processed by a six-layer bidirectional LSTM to capture the dynamic information in the time series, and are then classified and predicted by a two-layer multilayer perceptron.

[0037] The collision prediction model uses binary cross entropy as the loss function and is optimized using the Adam optimizer.

[0038] The second aspect of the present invention proposes a server, comprising at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to enable the at least one processor to perform the method described in the first aspect.

[0039] A third aspect of the present invention provides a computer-readable storage medium storing a computer program, which implements the method described in the first aspect when executed by a processor.

[0040] Compared with the existing technology, the sterile operation detection method based on deep learning algorithm described in the present invention has the following beneficial effects: by dividing the sterile area and movable targets and detecting the motion trajectory of the movable targets, real-time sterile operation detection is completed. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0042] Figure 1 Schematic diagram of the workflow of the aseptic operation detection method described in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0044] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0045] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0046] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0047] Example 1: A sterile operation detection method based on a deep learning algorithm, comprising:

[0048] S1. Use a camera device to continuously capture images in the operating room, identify the sterile area in the operating room, and obtain a segmentation mask of the movable object in the sterile area;

[0049] S2, determining whether each pixel of the image is a moving pixel by using a preset threshold;

[0050] S3. Obtain the foreground area through the segmentation mask, expand the foreground area outward to obtain the surrounding area, calculate the proportion of motion pixels in the foreground area and the surrounding area, and generate a motion area ratio matrix;

[0051] S4. Construct a touch prediction model and use the motion area ratio matrix to train the touch prediction model. Use the trained touch prediction model to complete real-time sterile operation detection.

[0052] The camera device in step S1 of this embodiment includes six cameras, including two operation area monitoring cameras and four operation room global monitoring cameras;

[0053] The operating area monitoring cameras are installed on two operating lights or on both sides of the main operating light to achieve real-time monitoring of key areas such as the sterile area of ​​the operating table, the doctor and his hands;

[0054] Global monitoring cameras are placed in the four corners of the operating room to ensure full coverage of the dynamic changes in the operating room. These six cameras are connected to the computing terminal via a wireless network, enabling real-time data transmission and processing.

[0055] Step S1 uses a region extraction algorithm based on the YOLOv10 convolutional neural network to identify sterile areas such as the operating table, the sterile surface area of ​​the operating table, the doctor, the doctor's hands, and the doctor's chest area, as well as other objects. A strong tracking Kalman filter algorithm is used to track movable objects (or people) and generate motion trajectories. A U-net-based segmentation model is used to obtain segmentation masks for the detected movable objects.

[0056] To improve the detection accuracy of dynamic targets, an over-complete dictionary is constructed for features such as hands and instruments within the sterile surgical area, and a sparse representation method is used to achieve high-precision reconstruction. To improve the recognition ability of the YOLO model in complex environments, this embodiment combines a sparse reconstruction algorithm and lowers the recognition threshold for recognition that meets the sparse reconstruction conditions.

[0057] The recognition accuracy of movable targets is improved by constructing an overcomplete dictionary. The construction process of the overcomplete dictionary is as follows:

[0058] The features of the movable target in the sterile area are used as a template dictionary, each template represents the features of the target in a certain state, and the features are the edges and optical flow patterns of the movable target;

[0059] The candidate target set is generated by Gaussian distribution sampling to form a dictionary matrix, and principal component analysis is used to perform dimensionality reduction projection on the dictionary atoms to reduce the computational complexity.

[0060] A sparse reconstruction algorithm is used to improve the recognition capability of movable targets in different environments, and the recognition threshold of movable targets is lowered when conditions are met. The sparse reconstruction algorithm is as follows:

[0061] Represent the target to be detected as a sparse linear combination of templates in the dictionary:

[0062] , ;

[0063] Among them, z t is the target to be detected in the current frame, For the template dictionary, c t is a sparse coefficient vector obtained by minimizing the error, and argmin means finding the c that minimizes the entire objective function. t , is the square of the L2 norm, λ is the regularization coefficient, is the L1 norm, which is the sum of the absolute values ​​of all elements.

[0064] By defining the coherence index between dictionary atoms, dictionary atoms with high correlation are screened to improve the efficiency of the sparse reconstruction algorithm.

[0065] The two atoms of the dictionary are d i and d j , then the coherence index is expressed as:

[0066] ;

[0067] Furthermore, the coherence is regularized, that is, the coherence between dictionary atoms is minimized while optimizing the dictionary. The calculation process is:

[0068] ;

[0069] Where λ is the weight hyperparameter of the coherence regularization term, which is used to control the balance between coherence and reconstruction error; x i Is the signal y i In atomic d j The coefficient under ; D is the new dictionary; using the coherence parameter at the back and the reconstruction error at the front to minimize the combined value.

[0070] When the characteristics of the movable target change, the template dictionary is dynamically updated. The update process is as follows:

[0071] The template dictionary is updated every 10 frames. During the use of the sparse reconstruction algorithm, the sparse coefficient of each frame is recorded, and the dictionary atom with the smallest weight is screened according to the sparse coefficient; wherein the sparse coefficient is: the coefficient of the recognized object mapped to the atom in the dictionary, specifically c t .

[0072] Calculate the similarity between the target to be detected and the atoms in the dictionary:

[0073] ; Among them, z t is the target to be detected in the current frame, d gi For atoms in the dictionary, is the square of the L2 norm.

[0074] If the minimum similarity is lower than the preset threshold, the dictionary update is triggered, and the target to be detected is added to the dictionary to replace the dictionary atom with the smallest weight, and updated to a new template; otherwise, the dictionary remains unchanged.

[0075] The process of determining whether each pixel of the image is a motion pixel by using a preset threshold is as follows:

[0076] Motion sparsity S is introduced to eliminate background noise or local optical flow jitter:

[0077] S = number of moving pixels / total number of pixels in the neighborhood window;

[0078] Introduce historical consistency H to avoid occasional jitter:

[0079] ;

[0080] Set a binary mask for each pixel to determine whether the pixel is stationary (0) or moving (1):

[0081] , where P is the threshold judgment parameter, , L is the optical flow value, L t is the streamer value at time t, L t-1 is the light flow value at the previous moment, w1, w2, w3 are weight parameters, σ is the judgment threshold for moving objects and non-moving objects, and n is the sum of the number of pixel frames within the specified time.

[0082] The foreground area gradually shrinks inward according to a certain pixel value:

[0083] ;

[0084] The surrounding area is gradually expanded outward according to a certain pixel value:

[0085] ;

[0086] Combining the images of the previous and next 10 frames, we get a motion area ratio matrix with p+q rows and 21 columns, where a r is the ratio of motion pixels in the foreground area, b r is the ratio of motion pixels in the surrounding area, a ij is the pixel in the foreground area, b ij is the pixel point in the surrounding area, m ij A binary mask to determine whether a pixel is a moving pixel.

[0087] The touch prediction model receives the image, motion region ratio matrix, and segmentation mask, uses independent convolutional encoders to extract features, and performs feature fusion in the channel dimension;

[0088] The fused features are processed by a six-layer bidirectional LSTM to capture the dynamic information in the time series, and are then classified and predicted by a two-layer multilayer perceptron.

[0089] The collision prediction model uses binary cross entropy as the loss function and is optimized using the Adam optimizer.

[0090] In this embodiment, the touch prediction model includes:

[0091] Input layer:

[0092] Directly acquired RGB images: provide appearance information of movable and immovable objects;

[0093] Optical flow: used to capture the motion information of human hands and objects between adjacent frames;

[0094] Foreground mask: A multi-channel binary mask used to indicate the areas of different objects, including: the mask of the sterile area of ​​the operating table, the sterile area on the surface of the operating table, the doctor, the doctor's hands, and the doctor's chest.

[0095] Data preprocessing:

[0096] Union all identified objects, crop the image area, and encode the input data:

[0097] The RGB image, optical flow information and motion region temporal sequence are input into independent encoder branches respectively;

[0098] RGB and optical flow encoder: It consists of 5 convolutional modules, each of which contains the following layers:

[0099] 3×3 convolutional layer: used to extract local features;

[0100] ReLU activation layer: introduces nonlinearity to help the model learn complex features;

[0101] LayerNorm layer: standardizes the features of each layer to improve training stability;

[0102] 2×2 Max-Pooling layer: downsamples the feature map to reduce the amount of computation;

[0103] The third and fifth convolution modules are equipped with a dropout structure:

[0104] The foreground mask passes through a separate encoder to process the spatial relationship between the hand and the object;

[0105] Foreground mask encoder: It consists of 3 convolutional layers, each followed by a ReLU activation layer. The output is used to encode the positional relationship between the sterile area, the operator's hands, the surgical area, and other objects.

[0106] It is worth noting that each object is identified separately, and the recognition results of each object are combined as the foreground mask;

[0107] The motion area ratio matrix only corresponds to each moving object, that is, the motion area ratio is not constructed for the foreground mask;

[0108] The function of the motion region ratio matrix is ​​to characterize the collision characteristics (the collision objects will be deformed during the collision process, and the collision characteristics are deformation information).

[0109] Furthermore, the above-mentioned aseptic operation detection mainly identifies the collision between the sterile area and the sterile area. Therefore, it is necessary to extract the foreground mask formed by the fusion of each area and extract the collision. The collision between the sterile and sterile areas is obtained through the collision and the identification fusion of each area.

[0110] Feature fusion: Concatenate the features extracted from the foreground mask and the motion region ratio matrix with the features of the RGB and optical flow encoders in the channel dimension.

[0111] LSTM layer: The fused features are processed by a 6-layer bidirectional LSTM (Long Short-Term Memory).

[0112] Multilayer Perceptron (MLP): The features output by the LSTM layer are passed into a 2-layer multilayer perceptron with two hidden layers, 300 parameters in the first hidden layer, 150 parameters in the second hidden layer, and the final output layer.

[0113] Loss function: Use binary cross entropy loss as the loss function and use Adam optimizer for optimization.

[0114] Embodiment 2: A server comprises at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to enable the at least one processor to perform the method described in embodiment 1.

[0115] Embodiment 3: A computer-readable storage medium stores a computer program, which implements the method described in embodiment 1 when executed by a processor.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and description of the present invention.

[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A sterile operation detection method based on a deep learning algorithm, characterized in that: The method comprises: Using a camera device to continuously capture images in the operating room, identify the sterile area in the operating room, and obtain a segmentation mask of the movable object in the sterile area; Determine whether each pixel of the image is a motion pixel through a preset threshold; The foreground area is obtained by segmenting the mask, and the foreground area is expanded outward to obtain the surrounding area. The proportion of motion pixels in the foreground area and the surrounding area is calculated, and a motion area ratio matrix is ​​generated. A touch prediction model based on the motion area ratio matrix is ​​constructed, and the touch prediction model is trained using the collected motion area ratio matrix. Real-time sterile operation detection is completed through the trained touch prediction model.

2. The aseptic operation detection method based on deep learning algorithm according to claim 1, characterized in that: The recognition accuracy of movable targets is improved by constructing an overcomplete dictionary. The construction process of the overcomplete dictionary is as follows: The features of the movable target in the sterile area are used as a template dictionary, each template represents the features of the target in a certain state, and the features are the edge and optical flow of the movable target; The candidate target set is generated by Gaussian distribution sampling to form a dictionary matrix, and principal component analysis is used to perform dimensionality reduction projection on the dictionary atoms.

3. The aseptic operation detection method based on deep learning algorithm according to claim 2, characterized in that: A sparse reconstruction algorithm is used to improve the recognition capability of movable targets in different environments, and the recognition threshold of movable targets is lowered when conditions are met. The sparse reconstruction algorithm is as follows: Represent the target to be detected as a sparse linear combination of templates in the dictionary: With t ≈D t-1 c t ,c t =arg min||z t -D t-1 c t || 2 +λ||c t ||1; Among them, z t is the target to be detected in the current frame, D t-1 For the template dictionary, c t is a sparse coefficient vector obtained by minimizing the error, and argmin means finding the c that minimizes the entire objective function. t ,||.|| 2 is the square of the L2 norm, λ is the regularization coefficient, and ||·||1 is the L1 norm, which is the sum of the absolute values ​​of all elements.

4. The aseptic operation detection method based on deep learning algorithm according to claim 3 is characterized in that: By defining the coherence index between dictionary atoms, dictionary atoms with high correlation are screened to improve the efficiency of the sparse reconstruction algorithm.

5. The aseptic operation detection method based on deep learning algorithm according to claim 3 is characterized in that: When the characteristics of the movable target change, the template dictionary is dynamically updated. The update process is as follows: The template dictionary is updated every 10 frames. During the use of the sparse reconstruction algorithm, the sparse coefficient of each frame is recorded, and the dictionary atom with the smallest weight is selected based on the sparse coefficient; Calculate the similarity between the target to be detected and the atoms in the dictionary: λ i =exp(-||Z t -d gi || 2 ); where z t is the target to be detected in the current frame, d gi For atoms in the dictionary, ||.|| 2 is the square of the L2 norm; If the minimum similarity is lower than the preset threshold, the dictionary update is triggered, and the target to be detected is added to the dictionary to replace the dictionary atom with the smallest weight, and updated to a new template; otherwise, the dictionary remains unchanged.

6. The aseptic operation detection method based on deep learning algorithm according to claim 1, characterized in that: The process of determining whether each pixel of the image is a motion pixel by using a preset threshold is as follows: Motion sparsity S is introduced to eliminate background noise or local optical flow jitter: S = number of moving pixels / total number of pixels in the neighborhood window; Introduce historical consistency H to avoid occasional jitter: Set a binary mask for each pixel: Among them, P is the threshold judgment parameter, P = w1L + w2S - w3H, L is the optical flow value, L t is the optical flow value at time t, L t-1 is the optical flow value at the previous moment, w1, w2, w3 are weight parameters, σ is the judgment threshold for moving objects and non-moving objects, and n is the sum of the number of pixel frames within the specified time.

7. The aseptic operation detection method based on deep learning algorithm according to claim 1, characterized in that: The foreground area gradually shrinks inward according to a certain pixel value: The surrounding area is gradually expanded outward according to a certain pixel value: Combine the images of the previous and next 10 frames to obtain the motion area ratio matrix, where a r is the ratio of motion pixels in the foreground area, b r is the ratio of motion pixels in the surrounding area, a ij is the pixel in the foreground area, b ij is the pixel point in the surrounding area, m ij A binary mask to determine whether a pixel is a moving pixel.

8. The aseptic operation detection method based on deep learning algorithm according to claim 1, characterized in that: The touch prediction model receives the image, motion region ratio matrix, and segmentation mask, uses independent convolutional encoders to extract features, and performs feature fusion in the channel dimension; The fused features are processed by a six-layer bidirectional LSTM to capture the dynamic information in the time series, and are then classified and predicted by a two-layer multilayer perceptron. The collision prediction model uses binary cross entropy as the loss function and uses the Adam optimizer for parameter optimization.

9. A server, characterized in that: The invention comprises at least one processor and a memory in communication with the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor performs the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images