Thoracic and laparoscopic disease location imaging detection system

Abstract

The invention relates to the technical field of endoscopic surgery, and discloses a thoracic and laparoscopic disease location imaging detection system which comprises an endoscopic image capturing module, a probing analysis module, a three-dimensional construction module and a decision analysis module. According to the thoracic and laparoscopic disease position imaging detection system, a classifier is constructed according to a least square support vector machine algorithm, real-time analysis is performed on an endoscope image, bleeding points or suspected bleeding areas are automatically detected and classified, meanwhile, three-dimensional reconstruction is performed on the endoscope image in combination with a light projection algorithm, data support is provided for auxiliary positioning of a hemorrhagic disease position, and the detection accuracy is improved. A ConvNeXt-Base model is used for learning global features and local features, then a TransUNet segmentation model is used for automatically segmenting a disease location area and a non-disease location area in a lesion image, automatic identification of hemorrhagic disease location information is achieved, a judgment basis is provided for a decision engine module, the decision engine module obtains a diagnosis and treatment path according to a medical rule base, and the diagnosis and treatment path is used for diagnosis and treatment of hemorrhagic diseases. Decision points in the clinical diagnosis and treatment process are positioned, and doctors are assisted in formulating and optimizing operation schemes.

Description

Technical Field

[0001] The present invention relates to the technical field of laparoscopic surgery, and specifically to a thoracoscopic and laparoscopic lesion imaging detection system. Background Technique

[0002] Thoracoscopic and laparoscopic surgery is a minimally invasive surgical treatment method that is widely used now. It mainly involves puncturing holes in the abdomen, usually 3 - 4 holes. Instruments are placed through the holes to perform surgical operations in the abdominal cavity and pelvic cavity. The other end is connected to a monitor, and the conditions inside the thoracic cavity, abdominal cavity, and pelvic cavity can be seen through the screen, thus greatly improving the surgical efficiency.

[0003] After retrieval, a thoracoscopic and laparoscopic 3D color fluorescence high - definition imaging method, system, and thoracoscope are disclosed in the invention patent with the Chinese patent publication number CN118614857A. The thoracoscopic and laparoscopic 3D color fluorescence high - definition imaging system in this invention patent generates and stitches 3D color fluorescence images by collecting intraoperative white - light images and fluorescence images, and performs image enhancement processing to improve image quality, facilitating doctors to accurately analyze and judge the lesion position.

[0004] However, this laparoscopic 3D color fluorescence high - definition imaging method relies on doctors to observe image information to give decision results for lesion position analysis and judgment. This method depends on doctors' clinical experience, has great subjectivity, is difficult to accurately judge the lesion position, cannot automatically analyze images through a model to judge lesion position information, and cannot automatically give a treatment decision plan according to the lesion position information. Therefore, there is still room for further improvement in the efficiency of lesion detection. Thus, a thoracoscopic and laparoscopic lesion imaging detection system is proposed. Summary of the Invention

[0005] (1) Technical Problems to be Solved

[0006] Aiming at the deficiencies of the prior art, the present invention provides a thoracoscopic and laparoscopic lesion imaging detection system, which has the advantages of analyzing endoscopic images based on a lesion recognition model, quickly locating the lesion area, and automatically generating a corresponding treatment decision plan, and solves the problems that the existing laparoscopic 3D color fluorescence high - definition imaging method and system rely on doctors to judge the lesion position in the image, have a certain degree of subjectivity, and are difficult to automatically identify lesion position information to improve the diagnosis efficiency.

[0007] (2) Technical Solutions

[0008] To achieve the above - mentioned purpose of analyzing endoscopic images based on a lesion recognition model, quickly locating the lesion area, and automatically generating a corresponding treatment decision plan, the present invention provides the following technical solution: A thoracoscopic and laparoscopic lesion imaging detection system includes an endoscopic image acquisition module, which realizes the functions of image acquisition, image recording, and image transmission inside the human chest and abdominal cavities.

[0009] The exploration and analysis module performs real-time analysis on the endoscopic images to automatically detect the bleeding site information or the location of suspected bleeding points;

[0010] The three-dimensional construction module assists in locating the bleeding site based on three-dimensional reconstruction technology;

[0011] The decision-making and analysis module extracts the characteristics of the bleeding site based on machine learning, constructs an identification model for the bleeding site, and provides a treatment plan according to the identification result.

[0012] Preferably, the endoscopic imaging module includes an integrated wireless high-definition camera sub-module, an LED cold light source sub-module, a wireless function display and control sub-module, and a main power control sub-module. The wireless high-definition camera sub-module includes a laparoscope body, and an optical interface and a high-definition field-of-view camera are configured for the laparoscope body. The optical interface adopts a C-Mount standard interface with optical zoom, and the zoom range of the optical zoom is set to 14-32 mm;

[0013] The high-definition field-of-view camera is provided with a video recording unit for storing image data on a local device or in the cloud. The high-definition field-of-view camera is also provided with a cloud interaction unit for remote teaching and guidance through real-time cloud interaction based on the image data stored in the cloud. The high-definition field-of-view camera is configured with a wired output and a wireless output mode.

[0014] Preferably, the LED cold light source sub-module includes a xenon lamp cold light source and a light source power supply unit. The color temperature range of the xenon lamp cold light source is 3000K-7000K. The light source power supply unit is divided into a self-powered mode and a wireless power supply mode. According to the self-powered mode, a storage battery electrically connected to the xenon lamp cold light source is configured for the light source power supply unit; according to the wireless power supply mode, an antenna transmitter is configured for the light source power supply unit, and a power conversion receiver signal-connected to the antenna transmitter, and the power conversion receiver is electrically connected to the xenon lamp cold light source;

[0015] The wireless function display and control sub-module adopts a wireless point-to-point radio frequency transmission mode, which includes a radio frequency transceiver configured with an antenna, a display configured with a main control panel, and a mobile terminal configured with a control APP. Both the display and the mobile terminal are signal-connected to the high-definition field-of-view camera through the radio frequency transceiver. The mobile terminal selects one of a mobile phone, a tablet computer, and a laptop computer.

[0016] Preferably, the exploration and analysis module analyzes the endoscopic images based on the least squares support vector machine algorithm, and its specific steps include:

[0017] S1. Access the image output port of the display, collect the endoscopic images in real time, and perform grayscale processing on the images based on the weighted average algorithm;

[0018] S2. Extract the gray-level co-occurrence matrix features from the generated grayscale image, and initialize the parameters of the least squares support vector machine algorithm;

[0019] S3. Obtain the learning samples of the lesion images in the cloud data platform, and construct an image lesion area classifier based on the learning and classification characteristics of the least squares support vector machine;

[0020] S4. Input the obtained endoscopic image into the image lesion area classifier, and output the lesion recognition result.

[0021] Preferably, in step S1, the endoscopic image is grayscale processed based on the weighted average algorithm, and the image grayscale processing formula is:

[0022] Gray = 0.299R + 0.587G + 0.114B;

[0023] In the above formula, R, G, and B respectively represent the component values of red, green, and blue in the RGB format endoscopic image, Gray represents the image grayscale value, and each component value in the endoscopic image is automatically traversed according to the image grayscale processing formula, and the endoscopic grayscale image is output;

[0024] In step S2, a gray-level co-occurrence matrix is constructed based on the endoscopic grayscale image, and its specific steps include:

[0025] a. Set any point i in the image as (x, y), then set a point j deviating from point i as (x + a, y + b), and form a point pair with point i and point j;

[0026] b. Let the gray-level value of this point pair be (f1, f2). Assuming that the maximum gray level of the endoscopic grayscale image is L, then there are L·L combinations of f1 and f2;

[0027] c. In the endoscopic grayscale image, count the number of occurrences of each (f1, f2) value and arrange them into a square matrix, and then normalize the total number of occurrences of (f1, f2) to obtain the probability P ij , then the gray-level co-occurrence matrix is obtained;

[0028] Calculate the eigenvalues according to the obtained gray-level co-occurrence matrix, including:

[0029] 1) Energy feature Asm:

[0030] 2) Contrast feature Con:

[0031] 3) Correlation feature IDE:

[0032] 4) Entropy feature Ent:

[0033] Preferably, in step S3, the learning samples of the lesion images are mapped to a high-dimensional feature space based on a non-linear mapping function and feature classification is performed. The search target optimization for the lesion image region division is expressed as:

[0034]

[0035] In the above formula, R(ω) is the structural risk, γ is the regularization factor, and ξ i is the tolerance error. The Lagrange method is introduced to solve the optimization problem, which is expressed as:

[0036]

[0037] In the above formula, α i is the Lagrange multiplier, and the optimization conditions are set as According to the optimization conditions, the following can be obtained through optimization:

[0038]

[0039] By eliminating ω and ξ, after arrangement, we get Define Then the kernel function Therefore, the optimal classification function, that is, the classifier is

[0040] Preferably, the three-dimensional construction module realizes three-dimensional reconstruction based on the ray projection algorithm. The specific steps of the ray projection algorithm include:

[0041] L1. Program to read the volume data based on the VTK function library, and perform filtering and denoising processing on the volume data based on the median filtering algorithm. The steps of the median filtering algorithm include:

[0042] a. Set a median filter with a template size of N*N. Its output value should be greater than or equal to (N*N - 1) / 2 in the template and also less than or equal to (N*N - 1) / 2 in the template, where N generally takes an odd value;

[0043] b. The formula of the median filtering algorithm is expressed as g(x,y) = median[f(s,t)]. Move the template in the image and coincide the center of the template with a certain pixel position in the image;

[0044] c. Read the gray values of the corresponding pixels under the template, arrange them in ascending order, determine the median value of the gray value sorting, and assign this median value to the pixel value at the center position of the template;

[0045] L2. Classify endoscopic images according to the volume data attributes, and assign corresponding color values and opacities to various endoscopic image data with reference to the set color mapping table and opacity table;

[0046] L3. Perform resampling along the line of sight according to the ray tracing method, accumulate the color values and opacities of each pixel on the image plane, and output and display the synthesized three-dimensional image.

[0047] Preferably, extract the bleeding lesion site features based on machine learning and construct a bleeding lesion site recognition model. The specific steps include:

[0048] T1. Set the input image as I, the label vector as L, and the output of the fully connected layer as P f (c│I);

[0049] T2. Initialize the parameters W g and W l of the global and local branches according to the ConvNeXt-Base model ImageNet pre-training parameters, and randomly initialize the fully connected layer;

[0050] T3. Learn the global feature F g according to the input image I, calculate the parameters P g (c│I) of the last fully connected layer of the global branch and the parameters Pool g of the global average pooling layer, and optimize the global branch parameters W g according to the following formula:

[0051]

[0052] where l c is the true label of the class, σ is the sigmoid function, and C is the number of pathological types;

[0053] T4. Calculate the mask M prob according to the input image I, learn the local feature F l , calculate the parameters P l (c│I) of the last fully connected layer of the local branch and the parameters Pool l of the global average pooling layer, and optimize the local branch parameters W l ;

[0054] T5. Connect Pool g and Pool l , calculate P f (c│I), and optimize W g according to the formula in T3 above;

[0055] T6. Construct a TransUNet segmentation model with the minimum Dice loss to automatically segment the diseased and non-diseased regions of the input image I. The Dice loss formula is expressed as:

[0056]

[0057] where M gt is the ground truth mask, and M prob is the predicted mask;

[0058] The construction steps of the TransUNet segmentation model include:

[0059] 1) CNN-based feature extraction: Select the pre-trained ResNet50 as the encoder backbone network, and extract the features of the image through multiple convolutional layers and residual blocks to generate feature maps of multiple different resolutions;

[0060] 2) Incorporate the Transformer module: According to the encoder, introduce the multi-head self-attention mechanism of the Transformer, and convert the 1×1 patches in the feature map extracted by ResNet50 into sequences through patch embedding and use them as the Transformer input;

[0061] 3) Restore the spatial resolution: Introduce a cascaded upsampler composed of multiple upsampling steps. Each step includes a 2x upsampling operation, a 3×3 convolutional layer, and a ReLU activation layer to restore the spatial resolution in the output of the Transformer encoder;

[0062] 4) Fuse high- and low-resolution features: The decoder restores the segmentation mask in the output of the Transformer encoder according to the CUP, combines the output feature map of the Transformer with the CNN feature map, restores the resolution of the original image through the upsampling step, generates the segmentation result through the convolutional layer, and adds a sigmoid activation function to scale the output pixel values between 0 and 1;

[0063] T7. Through the adaptive average pooling function, adjust the mask size to 1×h×w (h and w represent the height and width of the global feature map respectively). Through the argmax function, set the pixel values of the diseased region to 1 and the pixel values of the non-diseased region to 0, and obtain the local feature map (c×h×w) according to the element-wise multiplication. The element-wise multiplication calculation formula is:

[0064] Preferably, construct a CDSS framework, including a medical rule base, an identification model base, and a decision engine module, for providing treatment plans according to the identification results. The construction steps of the medical rule base include:

[0065] 1) Collect medical knowledge items and sort out the clinical diagnosis and treatment path of the reagent.

[0066] 2) Locate the decision-making points in the clinical diagnosis and treatment process and generate medical rule content.

[0067] 3) Generate rule trigger scenarios, rule verification and testing, and package the rules.

[0068] The recognition model library identifies the bleeding lesion location information in the lesion image based on the TransUNet segmentation model. According to the identified bleeding lesion location information, the decision engine module realizes dynamic decision recommendation based on the medical rule library.

[0069] (III) Beneficial effects

[0070] Compared with the prior art, the present invention provides a thoracoscopic and laparoscopic lesion imaging detection system, which has the following beneficial effects:

[0071] 1. For the thoracoscopic and laparoscopic lesion imaging detection system, through grayscale processing of the collected endoscopic images, extracting the gray-level co-occurrence matrix features, constructing a classifier according to the least squares support vector machine algorithm, performing real-time analysis on the endoscopic images, automatically detecting and classifying bleeding points or suspected bleeding areas, and at the same time combining the light projection algorithm to perform three-dimensional reconstruction on the endoscopic images, providing data support for the auxiliary positioning of the bleeding lesion location.

[0072] 2. For the thoracoscopic and laparoscopic lesion imaging detection system, according to the learning samples of the lesion images in the cloud data platform, learning the global features and local features through the ConvNeXt-Base model, and then using the TransUNet segmentation model to automatically segment the lesion area and non-lesion area in the lesion image, realizing the automatic recognition of the bleeding lesion location information, thereby providing a judgment basis for the decision engine module. The decision engine module obtains the diagnosis and treatment path according to the medical rule library, locates the decision-making points in the clinical diagnosis and treatment process, and automatically generates a decision plan recommendation to assist the doctor in formulating and optimizing the surgical plan. Brief description of the drawings

[0073] Figure 1 It is a schematic diagram of the modules of the thoracoscopic and laparoscopic lesion imaging detection system of the present invention. Specific implementation manners

[0074] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the embodiments and drawings of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0075] Please refer toFigure 1 , a thoracoscopic and laparoscopic disease position imaging detection system, including a laparoscopic imaging module, which realizes the functions of image acquisition, image recording and image transmission inside the human chest and abdomen;

[0076] An exploration and analysis module, which analyzes the laparoscopic images in real time and automatically detects the bleeding disease position information or the location of suspected bleeding points;

[0077] A three-dimensional construction module, which assists in positioning the bleeding disease position based on three-dimensional reconstruction technology;

[0078] A decision-making analysis module, which extracts the characteristics of the bleeding disease position based on machine learning, constructs a bleeding disease position recognition model, and provides a treatment plan according to the recognition result.

[0079] Further, the laparoscopic imaging module includes an integrated wireless high-definition camera sub-module, an LED cold light source sub-module, a wireless function display and control sub-module, and a total power supply control sub-module. The wireless high-definition camera sub-module includes a laparoscope body, which is configured with an optical interface and a high-definition surgical field camera. The optical interface adopts a C-Mount standard interface with optical zoom, and the zoom range of the optical zoom is set to 14-32 mm;

[0080] The high-definition surgical field camera is provided with a video recording unit for storing image data on a local device or in the cloud. The high-definition surgical field camera is also provided with a cloud interaction unit for remote teaching and guidance through real-time cloud interaction based on the image data stored in the cloud. The high-definition surgical field camera is configured with a wired output and a wireless output mode.

[0081] Specifically, the image display mode of the high-definition surgical field camera is wireless transmission, its signal-to-noise ratio ≥ 27 dB, the overall dimensions are 245 mm * 50 mm * 52.5 mm, and the overall weight is 450 g, the power consumption ≤ 10 W. The video recording method can store data locally (SD card, mobile terminal) and in the cloud, and remote teaching, remote guidance, and cloud storage functions can be carried out through real-time cloud interaction.

[0082] Further, the LED cold light source sub-module includes a xenon lamp cold light source and a light source power supply unit. The color temperature range of the xenon lamp cold light source is 3000K - 7000K. The light source power supply unit is divided into a self-powered mode and a wireless power supply mode. According to the self-powered mode, an energy storage battery electrically connected to the xenon lamp cold light source is configured for the light source power supply unit; according to the wireless power supply mode, an antenna transmitter is configured for the light source power supply unit, and a power conversion receiver signal-connected to the antenna transmitter, and the power conversion receiver is electrically connected to the xenon lamp cold light source;

[0083] The wireless function display and control sub-module adopts a wireless point-to-point radio frequency transmission mode, which includes a radio frequency transceiver configured with an antenna, a display configured with a main control panel, and a mobile terminal configured with a control APP. Both the display and the mobile terminal are signal-connected to the high-definition surgical field camera through the radio frequency transceiver. The mobile terminal selects one of a mobile phone, a tablet computer, and a laptop computer.

[0084] Specifically, the light source power supply unit selects a self-powered mode to supply power to the xenon lamp cold light source through a rechargeable energy storage battery. The light source power supply unit selects a wireless power supply mode. Using the principle of electromagnetic induction or electric field coupling, an inductive or capacitive coupling is established between the transmitter and the receiver, and the electric energy is converted by a power conversion receiver to supply power to the xenon lamp cold light source to achieve energy transfer.

[0085] Among them, the maximum video transmission distance of the radio frequency transceiver is 10m, and the image delay time < 100ms.

[0086] Furthermore, the exploration and analysis module analyzes the endoscopic image based on the least squares support vector machine algorithm. The specific steps include:

[0087] S1. Access the image output port of the display, collect the endoscopic image in real time, and perform grayscale processing on the image based on the weighted average algorithm;

[0088] S2. Extract the gray-level co-occurrence matrix features from the generated grayscale image, and initialize the parameters of the least squares support vector machine algorithm;

[0089] S3. Obtain the learning samples of the lesion images in the cloud data platform, and construct an image lesion area classifier based on the learning and classification characteristics of the least squares support vector machine;

[0090] S4. Input the obtained endoscopic image into the image lesion area classifier, and output the lesion recognition result;

[0091] In step S1, the endoscopic image is grayscale processed based on the weighted average algorithm. The image grayscale processing formula is:

[0092] Gray = 0.299R + 0.587G + 0.114B;

[0093] In the above formula, R, G, and B respectively represent the red, green, and blue component values in the RGB format endoscopic image, Gray represents the image grayscale value. Each component value in the endoscopic image is automatically traversed according to the image grayscale processing formula, and the endoscopic grayscale image is output;

[0094] In step S2, a gray-level co-occurrence matrix is constructed based on the endoscopic grayscale image. The specific steps include:

[0095] a. Let any point \(i\) in the image be represented as \((x,y)\), then a point \(j\) deviated from point \(i\) is represented as \((x + a,y + b)\), and a point pair is formed by point \(i\) and point \(j\).

[0096] b. Let the gray values of this point pair be represented as \((f1,f2)\). Assuming that the maximum gray level of the endoscopic gray image is \(L\), then there are \(L\cdot L\) combinations of \(f1\) and \(f2\).

[0097] c. In the endoscopic gray image, count the number of occurrences of each \((f1,f2)\) value and arrange them into a square matrix, and then normalize the total number of occurrences of \((f1,f2)\) to obtain the probability \(P\). ij , then the gray-level co-occurrence matrix is obtained.

[0098] Calculate the eigenvalues based on the obtained gray-level co-occurrence matrix, including:

[0099] 1) Energy feature Asm:

[0100] 2) Contrast feature Con:

[0101] 3) Correlation feature IDE:

[0102] 4) Entropy feature Ent:

[0103] In step S3, map the learning samples of the lesion image to the high-dimensional feature space based on the non-linear mapping function and perform feature classification. The search target optimization for the lesion image region division is expressed as:

[0104]

[0105] In the above formula, \(R(\omega)\) is the structural risk, \(\gamma\) is the regularization factor, \(\xi\) i is the allowable error. Introduce the Lagrange method to solve the optimization problem, which is expressed as:

[0106]

[0107] In the above formula, \(\alpha\) i is the Lagrange multiplier. Set the optimization conditions as According to the optimization conditions, the optimization result is:

[0108]

[0109] By eliminating \(\omega\) and \(\xi\), after arrangement, we get Define Then the kernel function Therefore, the optimal classification function is obtained, that is, the classifier is

[0110] Specifically, the collected endoscopic images are first grayscale processed, and then the gray-level co-occurrence matrix features are extracted from the grayscale images, which can assist in analyzing information such as the spatial correlation of the images, the complexity of the gray-level distribution, and the image clarity. A classifier is constructed according to the least squares support vector machine algorithm to classify the lesion areas of the endoscopic images.

[0111] Furthermore, the three-dimensional construction module realizes three-dimensional reconstruction based on the ray projection algorithm. The specific steps of the ray projection algorithm include:

[0112] L1. Program to read in the volume data based on the VTK function library, and perform filtering and denoising processing on the volume data based on the median filtering algorithm. The steps of the median filtering algorithm include:

[0113] a. Set a median filter with a template size of N*N, and its output value should be greater than or equal to (N*N - 1) / 2 in the template and also less than or equal to (N*N - 1) / 2 in the template, where N generally takes an odd value;

[0114] b. The median filtering algorithm formula is expressed as g(x,y) = median[f(s,t)]. Move the template in the image and coincide the center of the template with a certain pixel position in the image;

[0115] c. Read the gray-level values of the corresponding pixels under the template, arrange them in ascending order, determine the median value of the gray-level value sorting, and assign this median value to the pixel value at the center position of the template;

[0116] L2. Classify the endoscopic images according to the volume data attributes, and assign corresponding color values and opacity values to each type of endoscopic image data with reference to the set color mapping table and opacity table;

[0117] L3. Resample along the line-of-sight direction according to the ray tracing method, accumulate the color values and opacity of each pixel on the image plane, and output and display after synthesizing the three-dimensional image.

[0118] Specifically, filter and denoise the volume data of the endoscopic images according to the median filtering algorithm, then classify the endoscopic images according to the data attributes, and then assign corresponding color values and opacity values according to the set color mapping table and opacity table, and synthesize the three-dimensional image after accumulating the color values and opacity of each pixel on the image plane.

[0119] Furthermore, extract the bleeding lesion site features based on machine learning and construct a bleeding lesion site recognition model. The specific steps include:

[0120] T1. According to the learning samples of the lesion images in the cloud data platform, set the input image as I, the label vector as L, and the output of the fully connected layer as Pf (c│I);

[0121] T2. Initialize the parameters W of the global and local branches according to the ImageNet pre-training parameters of the ConvNeXt-Base model g and W l , and randomly initialize the fully connected layer;

[0122] T3. Learn the global feature F according to the input image I g , calculate the parameters P of the last fully connected layer of the global branch g (c│I) and the parameters Pool of the global average pooling layer g , and optimize the global branch parameters W according to the following formula g :

[0123]

[0124] where l c is the true label of the class, σ is the sigmoid function, and C is the number of pathological types;

[0125] T4. Calculate the mask M according to the input image I prob , learn the local feature F l , calculate the parameters P of the last fully connected layer of the local branch l (c│I) and the parameters Pool of the global average pooling layer l , and optimize the local branch parameters W according to the formula in T3 above l ;

[0126] T5. Connect Pool g and Pool l , calculate P f (c│I), and optimize W according to the formula in T3 above g ;

[0127] T6. Construct a TransUNet segmentation model with the minimum Dice loss to automatically segment the diseased and non-diseased regions of the input image I. The Dice loss formula is expressed as:

[0128]

[0129] where M gt is the true mask, and M prob is the predicted mask;

[0130] The construction steps of the TransUNet segmentation model include:

[0131] 1) CNN-based feature extraction: Select the pre-trained ResNet50 as the encoder backbone network, extract the features of the image through multiple convolutional layers and residual blocks, and generate feature maps with multiple different resolutions;

[0132] 2) Incorporate the Transformer module: According to the encoder, introduce the multi-head self-attention mechanism of the Transformer. Convert the 1×1 patches in the feature map extracted by ResNet50 into a sequence through patch embedding and use it as the input of the Transformer;

[0133] 3) Restore the spatial resolution: Introduce a cascaded upsampler composed of multiple upsampling steps. Each step includes a 2x upsampling operation, a 3×3 convolutional layer, and a ReLU activation layer to restore the spatial resolution in the output of the Transformer encoder;

[0134] 4) Fuse high- and low-resolution features: The decoder restores the segmentation mask in the output of the Transformer encoder according to the CUP, combines the output feature map of the Transformer with the CNN feature map, restores the resolution of the original image through the upsampling step, generates the segmentation result through the convolutional layer, and adds a sigmoid activation function to scale the output pixel values between 0 and 1;

[0135] T7. Through the adaptive average pooling function, adjust the mask size to 1×h×w (h and w represent the height and width of the global feature map respectively). Through the argmax function, set the pixel values of the diseased area to 1 and the pixel values of the non-diseased area to 0, and obtain the local feature map (c×h×w) according to the element-wise multiplication. The element-wise multiplication calculation formula is:

[0136] Specifically, according to the learning samples of the lesion images in the cloud data platform, input the lesion images into the ConvNeXt-Base model, continuously learn the global features and local features, and then construct a TransUNet segmentation model to automatically segment the diseased area and non-diseased area in the lesion images to achieve automatic recognition of the bleeding diseased area information;

[0137] Among them, ConvNeXt-Base consists of a convolutional layer with a kernel size of 4×4, a LayerForm layer, four ConvNeXt blocks with 3, 3, 27, and 3 CNBlocks respectively, an adaptive average pooling layer, and a classifier. Each CNBlock includes a depth convolutional layer with a kernel size of 7×7, a LayerForm layer, and two pointwise convolutional layers with a kernel size of 1×1 implemented by linear layers. The classifier layer consists of a LayerForm layer and a fully connected layer with 1000 neurons, and the output vector of the fully connected layer is modified from 1000 to the number of disease categories C.

[0138] Furthermore, a CDSS framework is constructed, including a medical rule base, an identification model base, and a decision engine module, which is used to provide treatment plans according to the identification results. The construction steps of the medical rule base include:

[0139] 1) Collect medical knowledge entries and sort out the clinical diagnosis and treatment paths of reagents;

[0140] 2) Locate the decision-making points in the clinical diagnosis and treatment process and generate medical rule content;

[0141] 3) Generate rule trigger scenarios, rule verification and testing, and package the rules;

[0142] The identification model base identifies the bleeding site information in the lesion image based on the TransUNet segmentation model. According to the identified bleeding site information, the decision engine module realizes dynamic decision recommendation based on the medical rule base.

[0143] Specifically, the bleeding site information in the lesion image is identified based on the TransUNet segmentation model, providing a judgment basis for the decision engine module. The decision engine module obtains the diagnosis and treatment path in the medical rule base according to the bleeding site information, locates the decision-making points in the clinical diagnosis and treatment process, and thus generates a decision plan recommendation.

[0144] To sum up, the thoracoscopic and laparoscopic lesion imaging detection system grayscales the collected endoscopic images, extracts the gray-level co-occurrence matrix features, constructs a classifier according to the least squares support vector machine algorithm, analyzes the endoscopic images in real time, automatically detects and classifies bleeding points or suspected bleeding areas, and at the same time combines the light projection algorithm to perform three-dimensional reconstruction on the endoscopic images, providing data support for the auxiliary positioning of the bleeding site;

[0145] Meanwhile, based on the learning samples of lesion images in the cloud data platform, the ConvNeXt-Base model is used to learn global features and local features, and then the TransUNet segmentation model is utilized to automatically segment the diseased area and non-diseased area in the lesion image, realizing the automatic recognition of bleeding diseased position information, so as to provide a judgment basis for the decision engine module. The decision engine module obtains the diagnosis and treatment path according to the medical rule base, locates the decision points in the clinical diagnosis and treatment process, automatically generates decision plan recommendations, and assists doctors in formulating and optimizing the surgical plan.

[0146] The relevant modules involved in this system are all hardware system modules or functional modules that combine computer software programs or protocols in the prior art with hardware. The computer software programs or protocols themselves involved in this functional module are all well-known technologies to those skilled in the art, and they are not the improvements of this system; the improvement of this system lies in the interaction relationship or connection relationship between each module, that is, the overall structure of the system is improved to solve the corresponding technical problems to be solved by this system.

[0147] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A thoracoscopic and laparoscopic disease imaging detection system, characterized in that: It includes a laparoscope imaging module to realize image acquisition, image recording and image transmission functions inside the human chest and abdominal cavity; The exploration and analysis module analyzes the laparoscope images in real time and automatically detects the bleeding location information or suspected bleeding point location; Three-dimensional construction module, which assists in locating the bleeding site based on three-dimensional reconstruction technology; The decision analysis module extracts the characteristics of bleeding sites based on machine learning, builds a bleeding site recognition model, and provides treatment plans based on the recognition results.

2. A thoracoscopic and laparoscopic disease imaging detection system according to claim 1, characterized in that: The laparoscope imaging module includes an integrated wireless high-definition camera submodule, an LED cold light source submodule, a wireless function display and control submodule, and a main power control submodule. The wireless high-definition camera submodule includes a laparoscope body, and an optical interface and a high-definition surgical field camera are configured for the laparoscope body. The optical interface adopts a C-Mount standard interface with optical zoom, and the optical zoom range is set to 14 to 32 mm. The high-definition surgical field camera is provided with a video recording unit for storing image data in a local device or in the cloud. The high-definition surgical field camera is also provided with a cloud interaction unit for performing remote teaching and guidance through real-time cloud interaction based on the image data stored in the cloud. The high-definition surgical field camera is configured with wired output and wireless output modes.

3. A thoracoscopic and laparoscopic disease imaging detection system according to claim 2, characterized in that: The LED cold light source submodule includes a xenon lamp cold light source and a light source power supply unit, the color temperature range of the xenon lamp cold light source is 3000K to 7000K, the light source power supply unit is divided into a self-powered mode and a wireless power supply mode, according to the self-powered mode, the light source power supply unit is configured with an energy storage battery electrically connected to the xenon lamp cold light source; according to the wireless power supply mode, the light source power supply unit is configured with an antenna transmitter and a transformer receiver connected to the antenna transmitter signal, and the transformer receiver is electrically connected to the xenon lamp cold light source; The wireless function display and control submodule adopts a wireless point-to-point RF transmission mode, which includes a RF transceiver equipped with an antenna, a display equipped with a main control panel, and a mobile terminal equipped with a control APP. The display and the mobile terminal are both connected to the high-definition surgical field camera signal through the RF transceiver. The mobile terminal is selected from one of a mobile phone, a tablet and a laptop computer.

4. A thoracoscopic and laparoscopic disease imaging detection system according to claim 3, characterized in that: The exploration and analysis module analyzes the laparoscope image based on the least squares support vector machine algorithm, and the specific steps include: S1. Connect to the image output port of the display, collect the laparoscope image in real time and grayscale the image based on the weighted average algorithm; S2. Extracting gray-level co-occurrence matrix features based on the generated grayscale image and initializing the least squares support vector machine algorithm parameters; S3. Obtain learning samples of lesion images in the cloud data platform, and construct an image lesion area classifier based on the learning classification characteristics of the least squares support vector machine; S4. Input the acquired laparoscope image into the image lesion area classifier, and output the lesion recognition result.

5. A thoracoscopic and laparoscopic disease imaging detection system according to claim 4, characterized in that: In step S1, the laparoscope image is grayed based on a weighted average algorithm, and the image graying processing formula is: Gray=0.299R+0.587G+0.114B; In the above formula, R, G, and B represent the red, green, and blue component values ​​in the laparoscope image in RGB format, respectively, and Gray represents the image grayscale value. Each component value in the laparoscope image is automatically traversed according to the image grayscale processing formula to output the laparoscope grayscale image; In step S2, a grayscale co-occurrence matrix is ​​constructed based on the laparoscope grayscale image, and the specific steps include: a. Let any point i in the image be represented by (x, y), then let a point j that deviates from point i be represented by (x+a, y+b), and let point i and point j form a point pair; b. Let the gray value of the point pair be (f1, f2). Assuming the maximum gray level of the laparoscope gray image is L, there are L˙L combinations of f1 and f2. c. Count the number of occurrences of each (f1, f2) value in the laparoscope grayscale image and arrange them into a square matrix, then normalize the total number of occurrences of (f1, f2) to obtain the probability P ij , then the gray-level co-occurrence matrix is ​​obtained; The eigenvalues ​​are calculated based on the obtained gray level co-occurrence matrix, including: 1) Energy characteristics Asm: 2) Contrast feature Con: 3) Related Features IDE: 4) Entropy characteristic Ent:

6. A thoracoscopic and laparoscopic disease imaging detection system according to claim 4, characterized in that: In step S3, the learning samples of the lesion image are mapped to the high-dimensional feature space based on the nonlinear mapping function and feature classification is performed. The target optimization of the lesion image area division is expressed as: In the above formula, R(ω) is the structural risk, γ is the regularization factor, ξ i In order to allow for errors, the Lagrangian method is introduced to solve the optimization problem, which can be expressed as: In the above formula, α i is the Lagrange multiplier, and the optimization conditions are set as According to the optimization conditions, we can get: By eliminating ω and ξ, we can get Defining ZZ T =(p ij ) N×N , then the kernel function Therefore, the optimal classification function is obtained, that is, the classifier is y(x) 7. A thoracoscopic and laparoscopic disease imaging detection system according to claim 1, characterized in that: The three-dimensional construction module realizes three-dimensional reconstruction based on a ray projection algorithm. The specific steps of the ray projection algorithm include: L1. Read the volume data based on the VTK function library programming, and perform filtering and denoising on the volume data based on the median filtering algorithm. The steps of the median filtering algorithm include: a. Set a median filter with a template size of N*N, whose output value should be greater than or equal to (N*N-1) / 2 in the template, and also less than or equal to (N*N-1) / 2 in the template, where N is generally an odd value; b. The median filter algorithm formula is expressed as g(x,y)=median[f(s,t)], which moves the template in the image and makes the center of the template coincide with a pixel position in the image; c. Read the grayscale values ​​of the corresponding pixels under the template, arrange them in ascending order, determine the middle value of the grayscale value sorting, and assign the middle value to the pixel value at the center of the template; L2. Classify the laparoscope images according to the volume data attributes, and assign corresponding color values ​​and opacity to each type of laparoscope image data with reference to the set color mapping table and opacity table; L3. Resample along the line of sight according to the ray tracing method, accumulate the color value and opacity of each pixel on the image plane, synthesize the three-dimensional image and output it for display.

8. A thoracoscopic and laparoscopic disease imaging detection system according to claim 1, characterized in that: Based on machine learning, the bleeding site features are extracted and a bleeding site recognition model is constructed. The specific steps include: T1. According to the learning samples of the lesion image in the cloud data platform, the input image is set to I, the label vector is set to L, and the output of the fully connected layer is set to P f (c│I); T2. Initialize the parameters W of the global and local branches according to the ImageNet pre-training parameters of the ConvNeXt-Base model g and W l , randomly initialize the fully connected layer; T3. Based on the input image I, learn the global feature F g , calculate the final fully connected layer parameters P of the global branch g (c│I) and the global average pooling layer parameter Pool g , optimize the global branch parameter W according to the following formula g : Among them l c is the true label of the class, σ is the sigmoid function, and C is the number of pathological types; T4. Calculate the mask M based on the input image I prob , learn local features F l , calculate the final fully connected layer parameters P of the local branch l (c│I) and the global average pooling layer parameter Pool l , optimize the local branch parameter W according to the formula in T3 above l ; T5. Connect to Pool g and pool l , calculate P f (c│I), optimize W according to the formula in T3 above g ; T6. Construct the TransUNet segmentation model with the minimum Dice loss to automatically segment the diseased area and non-disease area of ​​the input image I. The Dice loss formula is expressed as: Among them, M gt is the real mask, M prob is the prediction mask; The steps of constructing the TransUNet segmentation model include: 1) Based on CNN feature extraction: The pre-trained ResNet50 is selected as the encoder backbone network, and the image features are extracted through multiple convolutional layers and residual blocks to generate multiple feature maps with different resolutions; 2) Combined with Transformer module: According to the multi-head self-attention mechanism of Transformer introduced by the encoder, the 1×1 patches in the feature map extracted by ResNet50 are converted into sequences through patch embedding and used as Transformer input; 3) Restoring spatial resolution: Introducing a cascade upsampler consisting of multiple upsampling steps, each of which includes a 2x upsampling operation, a 3×3 convolutional layer, and a ReLU activation layer to restore the spatial resolution in the output of the Transformer encoder; 4) Fusion of high- and low-resolution features: The decoder recovers the segmentation mask in the output of the Transformer encoder according to the CUP, and combines the output feature map of the Transformer with the CNN feature map, restores the resolution of the original image through the upsampling step, generates the segmentation result through the convolution layer, and adds a sigmoid activation function to scale the output pixel value to between 0 and 1; T7. The mask size is adjusted to 1×h×w (h and w represent the height and width of the global feature map, respectively) through the adaptive average pooling function. The pixel value of the diseased area is set to 1 and the pixel value of the non-disease area is set to 0 through the argmax function. The local feature map (c×h×w) is obtained by element-by-element multiplication. The element-by-element multiplication calculation formula is:

9. A thoracoscopic and laparoscopic disease imaging detection system according to claim 8, characterized in that: Construct a CDSS framework, including a medical rule base, a recognition model base and a decision engine module, for providing a treatment plan based on the recognition results. The steps of constructing the medical rule base include: 1) Collect medical knowledge items and sort out the clinical diagnosis and treatment pathways of reagents; 2) Locate decision points in the clinical diagnosis and treatment process and generate medical rule content; 3) Generate rule triggering scenarios, rule verification and testing, and package the rules; The recognition model library recognizes the bleeding site information in the lesion image based on the TransUNet segmentation model, and according to the recognized bleeding site information, the decision engine module implements dynamic decision recommendation based on the medical rule library.

Images