Laparoscopic surgery image segmentation method based on lightweight neural network and terminal equipment thereof

Abstract

The invention relates to the technical field of image processing, and particularly discloses a laparoscopic surgery image segmentation method based on a lightweight neural network and terminal equipment thereof, and the method comprises the following steps: S1, obtaining a laparoscopic surgery image and video data, and generating a laparoscopic surgery database; s2, image preprocessing is conducted on the laparoscopic surgery database through a preset automatic encoder, and the image size is modified through a letterbox method; and S3, constructing a laparoscope image semantic segmentation network based on a Scaleform algorithm. And S4, segmenting the laparoscopic surgery database to train the laparoscopic image semantic segmentation network, storing the trained model, and performing real-time test on the test set. And S5, deploying the trained laparoscopic image semantic segmentation network model to a GPU module, and processing a real-time picture transmitted from a client.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a laparoscopic surgery image segmentation method based on a lightweight neural network and a terminal device thereof. Background Art

[0002] Compared with traditional open hysterectomy, laparoscopic hysterectomy has the advantages of less trauma and faster postoperative recovery of patients, and the patients have comparable oncological benefits. Laparoscopic hysterectomy has become a widely performed and first recommended surgical method. However, due to the narrow field of view under laparoscopy and the lack of global anatomical field of the uterus, the operation of laparoscopic instruments will lead to weakened tactile feedback. In addition, the blood vessels inside the uterus are staggered and variations are common. The safe and accurate completion of laparoscopic hysterectomy has high requirements for operators. Therefore, how to assist doctors to accurately identify the key anatomical structures inside the uterus during hysterectomy is a key and major clinical need for the safe implementation and popularization of laparoscopic hysterectomy. At present, the existing technical solutions are difficult to accurately segment the uterine arteries and veins with similar visual features, and cannot achieve real-time segmentation in laparoscopic surgery videos.

[0003] Currently, the widely used frameworks based on convolutional neural networks extract local features of multi-target anatomical structures through convolution and predict them in the input image, achieving good results. However, when extracting global features of images, the frameworks based on neural convolutional networks often use a larger receptive field to achieve this. However, due to storage limitations, the image will be divided into image blocks before input. The global features extracted using a larger receptive field cannot be fully utilized, which is not conducive to using global features to identify anatomical structures with similar visual features. For example, the segmentation effect of targets with similar visual features such as the left hepatic pedicle and left hepatic vein in the liver is not ideal. Moreover, when using convolutional networks to extract global features of images, more parameters will be generated, which cannot guarantee the efficiency of computational reasoning in real-time laparoscopic surgery videos, and it is difficult to perform real-time segmentation of multi-target anatomical structures in real-time laparoscopic surgery videos.

[0004] The patent document with the publication number "CN116205928A" discloses an image segmentation processing method, device, equipment, and medium for laparoscopic surgical video, wherein the method includes: obtaining video data during laparoscopic resection surgery, performing frame processing on the video data to form sequence frame image data; inputting the sequence frame image data of multiple consecutive frames into the multi-feature fusion module for feature extraction processing to obtain the extracted fusion feature image data of multiple consecutive frames; inputting the extracted fusion feature image data of multiple consecutive frames into the TimeSformer deep learning framework for image segmentation processing to obtain image segmentation results. Although the above patent can process the sequence frame image data of multiple consecutive frames through the TimeSformer deep learning framework to achieve real-time segmentation, the patent still does not solve the narrow field of view in laparoscopic surgery, the lack of global anatomical field of view, and the difficulty in accurately segmenting anatomical structures with similar visual features such as the hepatic pedicle and hepatic vein, and cannot effectively segment in real-time surgical videos.

[0005] Therefore, the applicant proposes a laparoscopic surgery image segmentation method based on a lightweight neural network. Summary of the invention

[0006] The main purpose of this application is to provide a laparoscopic surgical image segmentation method based on a lightweight neural network, aiming to improve the accuracy and real-time performance of key anatomical structure identification during surgery while reducing the demand for computing resources.

[0007] To achieve the above objectives, the present application provides a laparoscopic surgery image segmentation method based on a lightweight neural network, comprising the following steps:

[0008] S1. Obtain laparoscopic surgery images and video data, and generate a laparoscopic surgery database.

[0009] S2. The laparoscopic surgery database is image preprocessed using a preset autoencoder and the image size is modified using the letterbox method.

[0010] S3. Construct a laparoscopic image semantic segmentation network based on the Scaleformer algorithm.

[0011] S4. Segment the laparoscopic surgery database to train a laparoscopic image semantic segmentation network, save the trained model, and perform real-time testing on the test set.

[0012] S5. Deploy the trained laparoscopic image semantic segmentation network model to the GPU module to process the real-time images transmitted from the client.

[0013] Among them, the segmentation of the laparoscopic surgery database in the step S4 is specifically as follows: the preprocessed image data is input into the laparoscopic image semantic segmentation network, and the cross entropy loss function is used to calculate the difference between the model output and the true label, and then the gradient of the cross entropy loss function with respect to the model parameters is calculated by the back propagation algorithm, and the model parameters are updated in the opposite direction of the gradient. Subsequently, the above steps of image data input, calculation of the difference between the model output and the true label, back propagation calculation of the model parameter gradient and updating the model parameters are repeated, the parameter data of the model is iterated, and the training data is gradually fitted into the model.

[0014] Preferably, the cross entropy loss function in step S4 is defined as:

[0015]

[0016] in, K Indicates the total number of categories, y i，j represents the true label (0 or 1) of the jth category of the ith sample, is the probability value of the jth category of the i-th sample predicted by the model.

[0017] Preferably, the calculation and update of the model parameter gradient in step S4 include the following steps:

[0018] S4-1. First, forward propagation is performed to process the input image through the laparoscopic image semantic segmentation network to obtain the predicted probability distribution of each category.

[0019] S4-2. Use the cross entropy loss function to calculate the difference between the predicted probability and the true label, calculate the gradient of the loss function relative to the model output, and use the chain rule to propagate backward layer by layer starting from the output layer until it reaches the input layer.

[0020] S4-3. Update the weights and biases of the model according to the model output gradient obtained above through the SGD optimization algorithm.

[0021] S4-4. Repeat the above process, using batches of sample data for training in each iteration until the model converges or reaches the predetermined number of training rounds.

[0022] Preferably, the predetermined number of training rounds is set based on an early stopping method.

[0023] Preferably, the step S2 comprises the following steps:

[0024] S2-1. The autoencoder learns the low-dimensional representation of the input image, removes the noise of the laparoscopic surgery image, retains the key information, and performs feature enhancement processing to obtain the original image;

[0025] S2-2. Use the letterbox method to adjust the size of the above image, add blank pixels to the edge of the image, and perform binarization on it, scaling the pixel value range from [0,255] to [0,1] to obtain a grayscale image.

[0026] S2-3. Input the grayscale image obtained in the above steps into the autoencoder, and upsample the low-dimensional features of the grayscale image through a series of convolutional layers, gradually restoring it to the same size as the original image.

[0027] S2-4. The key points and edge information in the grayscale image after the above steps are enhanced by the activation function Sigmoid, and the feature map of the corresponding level in the encoder is directly connected to the decoder through the jump connection.

[0028] S2-5. A convolutional layer converts the feature map into an output with the same number of channels as the original image, and normalizes it to obtain the final reconstructed image.

[0029] Preferably, the step S2 further comprises: calculating a mean square error (MSE) between the reconstructed image and the original image, and performing a quality assessment on the reconstructed image C finally obtained.

[0030] Preferably, the calculation formula of the mean square error MSE is:

[0031]

[0032] Where n is the number of samples, is the value predicted by the model, y i , is the true value of the ii-th sample.

[0033] Preferably, the step between step S2-2 and step S2-3 further includes: performing connected domain analysis on the binarized image, calculating geometric parameters of each connected domain, and screening out connected domains with specific shapes or sizes.

[0034] Preferably, the method further includes: performing opening and closing operations on each connected domain marked in the grayscale image to remove small noise points and fill small holes inside the object.

[0035] To achieve the above-mentioned objectives, the present application also provides a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, the steps of the laparoscopic surgical image segmentation method based on a lightweight neural network are implemented as described above.

[0036] The present invention has the following technical effects:

[0037] The image is preprocessed by autoencoder, and the letterbox method is used to adjust the image size and standardize the image input data, which reduces the performance degradation of the model when processing unseen data and avoids the data heterogeneity and inconsistency during model training. A laparoscopic image semantic segmentation network is constructed through the lightweight neural network Scaleformer, and the model is trained using a laparoscopic surgery database specially generated for laparoscopic surgery image semantic segmentation.

[0038] During the model training process, the cross-entropy loss function is used to evaluate the difference between the model output and the true label, and the model parameters are updated through the back-propagation algorithm to ensure that the model learns features that are highly relevant to the actual surgical situation and improve the segmentation performance of the model in specific scenarios. Furthermore, the use of the cross-entropy loss function can effectively measure the difference between the model output and the true label, and the back-propagation algorithm can adjust the model parameters according to this difference, thereby ensuring that the model continues to improve during the iteration process, optimizing the model performance and enabling it to fit complex anatomical structures.

[0039] Real-time testing is performed on the test set to verify the generalization ability and practicality of the model, and deploying the model on the GPU module can achieve rapid processing of real-time transmitted images, meeting the high timeliness requirements during surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the steps of the laparoscopic surgery image segmentation method based on lightweight neural network;

[0041] Figure 2 is a flowchart of step S4 of the laparoscopic surgery image segmentation method based on lightweight neural network;

[0042] Figure 3 A schematic diagram of obtaining the original image of laparoscopic surgery in step S2-1;

[0043] Figure 4 To obtain a laparoscopic surgery grayscale image in step S2-2;

[0044] Figure 5 Acquiring a laparoscopic surgery reconstructed image for step S2-5;

[0045] Figure 6 Acquire the final image of laparoscopic surgery for step S2-5;

[0046] Figure 7 A flowchart of step S2 described in one embodiment of the present application;

[0047] Figure 8 A flowchart of step S2 described in another embodiment of the present application;

[0048] Fig. 9 This is a schematic block diagram of the internal structure of a terminal device according to an embodiment of the present application.

[0049] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0051] In addition, if the description of "first", "second", etc. is involved in this application, it is only used for descriptive purposes (such as for distinguishing the same or similar elements), and cannot be understood as indicating or implying its relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0052] See also Figure 1 , a laparoscopic surgery image segmentation method based on a lightweight neural network specifically includes the following steps:

[0053] S1. Obtain laparoscopic surgery images and video data, and generate a laparoscopic surgery database.

[0054] S2. The laparoscopic surgery database is image preprocessed using a preset autoencoder and the image size is modified using the letterbox method.

[0055] S3. Construct a laparoscopic image semantic segmentation network based on the Scaleformer algorithm.

[0056] S4. Segment the laparoscopic surgery database to train a laparoscopic image semantic segmentation network, save the trained model, and perform real-time testing on the test set.

[0057] S5. Deploy the trained laparoscopic image semantic segmentation network model to the GPU module to process the real-time images transmitted from the client.

[0058] Among them, the segmentation of the laparoscopic surgery database in the step S4 is specifically as follows: the preprocessed image data is input into the laparoscopic image semantic segmentation network, and the cross entropy loss function is used to calculate the difference between the model output and the true label, and then the gradient of the cross entropy loss function with respect to the model parameters is calculated by the back propagation algorithm, and the model parameters are updated in the opposite direction of the gradient. Subsequently, the above steps of image data input, calculation of the difference between the model output and the true label, back propagation calculation of the model parameter gradient and updating the model parameters are repeated, the parameter data of the model is iterated, and the training data is gradually fitted into the model.

[0059] In this embodiment, the trained laparoscopic image semantic segmentation network model is saved and tested in real time on a test set, and then the model is deployed on a GPU module to process real-time images transmitted from the client, thereby achieving the real-time and practicality of laparoscopic surgical image segmentation. Image preprocessing is performed through a preset autoencoder. At the same time, the letterbox method is used to modify the image size, so that the image data input into the laparoscopic image semantic segmentation network is more standardized, which is conducive to improving the accuracy of image segmentation.

[0060] Secondly, the laparoscopic surgery database is segmented to train the laparoscopic image semantic segmentation network, and the cross-entropy loss function is used to calculate the difference between the model output and the true label. The gradient of the cross-entropy loss function with respect to the model parameters is calculated through the back-propagation algorithm, and the model parameters are updated in the opposite direction of the gradient, which further effectively optimizes the training process of the model. Compared with the prior art, the TimeSformer deep learning framework is used to process multiple consecutive frames of sequence frame image data to achieve real-time segmentation. The above steps can also be used in combination to accurately segment anatomical structures with similar visual features in surgical images, such as hepatic pedicles and hepatic veins, in real-time surgical videos.

[0061] Specifically, we first preprocessed the images using an autoencoder, adjusted the image size using the letterbox method, standardized the image input data, reduced the performance degradation of the model when processing unseen data, and avoided data heterogeneity and inconsistency during model training. We then built a laparoscopic image semantic segmentation network using a lightweight neural network Scaleformer, and used the laparoscopic surgery database specifically generated for laparoscopic surgery image semantic segmentation for model training.

[0062] During the model training process, the cross-entropy loss function is used to evaluate the difference between the model output and the true label, and the model parameters are updated through the back-propagation algorithm to ensure that the model learns features that are highly relevant to the actual surgical situation and improve the segmentation performance of the model in specific scenarios. Furthermore, the use of the cross-entropy loss function can effectively measure the difference between the model output and the true label, and the back-propagation algorithm can adjust the model parameters according to this difference, thereby ensuring that the model continues to improve during the iteration process, optimizing the model performance and enabling it to fit complex anatomical structures.

[0063] Finally, real-time testing is performed on the test set to verify the generalization ability and practicality of the model. Deploying the model to the GPU module can realize rapid processing of real-time transmitted images, meeting the high requirements for timeliness during surgery.

[0064] In addition, the use of Scaleformer algorithms to construct lightweight neural networks can maintain high segmentation accuracy while having fewer parameters and computational complexity, thereby improving the efficiency of image segmentation.

[0065] In one embodiment, the cross entropy loss function in step S4 is defined as:

[0066]

[0067] in, K Indicates the total number of categories, y i，j represents the true label (0 or 1) of the jth category of the ith sample, is the probability value of the jth category of the i-th sample predicted by the model.

[0068] In this embodiment, the error is quantified by calculating the logarithmic loss between the predicted probability and the true label using the cross entropy loss function, so that the model learning can predict the correct probability of each category, and continuously optimize its parameters during the training process to minimize the cross entropy loss, so that the predicted probability is as close to the true label as possible, thereby improving the accuracy of the model in the classification task, and at the same time facilitating the convergence of subsequent optimization algorithms such as gradient descent, so that the parameter space can be searched more efficiently to find the parameter configuration that minimizes the loss.

[0069] Secondly, the cross entropy loss function provides the model with a continuous penalty for misclassification. This continuous penalty mechanism allows the model to adjust parameters more smoothly during training, rather than causing the parameter adjustment to be too abrupt through a fixed penalty value. Furthermore, because the cross entropy loss function provides a continuous penalty, the model exhibits better robustness when facing noisy data or outliers. The model will not over-adjust parameters due to extreme penalties for individual errors, thereby being able to maintain stable performance in noisy datasets.

[0070] In addition, by minimizing the cross entropy loss, the model can better fit the data during the learning process, thereby improving its prediction performance on unseen samples.

[0071] See also Figure 2 In one embodiment, the step S4 regarding the model parameter gradient calculation and updating specifically includes the following steps:

[0072] S4-1. First, forward propagation is performed to process the input image through the laparoscopic image semantic segmentation network to obtain the predicted probability distribution of each category.

[0073] S4-2. Use the cross entropy loss function to calculate the difference between the predicted probability and the true label, calculate the gradient of the loss function relative to the model output, and use the chain rule to propagate backward layer by layer starting from the output layer until it reaches the input layer.

[0074] S4-3. Update the weights and biases of the model according to the model output gradient obtained above through the SGD optimization algorithm.

[0075] S4-4. Repeat the above process, using batches of sample data for training in each iteration until the model converges or reaches the predetermined number of training rounds.

[0076] Furthermore, the predetermined number of training rounds is set based on the early stopping method. Specifically, the patience parameter of early stopping is first set, that is, the number of consecutive training cycles without improvement allowed, and a counter is defined to record the number of cycles without improvement. Then, at the end of each training cycle, the model performance is evaluated using an independent validation set.

[0077] That is, if the verification performance of the current cycle is improved compared to the previous cycle, the counter is reset and training continues. If the verification performance does not improve in multiple consecutive cycles, the counter is increased. When the counter reaches the preset patience parameter, the training is terminated early and the model of the current cycle is saved as the final model.

[0078] In the embodiment, through the forward propagation and back propagation algorithms, the model can accurately learn the complex mapping relationship between the input image and the category prediction probability, and use the gradient calculated by the cross entropy loss function, combined with the SGD optimization algorithm, to effectively update the weights and biases of the model and minimize the loss function, thereby making the model's prediction closer to the true label.

[0079] Secondly, by introducing the early stopping method to set the predetermined number of training rounds, the training can be terminated in time when the verification performance stops improving, effectively preventing the model from overfitting on the training data and improving the generalization ability of the model. The model is saved after the continuous cycle with no improvement in verification performance, ensuring that the saved model is the optimal version on the verification data, thereby obtaining better practical application results.

[0080] In addition, the "patience parameter" in the early stopping method is an adjustable hyperparameter, that is, it can be adjusted according to the characteristics of specific tasks and data sets to find the best time to terminate training.

[0081] In the embodiment, the step S2 specifically includes the following steps:

[0082] S2-1. First, the autoencoder learns the low-dimensional representation of the input image, removes the noise of the laparoscopic surgery image, retains the key information, and performs feature enhancement to obtain the original image, see Figure 3 ;

[0083] S2-2. Next, the letterbox method is used to adjust the size of the above image, add blank pixels to the edge of the image, and perform binarization processing on it, scaling the pixel value range from [0,255] to [0,1] (i.e., if the pixel value is lower than or equal to the threshold, it is set to 0 (black); if the pixel value is higher than the threshold, it is set to 1 (white)), and a grayscale image is obtained, see Figure 4 .

[0084] S2-3. Then, the grayscale image obtained in the above steps is input into the autoencoder, and the low-dimensional features of the grayscale image are upsampled through a series of convolutional layers, gradually restoring it to the same size as the original image.

[0085] S2-4. Subsequently, the key point and edge information in the grayscale image after the above steps are enhanced by the activation function Sigmoid, and the feature map of the corresponding level in the encoder is directly connected to the decoder through the jump connection.

[0086] S2-5. Finally, a convolutional layer converts the feature map into an output with the same number of channels as the original image and normalizes it to obtain the final reconstructed image, see Figure 5-Figure 6 .

[0087] In the embodiment, the low-dimensional representation of the input image is learned by the autoencoder, which can effectively remove the noise in the laparoscopic surgery image while retaining the key information, thereby improving the data quality of the subsequent image processing steps, reducing the interference of noise on the model training, and at the same time, the letterbox method is used to adjust the image size and add blank pixels at the edge, so that the size of the input image is standardized, and the letterbox method is used to adjust the image size and add blank pixels at the edge, so that the size of the input image is standardized, which is convenient for subsequent convolution operations. Then, through the upsampling operation of the autoencoder, the spatial resolution of the image is gradually restored, ensuring that the output image has the same size as the original image, and further maintaining the clarity of the details in the image. Then, the use of the Sigmoid activation function enhances the key points and edge information in the grayscale image, and through the connection mode of directly passing the feature map in the encoder to the decoder, the detail information of the original image is retained, and the information that may be lost during the encoding process is reduced. Finally, the pixel value range of the output image is ensured to be consistent through normalization processing, so that the model converges faster during training, and improves the accuracy and robustness of the model in the task of semantic segmentation of laparoscopic images.

[0088] In summary, the entire preprocessing process reduces manual intervention through automated steps, improves processing speed, and reduces possible human errors. In addition, the consistency of input data is ensured by standardizing image size and number of channels.

[0089] In one embodiment, the step S2 further includes: calculating the mean square error MSE between the reconstructed image and the original image, and performing quality assessment on the reconstructed image C finally obtained.

[0090] Specifically, the quality of the reconstructed image C is evaluated by a preset mean square error (MSE) calculation formula.

[0091] When the MSE value is greater than the preset threshold, the quality of the reconstructed image C is low, and the back propagation algorithm is required to update the weights and biases of the autoencoder layer by layer according to the gradient of the MSE loss function until the MSE value drops below the preset threshold.

[0092] When the MSE value is less than the preset threshold, the performance of the autoencoder in image preprocessing meets expectations.

[0093] The calculation formula of the above mean square error MSE is:

[0094]

[0095] Where n is the number of samples, is the value predicted by the model, y i , is the true value of the ii-th sample.

[0096] Furthermore, the linear regression model established in step S2 is specifically: Where W is the weight, X is the input, and b is the bias.

[0097] The gradient calculation formula of the MSE loss function is as follows:

[0098] Gradient of weight W:

[0099] The gradient of bias b:

[0100] In the embodiment, quality assessment is performed by calculating the mean square error (MSE) between the reconstructed image and the original image, and the weights and biases of the autoencoder are updated using the back propagation algorithm and the gradient of the MSE loss function until the MSE value drops below a preset threshold, thereby achieving the purpose of improving image preprocessing quality and model performance.

[0101] Specifically, when the MSE value is greater than the preset threshold, it indicates that the quality of the reconstructed image C is low, and the weights and biases of the autoencoder need to be updated layer by layer according to the gradient of the MSE loss function through the back propagation algorithm. The above process is repeated until the MSE value drops below the preset threshold, thereby ensuring that the quality of the reconstructed image C meets the requirements. When the MSE value is less than the preset threshold, it means that the performance of the autoencoder in image preprocessing has met expectations, and there is no need to further update the weights and biases.

[0102] In addition, the established linear regression model can be trained multiple times through minimizing the MSE loss function, so that it can accurately predict the value of the sample.

[0103] In one of the embodiments, the steps between step S2-2 and step S2-3 also include: performing connected domain analysis on the binarized image, calculating geometric parameters of each connected domain, and screening out connected domains with specific shapes or sizes.

[0104] See also Figure 7 Specifically, the connected domain screening includes the following steps:

[0105] S2-3a. Traverse each pixel in the binary image, classify adjacent pixels with the same pixel value into the same connected domain, and assign a unique label to each connected domain;

[0106] S2-3b calculates the area, perimeter and geometric parameters of the circumscribed rectangle of each marked connected domain;

[0107] S2-3c. According to a predetermined shape or size standard, the geometric parameters obtained above are screened to retain the connected domain that meets the requirements;

[0108] S2-3d. Remove the connected domains that do not meet the conditions from the binary image.

[0109] See also Figure 8 In one embodiment, the step S2-3a and step S2-3b further include: performing opening and closing operations on each connected domain marked in the grayscale image to remove small noise points and fill small holes inside the object.

[0110] In an embodiment, by performing opening and closing operations on each connected domain marked in the grayscale image, small noise points can be effectively removed and small holes inside the object can be filled. While cleaning up the random noise in the image, the key details in the image can be maintained or even enhanced. At the same time, by calculating the geometric parameters of each connected domain (such as area, perimeter, and circumscribed rectangle, etc.) and screening based on pre-set shape or size standards, connected domains that meet specific requirements can be accurately identified and retained, further improving the ability to accurately extract target scenes of specific shapes or sizes from complex backgrounds.

[0111] Secondly, by removing the connected domains that do not meet the conditions from the binary image, the amount of data that needs to be analyzed in the subsequent processing steps is reduced, the overall image processing time is shortened, the demand for computing resources is alleviated, and the image processing algorithm is made more efficient.

[0112] In addition, this connected domain screening method is highly adaptable and can adjust the shape or size standards according to different application scenarios to meet image processing requirements under different conditions.

[0113] In addition, a terminal device is also provided in an embodiment of the present application. The internal structure of the terminal device can be as follows: Fig. 9 As shown. The terminal device includes a processor, a memory, a communication interface and a database connected via a system bus. The processor is used to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the terminal device is used to store data called by the computer program. The communication interface of the terminal device is used to communicate data with an external terminal. The input device of the terminal device is used to receive signals input by an external device. When the computer program is executed by the processor, the laparoscopic surgery image segmentation method based on a lightweight neural network as described in the above embodiment is implemented.

[0114] Those skilled in the art will understand that Fig. 9 The structure shown in is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the terminal device to which the solution of the present application is applied.

[0115] In addition, the present application also proposes a readable storage medium, the readable storage medium includes a computer program, and when the computer program is executed by a processor, the steps of the laparoscopic surgery image segmentation method based on a lightweight neural network as described in the above embodiment are implemented. It can be understood that the readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0116] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment of the laparoscopic surgical image segmentation method based on a lightweight neural network can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the processes of the above-mentioned embodiments of the laparoscopic surgical image segmentation method based on a lightweight neural network. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0117] It should be noted that, in this article, the terms "include", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, device, article or laparoscopic surgical image segmentation method based on a lightweight neural network that includes a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, device, article or laparoscopic surgical image segmentation method based on a lightweight neural network. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, device, article or laparoscopic surgical image segmentation method based on a lightweight neural network that includes the element.

[0118] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A laparoscopic surgery image segmentation method based on a lightweight neural network, characterized in that: The following steps are involved: S1. Obtain laparoscopic surgery images and video data, and generate a laparoscopic surgery database; S2. Preprocess the images of the laparoscopic surgery database using a preset autoencoder and modify the image size using the letterbox method; S3. Build a laparoscopic image semantic segmentation network based on Scaleformer algorithm; S4. Segment the laparoscopic surgery database to train the laparoscopic image semantic segmentation network, save the trained model, and perform real-time testing on the test set; S5. Deploy the trained laparoscopic image semantic segmentation network model to the GPU module to process the real-time images transmitted from the client; Among them, the segmentation of the laparoscopic surgery database in the step S4 is specifically as follows: the preprocessed image data is input into the laparoscopic image semantic segmentation network, and the cross entropy loss function is used to calculate the difference between the model output and the true label, and then the gradient of the cross entropy loss function with respect to the model parameters is calculated by the back propagation algorithm, and the model parameters are updated in the opposite direction of the gradient. Subsequently, the above steps of image data input, calculation of the difference between the model output and the true label, back propagation calculation of the model parameter gradient and updating the model parameters are repeated, the parameter data of the model is iterated, and the training data is gradually fitted into the model.

2. The laparoscopic surgery image segmentation method based on lightweight neural network according to claim 1 is characterized in that: The cross entropy loss function in step S4 is defined as: in, K Indicates the total number of categories, y i，j represents the true label (0 or 1) of the jth category of the ith sample, is the probability value of the jth category of the i-th sample predicted by the model.

3. The laparoscopic surgery image segmentation method based on lightweight neural network according to claim 2 is characterized in that: The calculation and update of the model parameter gradient in step S4 include the following steps: S4-1. First, forward propagation is performed to process the input image through the laparoscopic image semantic segmentation network to obtain the predicted probability distribution of each category; S4-2. Use the cross entropy loss function to calculate the difference between the predicted probability and the true label, calculate the gradient of the loss function relative to the model output, and propagate backward layer by layer from the output layer through the chain rule until it reaches the input layer; S4-3. Update the weights and biases of the model according to the model output gradient obtained above through the SGD optimization algorithm; S4-4. Repeat the above process, using batches of sample data for training in each iteration until the model converges or reaches the predetermined number of training rounds.

4. The laparoscopic surgery image segmentation method based on lightweight neural network according to claim 3 is characterized in that: The predetermined number of training rounds is set based on the early stopping method.

5. The laparoscopic surgery image segmentation method based on lightweight neural network according to claim 1, characterized in that: The step S2 comprises the following steps: S2-1. The autoencoder learns the low-dimensional representation of the input image, removes the noise of the laparoscopic surgery image, retains the key information, and performs feature enhancement processing to obtain the original image; S2-2. Use the letterbox method to adjust the size of the above image, add blank pixels to the edge of the image, and perform binarization on it, scaling the pixel value range from [0,255] to [0,1] to obtain a grayscale image; S2-3. Input the grayscale image obtained in the above steps into the autoencoder, and upsample the low-dimensional features of the grayscale image through a series of convolutional layers, gradually restoring it to the same size as the original image; S2-4. Use the activation function Sigmoid to enhance the key points and edge information in the grayscale image after the above steps, and directly connect the feature map of the corresponding level in the encoder to the decoder through the jump connection; S2-5. A convolutional layer converts the feature map into an output with the same number of channels as the original image, and normalizes it to obtain the final reconstructed image.

6. The laparoscopic surgery image segmentation method based on lightweight neural network according to claim 5, characterized in that: The step S2 further includes: calculating the mean square error MSE between the reconstructed image and the original image, and performing quality assessment on the reconstructed image C finally obtained.

7. The laparoscopic surgery image segmentation method based on lightweight neural network according to claim 6, characterized in that: The calculation formula of the mean square error MSE is: Where n is the number of samples, is the value predicted by the model, y i , is the true value of the ii-th sample.

8. The laparoscopic surgery image segmentation method based on lightweight neural network according to claim 7, characterized in that: The steps between step S2-2 and step S2-3 also include: performing connected domain analysis on the binarized image, calculating geometric parameters of each connected domain, and screening out connected domains with specific shapes or sizes.

9. The laparoscopic surgery image segmentation method based on lightweight neural network according to claim 8, characterized in that: Also includes: Open and close operations are performed on each connected domain marked in the grayscale image to remove small noise points and fill small holes inside the object.

10. A terminal device, characterized in that The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the laparoscopic surgical image segmentation method based on a lightweight neural network as described in any one of claims 1 to 7 are implemented.

Images