Model training method applied to endoscopic surgery image labeling, image recognition method, device and product

Abstract

The embodiment of the application provides a model training method, an image recognition method, a device and a product applied to endoscopic surgery image labeling, and relates to the technical field of image recognition. The method comprises the following steps: constructing an anatomical structure model according to sample image data, and generating an organ digital model by biological simulation according to the anatomical structure model; inputting the organ digital model into a virtual reality simulation scene, and acquiring a simulation sample set according to a preset sampling period when a simulation tool in the organ digital model moves and / or the simulation tool performs endoscopic surgery simulation work in the organ digital model under force feedback guidance; training an image labeling model constructed according to the simulation sample set and a video sample set, to obtain a trained image labeling model, so as to output labels of images collected in an endoscopic surgery process. The method is used to improve the sample data amount for model training, and reduce the burden of training sample collection and manual labeling.

Description

Technical Field

[0001] This application relates to the technical field of image recognition, and in particular to a model training method, image recognition method, device and product for laparoscopic surgical image annotation. Background Technology

[0002] Laparoscopic surgery is a surgical technique that involves inserting a light source, camera, and surgical instruments through a tiny incision, guided by images from a monitor. During laparoscopic surgery, the image recognition and decision-making capabilities of artificial intelligence (AI) models can be utilized to assist in intelligent surgical path planning, real-time intraoperative image navigation, automatic instrument posture correction, and intelligent prediction of surgical risks.

[0003] In existing technologies, the sample set used for training such laparoscopic surgery AI models needs to be obtained by manually annotating surgical recording videos. This is not only inefficient, but also results in a scarcity of sample data for some complex surgical scenarios, leading to low accuracy of AI models in image recognition during laparoscopic surgery. Summary of the Invention

[0004] This application provides a model training method, image recognition method, device, and product for laparoscopic surgical image annotation, which aims to increase the amount of sample data used for model training.

[0005] In a first aspect, embodiments of this application provide a model training method for image annotation in laparoscopic surgery, comprising:

[0006] First, a static anatomical structure model is constructed based on sample image data. Then, biomechanical parameters are assigned to this anatomical structure model, and physical deformation simulation is performed to generate an organ digital model that can be used for interactive operation. The anatomical structure model focuses on geometric and topological information, while the organ digital model is a model with physical interaction capabilities after biological simulation of the anatomical structure model. The organ digital model includes at least one simulated organ.

[0007] Inputting digital organ models into virtual reality simulation scenarios;

[0008] In a virtual reality simulation scenario, when a simulator moves within an organ digital model, and / or when the simulator performs laparoscopic surgery simulation in the organ digital model under force feedback guidance, a simulation sample set is acquired according to a preset sampling period. The simulation sample set includes simulation operation images and simulation image labels corresponding to the simulation operation images.

[0009] The constructed image annotation model is trained based on the simulation sample set and the video sample set to obtain the trained image annotation model. The video sample set includes video sample images and corresponding video image labels. The trained image annotation model is used to output the labels of images acquired during laparoscopic surgery.

[0010] In one possible implementation, the method further includes:

[0011] According to the preset noise rules, noise is added to the simulation operation image, and the simulation sample set is updated based on the simulation operation image after adding noise. The noise includes at least one of the following: electrosurgical smoke noise, lens contamination noise, and light fluctuation noise.

[0012] In one possible implementation, when the noise is electrosurgical smoke noise, noise is added to the simulated operation image according to a preset noise rule, specifically including:

[0013] In response to the received electrosurgical activation trigger command, the trigger time is timed;

[0014] When the timing duration exceeds the preset electrosurgical activation trigger time, an electrosurgical smoke noise layer is generated based on the smoke particle density function;

[0015] Add the electrosurgical smoke noise layer as a noise image to the simulation job image.

[0016] In one possible implementation, when the noise is lens contamination noise, noise is added to the simulated work image according to a preset noise rule, specifically including:

[0017] Based on a preset time interval and a pollution event probability function, random sampling is performed from the texture library to obtain semi-transparent texture stain images;

[0018] Based on the transparency coefficient, semi-transparent textured stain images are added to the simulation job image in a weighted manner.

[0019] In one possible implementation, when the noise is illumination fluctuation noise, noise is added to the simulation image according to a preset noise rule, specifically including:

[0020] The illumination intensity function is generated based on sinusoidal wave noise and random flicker noise;

[0021] Adjust the illumination intensity of the simulation image based on the illumination intensity function.

[0022] In one possible implementation, the constructed image annotation model is trained based on a simulation sample set and a video sample set, including:

[0023] The training batches of the image annotation model are accumulated and recorded as training rounds;

[0024] Determine the proportion of simulated data in the simulation sample set based on the training rounds;

[0025] Based on the proportion of simulated data, random samples are taken from both the simulation sample set and the video sample set, and a mixed batch training sample set is formed based on the random sampling results.

[0026] The constructed image annotation model is trained using a mixed batch training sample set.

[0027] In one possible implementation, the image annotation model is a generative adversarial network (GAN) model. The constructed image annotation model is trained using a simulation sample set and a video sample set, including:

[0028] The simulated task image and video sample image are input into the generative adversarial network model to generate the first predicted label corresponding to the simulated task image and the second predicted label corresponding to the video sample image.

[0029] Based on the first predicted label and the corresponding simulated image label, and the second predicted label and the corresponding video image label, the expected distribution of simulated data and the expected distribution of real data are generated respectively.

[0030] Generate a domain discrimination loss function based on the expected distribution of simulated data and the expected distribution of real data.

[0031] The constructed image annotation model is trained based on the domain discrimination loss function.

[0032] In one possible implementation, the method further includes:

[0033] Based on the first predicted label and the corresponding simulated image label, and the second predicted label and the corresponding video image label, a cross-entropy loss function and a similarity coefficient loss function are constructed, and the weighted result of the cross-entropy loss function and the similarity coefficient loss function is denoted as the downstream task main loss function.

[0034] Based on the preset coefficients, calculate the loss functions and values ​​of the downstream task's main loss function, domain discrimination loss function, and auxiliary regression loss;

[0035] The constructed image annotation model is trained based on the loss function and its value.

[0036] Secondly, embodiments of this application provide an image recognition method, including:

[0037] Acquire images during laparoscopic surgery;

[0038] The acquired images are input into the trained image annotation model to generate image labels for the acquired images. The trained image annotation model is a model trained according to the model training method for laparoscopic surgery image annotation described in the first aspect above.

[0039] Thirdly, embodiments of this application provide a model training device for laparoscopic surgical image annotation, comprising:

[0040] The organ generation module is used to construct an anatomical structure model based on sample image data, and to perform biological simulation based on the anatomical structure model to generate digital organ models; wherein, the digital organ model includes at least one simulated organ.

[0041] The simulation module is used to input digital organ models into a virtual reality simulation scene;

[0042] The data acquisition module is used to acquire a simulation sample set according to a preset sampling period when the simulation device moves in the organ digital model in the virtual reality simulation scene, and / or when the simulation device performs laparoscopic surgery simulation operation in the organ digital model under the guidance of force feedback. The simulation sample set includes simulation operation images and simulation image labels corresponding to the simulation operation images.

[0043] The training module is used to train the constructed image annotation model based on the simulation sample set and the video sample set to obtain the trained image annotation model. The video sample set includes video sample images and corresponding video image labels. The trained image annotation model is used to output the labels of the images acquired during laparoscopic surgery.

[0044] In one possible implementation, the device further includes:

[0045] The noise module is used to add noise to the simulation operation image according to the preset noise rules, and update the simulation sample set according to the simulation operation image after adding noise. The noise includes at least one of the following: electrosurgical smoke noise, lens contamination noise, and light fluctuation noise.

[0046] In one possible implementation, when the noise is electrosurgical smoke noise, the noise module specifically includes:

[0047] The first noise submodule is used to time the trigger time in response to the received electrosurgical activation trigger command;

[0048] The first noise submodule is also used to generate an electrosurgical smoke noise layer based on the smoke particle density function when the timing duration is longer than the preset electrosurgical activation trigger time.

[0049] The first noise submodule is also used to add the electrosurgical smoke noise layer as a noise image to the simulation job image.

[0050] In one possible implementation, when the noise is lens contamination noise, the noise module specifically includes:

[0051] The second noise submodule is used to randomly sample from the texture library according to a preset time interval and a pollution event probability function to obtain a semi-transparent texture stain image.

[0052] The second noise submodule is also used to add semi-transparent textured stain images to the simulation job image in a weighted manner based on the transparency coefficient.

[0053] In one possible implementation, when the noise is light fluctuation noise, the noise module specifically includes:

[0054] The third noise submodule is used to generate a light intensity function based on sinusoidal wave noise and random flicker noise;

[0055] The third noise submodule is also used to adjust the illumination intensity of the simulation image according to the illumination intensity function.

[0056] In one possible implementation, the training module includes:

[0057] The accumulation submodule is used to accumulate the training batches of the image annotation model, which is recorded as the training rounds;

[0058] The proportion submodule is used to determine the proportion of simulated data in the simulation sample set based on the training rounds;

[0059] The sampling module is used to randomly sample from the simulation sample set and the video sample set according to the proportion of simulated data, and to form a mixed batch training sample set based on the random sampling results.

[0060] The first training submodule is used to train the constructed image annotation model based on the mixed batch training sample set.

[0061] In one possible implementation, the image annotation model is a generative adversarial network model, and the training module includes:

[0062] The labeling submodule is used to input the simulation task image and the video sample image into the generative adversarial network model to generate the first predicted label corresponding to the simulation task image and the second predicted label corresponding to the video sample image.

[0063] The expectation submodule is used to generate the expected distribution of simulated data and the expected distribution of real data based on the first predicted label and the corresponding simulated image label, and the second predicted label and the corresponding video image label, respectively.

[0064] The loss submodule is used to generate a domain discrimination loss function based on the expected distribution of the simulated data and the expected distribution of the real data.

[0065] The second training submodule is used to train the constructed image annotation model based on the domain discrimination loss function.

[0066] In one possible implementation, the device further includes:

[0067] The loss calculation module is used to construct a cross-entropy loss function and a similarity coefficient loss function based on the first predicted label and the corresponding simulation image label, and the second predicted label and the corresponding video image label, and to record the weighted result of the cross-entropy loss function and the similarity coefficient loss function as the main loss function of the downstream task.

[0068] The loss calculation module is also used to calculate the loss functions and values ​​of the main loss function, domain discrimination loss function, and auxiliary regression loss of downstream tasks based on preset coefficients;

[0069] The training module is also used to train the constructed image annotation model based on the loss function and values.

[0070] Fourthly, embodiments of this application provide an image recognition device, comprising:

[0071] The acquisition module is used to acquire images during laparoscopic surgery.

[0072] The generation module is used to input the acquired image into the trained image annotation model and generate image labels for the acquired image. The trained image annotation model is a model trained according to the model training method for laparoscopic surgery image annotation described in the first aspect above.

[0073] Fifthly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0074] The memory stores the instructions that the computer executes;

[0075] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above, or to perform the implementation of the second aspect as described above.

[0076] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect, or to implement the implementations of the second aspect.

[0077] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect, or implements the implementation of the second aspect.

[0078] The model training method, image recognition method, device, and product for laparoscopic surgery image annotation provided in this application construct an anatomical structure model based on sample image data and generate corresponding organ digital models. These organ digital models are then input into a virtual reality simulation scene to enable the simulation apparatus to move and / or, under force feedback guidance, control the simulation apparatus to perform laparoscopic surgical simulation operations on the organ digital models. Subsequently, according to a preset sampling period, a simulation sample set containing simulation operation images and their corresponding simulation image labels is acquired. This sample set is then combined with a video sample set containing video sample images and their corresponding video image labels to train the constructed image annotation model. The trained model can then output labels for images acquired during laparoscopic surgery to assist in the procedure. This approach reduces the pressure of real surgical data acquisition and manual annotation costs while providing training samples with clear structure, complete labels, and close resemblance to real surgical processes. This increases the amount of sample data used for model training, thereby improving the generalization ability, recognition stability, and robustness of the trained image annotation model for real complex laparoscopic surgical scenarios. Attached Figure Description

[0079] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0080] Figure 1 A flowchart illustrating the model training method for laparoscopic surgical image annotation provided in this application. Figure 1 ;

[0081] Figure 2 A flowchart illustrating the model training method for laparoscopic surgical image annotation provided in this application. Figure 2 ;

[0082] Figure 3 A flowchart illustrating the image recognition method provided in this application;

[0083] Figure 4 Schematic diagram of the model training device for laparoscopic surgical image annotation provided in this application Figure 1 ;

[0084] Figure 5 Schematic diagram of the model training device for laparoscopic surgical image annotation provided in this application Figure 2 ;

[0085] Figure 6 A schematic diagram of the image recognition device provided in this application;

[0086] Figure 7 A schematic diagram of the structure of the electronic device provided in this application.

[0087] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0088] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0089] Laparoscopic surgery is a surgical technique that involves inserting a light source, camera, and surgical instruments through tiny incisions, guided by images on a monitor. During laparoscopic surgery, AI models can be used to recognize images acquired during the procedure, providing necessary visual assistance. Related systems typically include a laparoscopic imaging unit, a surgical video acquisition unit, a sample management unit, and a computational processing unit for training and deploying image annotation models.

[0090] In existing solutions, the training of image annotation / recognition models is mostly based on real laparoscopic surgical videos. The positions (such as distance and pose) of organs, blood vessels, lesions or instruments in the video frames are manually annotated, and then the model is trained using the annotated sample set so that the model can output corresponding labels for images acquired during surgery.

[0091] While such solutions can directly utilize information from real clinical scenarios, the acquisition of real surgical videos is often limited by privacy protection, ethical approvals, and the scope of data sharing. Furthermore, for some complex surgical scenarios, the size of the sample set available for training is limited. Meanwhile, manual frame-by-frame annotation requires the continuous participation of personnel with specialized medical knowledge, which is not only time-consuming and costly, but also prone to affecting label consistency due to differences in understanding of boundaries and categories among different annotators.

[0092] In cases of insufficient samples and fluctuating image annotation quality, the trained image annotation model is difficult to fully cover various complex intraoperative situations. In particular, when tissue deformation, lighting changes, or visual field disturbances occur, the stability and accuracy of the image label output are prone to decline, thus limiting its application in real laparoscopic surgery.

[0093] Therefore, how to improve the model's adaptability to real laparoscopic surgical scenarios and the accuracy of image recognition while reducing the burden of training sample acquisition and annotation has become an urgent technical problem to be solved.

[0094] To address the aforementioned issues, this application provides a model training method for image annotation in laparoscopic surgery. This method involves constructing an anatomical structure model based on sample image data and performing biological simulation to generate digital organ models. These digital organ models are then input into a virtual reality (VR) simulation scene, where a simulation device performs a laparoscopic surgical simulation along a preset path. A simulation sample set containing images of the simulation operation and their corresponding simulation image labels is acquired according to a preset sampling period. This sample set is then combined with a video sample set to train the image annotation model, resulting in a trained model that can be used to output image labels acquired during laparoscopic surgery.

[0095] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0096] Figure 1 A flowchart illustrating the model training method for laparoscopic surgical image annotation provided in this application. Figure 1 ,like Figure 1 As shown, the method includes:

[0097] S101: Based on the sample image data, construct an anatomical structure model, and perform biological simulation based on the anatomical structure model to generate organ digital models, wherein the organ digital model includes at least one simulated organ.

[0098] In one specific implementation, a static anatomical structure model is first constructed based on sample image data. Then, biomechanical parameters are assigned to this anatomical structure model, and physical deformation simulation is performed to generate an organ digital model that can be used for interactive operation. The anatomical structure model focuses on geometric and topological information, while the organ digital model is a model of the anatomical structure model that has undergone biological simulation and possesses physical interactivity. Taking the liver as an example, the process of generating the organ digital model is as follows:

[0099] (1) Obtain enhanced CT images of the abdomen, and use a pre-trained nnU-Net network to automatically segment the liver parenchyma, hepatic vein, portal vein and tumor lesions in the images to obtain voxel-level segmentation results.

[0100] (2) The surface mesh of each structure is extracted using the moving cube algorithm, and the mesh is simplified and the holes are repaired to obtain the triangular facet model.

[0101] (3) Use the TetGen tool to convert the surface mesh into a tetrahedral mesh for finite element calculation.

[0102] (4) Biomechanical parameters were assigned to different tissues: liver parenchyma elastic modulus E=5kPa, Poisson's ratio ν=0.49, density ρ=1050kg / m³; hepatic vein elastic modulus E=200kPa, Poisson's ratio ν=0.45.

[0103] (5) Using the co-rotational finite element solver in the SOFA physics engine, fixed boundary constraints were set to complete the deformation simulation configuration. This generates a digital model of the organ suitable for VR interaction.

[0104] For example, sample image data, as the original input for constructing the anatomical structure model, can be computed tomography (CT) images, magnetic resonance imaging (MRI) images, ultrasound tomography images obtained before surgery, or desensitized medical image sequences from a standard case database. The anatomical structure model carries the anatomical information from the sample image data. Specifically, it can be formed into a three-dimensional digital structure using a combination of voxel reconstruction and surface mesh reconstruction. During processing, the sample image data is first subjected to inter-slice registration, noise suppression, and grayscale normalization. Then, segmentation processing is performed based on the grayscale differences and boundary continuity of organs, blood vessels, lesions, or areas of interest in laparoscopic surgery to determine the contour of the target anatomical structure. Segmentation methods can include one or more combinations of threshold segmentation, region growing, graph cut segmentation, and neural network-based semantic segmentation. After hole filling, boundary smoothing, and topological correction, the segmented contour is organized into model data suitable for biosimulation.

[0105] In one example, taking a patient's preoperative abdominal CT image as an example, a pre-trained adaptive medical image segmentation network model can be used to perform semantic segmentation on the abdominal CT image to obtain voxel-level multi-organ segmentation results. Then, a 3D mesh model containing anatomical structures such as organ parenchyma, vascular networks, nerve bundles, and tumor lesions can be reconstructed using algorithms such as Moving Cubes (MC) 3D isosurface extraction and mesh post-processing. This 3D mesh model is then used as the anatomical structure model.

[0106] The bio-simulation in step S101 above is used to convert the static anatomical structure model into an interactive digital organ model. For example, differentiated biomechanical parameters (such as elastic modulus E, Poisson's ratio ν, density ρ, damping coefficient η, etc.) can be assigned to the anatomical structure model, and deformation simulation can be performed using a method such as the co-rotational finite element method (Co-rotational FEM). Through the internal structure coupling and binding method based on the center of gravity coordinate, it is ensured that the embedded blood vessels and nerves move in conjunction with the organ parenchyma, so that the obtained digital organ model can present compression, traction, displacement and local deformation in a virtual environment.

[0107] In another example, the aforementioned organ digital model includes at least one simulated organ, which can correspond to any object in the liver, gallbladder, stomach, lung, intestine, vascular bundle, or lesion area, or can be composed of multiple interconnected anatomical structures to reflect the spatial relationships that need to be observed and manipulated during laparoscopic operations.

[0108] For example, by reading sample image data and simultaneously loading corresponding case data according to the corresponding case number, a mapping relationship between pixel coordinates and real spatial coordinates is established for each layer of image. Then, automatic segmentation is performed according to the pre-set target anatomical category. If the automatic segmentation result has local edge breaks, boundary completion can be performed by calling morphological closing operations and surface interpolation algorithms, and then a triangular mesh model or a tetrahedral volume mesh model is output. For scenarios that need to reflect the stress state of organs, mass, stiffness, and damping parameters can be further assigned to the relevant mesh nodes in the above mesh model, and fixed points, slip boundaries, or tissue attachment boundaries can be set so that corresponding deformations are generated when the simulation instrument comes into contact with them.

[0109] Once the above organ digital models are generated, they can be saved as model files suitable for graphics engines to read, along with information such as anatomical category identifiers, spatial hierarchy relationships, and visibility attributes.

[0110] Based on the above analysis, step S101 transforms real sample image data constrained by privacy and manual annotation into a reusable, structured digital organ model, enabling subsequent simulation scenarios to directly output images and labels consistent with anatomical structures, serving as the basic data output link in the entire training process.

[0111] S102: Input the digital model of the organ into the virtual reality simulation scene.

[0112] For example, a virtual reality simulation scene can be built using a graphics rendering engine, a physics interaction engine, and corresponding functional modules. After the digital organ model is imported, a world coordinate system, a laparoscopic viewpoint coordinate system (camera coordinate system), and an instrument motion coordinate system can be established within the VR simulation scene. The laparoscopic lens parameters, light source position, illumination angle, and field of view can be configured to form an observation environment corresponding to laparoscopic or thoracoscopic surgery.

[0113] After the digital organ model is input into the virtual reality simulation scene, a simulation device can be built in the virtual reality simulation scene. This simulation device is used to perform operations on the input digital organ model in the virtual reality simulation scene. The simulation device can be constructed as a virtual instrument model of a separation forceps, grasping forceps, electrocoagulation hook, scissors, suction device or lens body (endoscope). Each simulation device has a geometric shape, collision body, operating end posture and motion constraints.

[0114] In one example, virtual reality simulation scenes can be built by inputting digital organ models into a game engine. For instance, by configuring the parameters of a binocular camera (camera intrinsic and extrinsic parameters), the lens body of an endoscope in a simulation device can be simulated. The rigid body collision body and interaction force calculation configuration of the simulation device, as well as the real-time rendering configuration of the three-dimensional organ model (such as setting shaders to support semi-transparency and texture mapping) can also be configured.

[0115] When configuring real-time rendering, you can first configure the rendering hierarchy in the VR simulation engine, assigning an independent rendering hierarchy to each anatomical structure category. These anatomical structure categories can include liver parenchyma, gallbladder, kidney, spleen, portal vein, hepatic artery, hepatic vein, inferior vena cava, and tumor lesions. Each hierarchy should be assigned a unique color code, which can be represented by the following formula:

[0116]

[0117]

[0118] In the formula, For color coding, K represents the number of anatomical structure categories, and K represents the total number of anatomical structure categories. , , The color component values ​​of the anatomical structure category k in the red, green, and blue channels are integers ranging from 0 to 255.

[0119] It should be noted that the color codes corresponding to different anatomical structure categories are unique to each other, so as to ensure that the decoding stage can uniquely map the pixel color back to the anatomical structure category.

[0120] Subsequently, utilizing the game engine's multi-render target (MRT) functionality, simultaneous rendering of multiple render targets (MRTs) is performed. This outputs three render targets simultaneously in a single rendering pass on the general-purpose processor (GPU), including:

[0121] 1. RGB channel rendering target: RGB color image rendered under normal lighting. ,in, and These are the height and width (in pixels) of the output image, respectively. For frame time index.

[0122] 2. SegMask segmentation mask channel rendering target: pixel-level semantic segmentation mask based on the rendering layer. In this context, each pixel value is equal to the category number of the anatomical structure to which that pixel belongs. The background class has a corresponding value of 0.

[0123] 3. Depth channel rendering target: Scene depth map Record the three-dimensional spatial depth value corresponding to each pixel. This depth value can be the Euclidean distance from the optical center of the binocular camera along the optical axis to the corresponding pixel.

[0124] Furthermore, the three rendering targets mentioned above need to be timestamped, using the following alignment formula:

[0125]

[0126] In the formula, The simulation scene state at time t can include parameters such as organ deformation state, instrument position, lighting conditions, and smoke particle distribution; This is a set of rendering parameters, which may include parameters such as camera intrinsics, lighting parameters, and shader parameters. This indicates an MRT rendering call that returns three time-aligned output channels.

[0127] During the rendering process, at each simulation frame time t, the intrinsic and extrinsic parameter matrices of the stereo camera are recorded synchronously. The intrinsic parameter matrix of the left eye is set as follows: The right eye's intrinsic parameter matrix is It includes focal length and principal point coordinates; the left eye extrinsic parameter matrix is... The extrinsic matrix of the right eye is .

[0128] Among them, the extrinsic parameter matrix Used to locate three-dimensional points in the world coordinate system Transforming to the stereo camera coordinate system, the corresponding calculation formula is:

[0129]

[0130] In the formula, for Rotation matrix (belongs to a special orthogonal group in three dimensions). It is a translation vector. These are the three-dimensional coordinates of the transformed camera coordinate system.

[0131] Finally, when acquiring the simulation sample set, the outputs of each channel and the camera pose matrix can be assembled into a structured multimodal training data package according to the timestamps. Its expression can be in the following forms:

[0132]

[0133] In the formula, For frame index, T is the total number of frames; subscript and These represent the left and right cameras in a stereo camera system; These are the RGB image frames corresponding to the left and right cameras, respectively. These are the semantic segmentation masks corresponding to the image frames from the left and right cameras, respectively. This is the corresponding depth map.

[0134] It should be noted that if there are multiple simulated organs or instruments in the scene, the operations can be performed in a preset order or in parallel relationship based on the hierarchical scheduling strategy, so that the tissue occlusion, instrument crossing and local exposure state in the same surgical step can be completely reproduced.

[0135] S103: When a simulator moves within an organ digital model in a virtual reality simulation scene, and / or when the simulator performs laparoscopic surgery simulation operations within the organ digital model under force feedback guidance, a simulation sample set is acquired according to a preset sampling period. The simulation sample set includes simulation operation images and simulation image labels corresponding to the simulation operation images.

[0136] For example, after completing the above steps S101 and S102, in order to obtain the simulation sample set for model training, it is necessary to control the movement of the simulation instrument in the digital organ model in the virtual reality simulation scene to simulate the laparoscopic surgery operation scene. The methods for controlling the movement of the simulation instrument can include at least three types:

[0137] 1. Through the corresponding functional modules, the movement path of the simulation device and the preset actions at each position point in the path can be configured in the virtual reality simulation scene, so that the simulation device can automatically move according to the preset path and execute the relevant preset actions.

[0138] 2. Through the corresponding functional modules, the movement path of the simulation device and the preset actions at each position point in the path can be configured in the virtual reality simulation scene. The preset path and preset actions can be prompted in the virtual reality simulation scene, and then controlled by relevant control personnel, so that the simulation device moves along the preset path under the guidance of force feedback and performs corresponding actions under the control of the control personnel.

[0139] 3. Relevant control personnel operate the simulation equipment according to the display of the organ digital model in the virtual reality simulation scene, based on their own experience or relevant work instructions, so that it simulates laparoscopic surgery under the guidance of force feedback.

[0140] In this embodiment, the aforementioned preset path is used to define the motion trajectory of the simulation instrument in the virtual reality simulation scene. It can be determined by the case data corresponding to the sample image data. Specifically, it can be based on the instrument trajectory extracted from historical real surgical videos, the key point sequence defined by the standard operating procedure of the surgical procedure, or pre-recorded by the operator in the simulation platform.

[0141] Laparoscopic surgery simulation is a continuous operation process under the above-mentioned simulation constraints, in which the simulation instruments are controlled to perform actions such as pushing, rotating, contacting, clamping, pulling, separating, cutting or hemostasis on the digital model of the organ.

[0142] In one example, taking the movement of a simulation device under the control of relevant functional modules as an example, the digital model of the organ can be loaded into the graphics rendering engine first, and the initial pose can be determined according to the connection relationship between the anatomical structures. Then, the simulation device can be loaded into the scene entrance position. Subsequently, the configured functional modules can read the preset path file and drive the position, posture and opening and closing state changes of the simulation device according to the timestamp to simulate the laparoscopic surgery corresponding to the digital model of the organ.

[0143] In this embodiment, the preset path can be represented as a trajectory sequence consisting of multiple path nodes. Each path node can include three-dimensional coordinates, attitude angles, velocity information, and corresponding action commands. When controlling the movement of the simulated device, its smooth movement can be controlled according to the interpolation results between adjacent nodes, and when it comes into contact with the simulated organ, the corresponding physical interaction engine is invoked to calculate the contact area, force direction, and deformation.

[0144] To make the simulation of laparoscopic surgery closer to the real laparoscopic scene, the virtual lens in the simulation instrument can be moved forward, backward, rotated or zoomed in synchronously to form an image sequence in which the instrument operation and the observation perspective change together.

[0145] For example, the preset sampling period is used to control the acquisition frequency of the simulation samples. It can be a fixed time interval or an interval based on the rendering frame number. In the implementation corresponding to the frame rate of the cavity mirror video, the preset sampling period can be set to a time interval that matches 25 frames per second, 30 frames per second, or 60 frames per second, so that the temporal resolution of the simulation samples is consistent with that of the real video sample set.

[0146] The simulation operation image reflects the surgical procedure in a virtual reality simulation scene. It can be a two-dimensional image frame captured by a virtual camera at the current moment. The image content can include information such as simulated organs, simulated instruments, shadows, occlusions, local deformations, and field of view edges. Simulation image labels correspond one-to-one with the simulation operation image and are used to characterize target information in the image. These can include organ labels, instrument labels, and distance / attitude. The organ label can be the name of the simulated organ within a preset range in front of the simulated instrument. The instrument angle can be the three-dimensional angle of the simulated instrument in the world coordinate system at the current simulation moment. The distance / attitude refers to the distance between the front end of the simulated instrument and the simulated organ within the preset range in front of it, as well as the attitude of the simulated instrument at the current simulation moment, which can include yaw angle, pitch angle, and roll angle.

[0147] For organ labels, they can include one or more of the following: semantic segmentation labels, instance segmentation labels, organ boundary labels, instrument outline labels, key point labels, or object detection box labels.

[0148] Since organ digital models and simulation devices have known category identifiers, spatial locations, and surface topologies in virtual reality simulation scenes, color maps and corresponding label maps can be output synchronously through the rendering pipeline at each sampling time, or corresponding simulation image labels can be automatically generated by reading scene object ID buffers, depth buffers, and occlusion relationship tables.

[0149] In one example, after the laparoscopic surgery simulation begins, a sampling timer is started. When the preset sampling period is reached, the image acquisition interface is called to capture the current rendered frame as the simulation image. At the same time, the label generation interface is called to map the pixel regions of the visible simulated organs and instruments in the current viewpoint to the corresponding category codes, forming a label matrix of the same size as the image. Meanwhile, the camera pose matrix, depth map, etc. at the corresponding time are acquired to form the simulation image label corresponding to the simulation image.

[0150] If object detection is used, the coordinates of the detection box and category information can be further output based on the bounding rectangle of the visible area.

[0151] It should be noted that the above simulation sample set can be organized into a structured dataset according to case, procedure, time sequence or scene number when stored. Each sample unit contains at least an image file, a label file and index metadata.

[0152] Based on the above analysis, it can be seen that step S103 directly generates paired images and labels during the surgical simulation, eliminating the need for manual frame-by-frame annotation. This provides continuous training data covering the instrument movement, tissue deformation, and perspective changes, thus forming the core sample source for subsequent model training.

[0153] S104: Based on the simulation sample set and the video sample set, the constructed image annotation model is trained to obtain the trained image annotation model. The video sample set includes video sample images and corresponding video image labels. The trained image annotation model is used to output the labels of the images acquired during the laparoscopic surgery.

[0154] For example, the image annotation model is used to output labels for the input image. The model can be a semantic segmentation network, an instance segmentation network, an object detection network, or a multi-task network that outputs labels for multiple classes at the same time. Before training, it is in a state of waiting to be trained.

[0155] The video sample set serves as a source of real or existing video data. The video sample images can be desensitized real laparoscopic surgery video frames. The video image labels are labels containing information such as the corresponding organs, blood vessels, lesions, or instruments in the image. The label format is consistent with the simulated image labels or is consistent after mapping and transformation.

[0156] In one example, a unified data processing procedure can be performed on the simulation sample set and the video sample set, including image size normalization, color space unification, label category mapping, and data partitioning. Then, samples are extracted from the simulation sample set and the video sample set to form training subsets, validation subsets, and test subsets. These subsets are then input into the image annotation model to be trained for training, validation, and testing. If supervised learning is used during training, the loss value between the model output and the corresponding label can be calculated for each input batch, and the model parameters can be updated through backpropagation.

[0157] It should be noted that if there are naming differences between the label categories in the video sample images and the simulation operation images, a unified category dictionary can be established before training to map synonymous organ categories, instrument categories, and lesion categories to the same output space.

[0158] During training, the model's training effectiveness can be evaluated based on a validation subset after training, and the trained image annotation model can be output when preset conditions are met. The trained image annotation model can be deployed in a laparoscopic surgery system to receive images acquired in real time by the laparoscopic imaging end during surgery and output corresponding label maps, organ category information, or instrument position results.

[0159] Based on the above analysis, this application provides a model training method for laparoscopic surgery image annotation. The method includes constructing an anatomical structure model and generating organ digital models based on sample image data. The organ digital models are then input into a virtual reality simulation scene to control the simulation apparatus to perform laparoscopic surgery simulation operations. A simulation sample set containing simulation operation images and their corresponding simulation image labels is acquired according to a preset sampling period. This is combined with a video sample set containing video sample images and their corresponding video image labels to train the image annotation model, resulting in a trained image annotation model. Through the above method in this application embodiment, anatomical information in real medical images is transformed into interactive simulation objects to obtain organ digital models. The virtual reality simulation process automatically generates continuous, one-to-one corresponding images and labels, which are then jointly trained with real video samples. This reduces the pressure of real surgical data acquisition and the cost of manual annotation while providing training samples with clear structure, complete labels, and close resemblance to the real surgical process. This increases the amount of sample data used for model training, enabling the trained model to output labels for images acquired during laparoscopic surgery.

[0160] Figure 2 A flowchart illustrating the model training method for laparoscopic surgical image annotation provided in this application. Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the examples, a model training method for image annotation in laparoscopic surgery is described in detail. This method includes:

[0161] S201: Based on the sample image data, construct an anatomical structure model, and perform biological simulation based on the anatomical structure model to generate organ digital models; wherein, the organ digital model includes at least one simulated organ.

[0162] S202: Input the digital model of the organ into the virtual reality simulation scene.

[0163] S203: When a simulator moves within an organ digital model in a virtual reality simulation scene, and / or when the simulator performs laparoscopic surgery simulation operations within the organ digital model under force feedback guidance, a simulation sample set is acquired according to a preset sampling period. The simulation sample set includes simulation operation images and simulation image labels corresponding to the simulation operation images.

[0164] S204: Add noise to the simulation operation image according to the preset noise rules, and update the simulation sample set according to the simulation operation image after adding noise. The noise includes at least one of the following: electrosurgical smoke noise, lens contamination noise, and illumination fluctuation noise.

[0165] S205: Based on the simulation sample set and the video sample set, train the constructed image annotation model to obtain the trained image annotation model. The video sample set includes video sample images and corresponding video image labels. The trained image annotation model is used to output the labels of the images acquired during the laparoscopic surgery.

[0166] For example, preset noise rules are used to constrain the triggering time, superposition method, and parameter values ​​of noise, so that the noise acts on the simulated work image according to common interference patterns in laparoscopic surgery. Electrosurgical smoke noise is used to simulate the obstruction effect of smoke on the field of vision after the electrosurgical unit is activated. It can generate a smoke particle layer based on the electrosurgical activation trigger command and preset duration, and superimpose it onto the corresponding area of ​​the simulated work image. Lens contamination noise is used to simulate the obstruction caused by droplets, stains, or tissue residue adhering to the lens surface. It can extract a semi-transparent textured stain image from a texture library based on a preset time interval and the probability of contamination events, and then fuse it onto the image according to the transparency coefficient. Illumination fluctuation noise is used to simulate the brightness changes of the laparoscopic light source caused by tissue reflection, instrument obstruction, or equipment fluctuations. It can construct an illumination intensity function based on sinusoidal fluctuation noise and random flicker noise, and adjust the image pixel brightness accordingly.

[0167] In one example, the simulation task image can be a frame of image obtained by continuous sampling from a virtual reality simulation scene. Then, according to a preset noise rule, a single noise or a combination of multiple noises is applied to the image frame to form image samples with different interference intensities and spatial distributions.

[0168] For electrosurgical smoke noise, the grayscale value, diffusion radius, and opacity of the smoke particle layer can be set in relation to the electrosurgical activation duration; for lens contamination noise, the shape, area, and transparency of the semi-transparent textured stain image can be randomly sampled and limited to the lens pixel area; for illumination fluctuation noise, the brightness variation amplitude and fluctuation frequency can be generated according to a preset range so that the simulated operation image after adding noise retains organ boundaries, instrument outlines, and surgical field structure information.

[0169] In this embodiment, the simulated surgical images with added noise and their corresponding simulated image labels together constitute an updated simulated sample set, which is then used in subsequent image annotation model training. In this way, the images in the simulated sample set can contain multiple types of intraoperative visual interference factors. The trained model can maintain relatively stable label output consistency when faced with smoke obstruction, lens contamination, and lighting changes in real laparoscopic images, and improve its adaptability to complex surgical fields.

[0170] In one possible implementation, when the noise is electrosurgical smoke noise, noise is added to the simulation operation image according to a preset noise rule. Specifically, this includes: timing the trigger time in response to a received electrosurgical activation trigger command; when the timing duration is longer than the preset electrosurgical activation trigger time, generating an electrosurgical smoke noise layer according to the smoke particle density function; and adding the electrosurgical smoke noise layer as a noise image to the simulation operation image.

[0171] For example, after detecting a change in the electrosurgical knife's action state, an electrosurgical knife activation trigger command can be output, and the current moment can be used as the starting point for timing. When the accumulated time reaches the preset electrosurgical knife activation trigger time, the distribution intensity of smoke particles in the center region, edge region, and adjacent region of the instrument can be calculated based on the smoke particle density function, and an electrosurgical knife smoke noise layer can be generated accordingly.

[0172] In one example, the smoke particle density function described above can be set as a piecewise exponential function that varies with time t, and the corresponding calculation formula can be:

[0173]

[0174] In the formula, Let be the smoke particle density function. For time, The maximum smoke concentration can be set to a range of [0.3, 0.9], and random sampling will be performed from this range during simulation session initialization. This is the smoke rise time constant, which can be set to 0.3–1.0 seconds to control the speed at which smoke rises from its initial concentration to its peak concentration. This is the smoke dissipation time constant, which can be set to 2.0–5.0 seconds to control the speed at which the smoke dissipates naturally. The activation trigger moment of the electrosurgical unit. For indicator functions, when When the value is set to 1, it is set to 0 otherwise, to ensure that smoke is only generated after the electrosurgical unit is activated.

[0175] Subsequently, the electrosurgical smoke noise layer is written into the simulation image either by pixel-by-pixel overlay or by fusion based on transparency weights, thereby forming a simulation sample that includes the electrosurgical smoke occlusion effect.

[0176] In another example, the smoke particle density function can also continuously adjust the particle sparsity and coverage range with the duration of the time, so that the smoke gradually spreads from a local appearance to a larger area. This application does not limit this.

[0177] Through the above method, a correspondence was established between the generation of electrosurgical smoke noise and the activation time of the electrosurgical unit. The distribution pattern of smoke in the image was controlled by the smoke particle density function, so that the simulated operation image can present a smoke interference effect that is closer to the real laparoscopic electrosurgical operation. This provides a simulation sample with scene consistency for subsequent image annotation model training and improves the model's ability to recognize and adapt to electrosurgical smoke occlusion scenes.

[0178] In one possible implementation, when the noise is lens contamination noise, noise is added to the simulation work image according to a preset noise rule. Specifically, this includes: randomly sampling from a texture library according to a preset time interval and a contamination event probability function to obtain a semi-transparent texture stain image; and adding the semi-transparent texture stain image to the simulation work image in a weighted manner according to the transparency coefficient.

[0179] For example, when acquiring or generating images for a simulation operation, the interval between the current time and the last time pollution noise was generated can be determined, and this interval can be input into a pollution event probability function to calculate the probability of a pollution event occurring. The corresponding calculation formula can be:

[0180]

[0181] In the formula, This is a parameter representing the incidence rate of pollution events, which can be set to 0.01–0.05 times / second, characterizing the average number of pollution events occurring per unit time. For time intervals.

[0182] When the calculation results meet the preset trigger conditions, lens contamination simulation can be performed by overlaying randomly generated semi-transparent textured stains onto the virtual lens of the endoscope. This involves randomly selecting a corresponding stain texture from a predefined texture library. The selected texture can be teardrop-shaped, oil film-shaped, or dot-shaped residual texture, and then transparently processed to form a semi-transparent textured stain image, which can be represented by the following formula:

[0183]

[0184] In the formula, For a predefined texture library, The total number of predefined contamination textures.

[0185] Accordingly, the transparency coefficient can be sampled from a uniform distribution, and the corresponding calculation formula is:

[0186]

[0187] That is, uniform sampling is performed within the interval [0.1, 0.5]. This is the blending transparency coefficient for the contaminated texture.

[0188] Subsequently, the semi-transparent textured stain image and the simulated job image are fused using pixel-level weighted fusion based on the transparency coefficient. The corresponding calculation formula can be expressed as follows:

[0189]

[0190] In the formula, These are the pixel coordinates in the image. To overlay the image after lens contamination noise, To overlay the image before lens contamination noise, This represents the pixel value of the contaminated texture at the corresponding coordinates.

[0191] It should be noted that the aforementioned weighting coefficients can also be set according to the occlusion intensity of the stain texture, so that the contaminated area presents a local blurring, occlusion, or mottled effect, without completely covering the anatomical structure information in the simulated image. The texture library can pre-store texture tiles processed with normalized dimensions. In practical applications, other forms of texture tiles can also be selected, which is not limited in this application.

[0192] In this way, lens contamination noise can exhibit random occurrence characteristics in time, diverse stain morphologies in appearance, and be superimposed on the simulated operation image with controllable transparency. This results in the generated simulation sample set containing image perturbations that are closer to the actual endoscope contamination scenario, improving the adaptability of the trained image labeling model to image label output under lens contamination conditions.

[0193] In one possible implementation, when the noise is light fluctuation noise, noise is added to the simulation work image according to a preset noise rule, specifically including: generating a light intensity function based on sinusoidal fluctuation noise and random flicker noise; and adjusting the light intensity of the simulation work image according to the light intensity function.

[0194] For example, the simulated image can be mapped to a time variable according to the sampling time, and a periodic term and a perturbation term can be constructed based on the time variable. The periodic term can be generated from sinusoidal wave noise, and the perturbation term can be generated from a random number sequence or a random pulse sequence. The two are superimposed to form a light intensity function. The corresponding calculation formula can be:

[0195]

[0196] In the formula, (Based illumination intensity, which is the default illumination parameter in VR simulation scenes). , represents the amplitude of randomized illumination fluctuations, ranging from 5% to 30% of the base illumination; The angular frequency of the light wave. Hz is a fluctuating frequency used to simulate the typical frequency of a surgical light moving slowly or being blocked by people; The initial phase is randomized to ensure that the initial state of illumination fluctuations is different in different simulation sessions; The random flicker noise term has a mean of zero and a standard deviation of . Gaussian distribution, These are parameters that can be configured according to requirements.

[0197] The above-mentioned illumination intensity function can be written with the illumination coefficient corresponding to the current frame. The value range of the illumination coefficient is limited to a preset threshold range to avoid overexposure or underexposure.

[0198] Subsequently, the pixel brightness channels of the simulated operation image are scaled as a whole, or the gain of a preset area is adjusted locally to obtain the simulated operation image after adding illumination fluctuation noise. When implementing local adjustment, the preset area can correspond to the central surgical area or the edge reflective area in the laparoscopic field of view, and the local gain can be calculated based on the linkage between the average gray value and the illumination intensity function within the area.

[0199] It should be noted that, in the embodiments of this application, when the above three types of noise are introduced simultaneously, the injection parameters of all complexity factors can be randomly sampled according to their respective prior distributions to form a joint configuration vector. The corresponding formula can be expressed as:

[0200]

[0201] In the formula, This is a set of scene enhancement parameters, containing all independently sampleable injection parameters from the three types of noise mentioned above.

[0202] By mapping periodic fluctuations and random disturbances together as a light intensity function, the brightness changes in the simulated images exhibit both continuous fluctuations and instantaneous flickering, thus making the generated simulation sample set closer to the lighting environment of real endoscope images. After the samples generated based on this noise are used to train the image annotation model, the stability of the model's label output under varying lighting conditions is enhanced.

[0203] In one possible implementation, the constructed image annotation model is trained based on the simulation sample set and the video sample set, including: accumulating the training batches of the image annotation model and recording them as training rounds; determining the proportion of simulated data in the simulation sample set based on the training rounds; randomly sampling from the simulation sample set and the video sample set according to the proportion of simulated data, and forming a mixed batch training sample set based on the random sampling results; and training the constructed image annotation model based on the mixed batch training sample set.

[0204] For example, the training rounds are used to characterize the total number of training batches that the image annotation model has completed, and serve as the basis for adjusting the mixing ratio of subsequent samples; the proportion of simulated data is used to limit the proportion of samples in each training round that the simulated sample set participates in; the mixed batch training sample set is a training data set formed by proportionally extracting the simulated sample set and the video sample set.

[0205] In one example, during image annotation model training, each training batch is accumulated after completing forward propagation, loss calculation, and parameter backpropagation, forming the current training epoch. Training epochs can be recorded using integer counting. Before entering a new training epoch, a preset epoch-to-simulated data ratio correspondence table can be read, or the simulated data ratio for the current epoch can be calculated based on a preset function. For example, the simulated data ratio can be set. For a function that decreases progressively according to an exponential decay strategy, the corresponding calculation formula can be:

[0206]

[0207] In the formula, The initial simulated data constitutes the proportion of the training data. In the early stages of training, the simulation sample set is the main focus, enabling the model to quickly master basic feature extraction and task capabilities (such as visual patterns of organ contours and appearance features of instruments) on sufficient and diverse simulation data. , which is the decay factor, controls the rate at which the proportion of data in the simulation sample set decays from the initial value to the minimum value. The closer it is to 1.0, the more gradual the decay, giving the model more time to gradually adapt to the characteristics of the real domain during the training process; This represents the minimum proportion of simulated data, used to ensure that the regularization effect of real data does not completely overwhelm the simulated data in the later stages of training. This is the current training round number. This refers to the total number of training rounds.

[0208] This percentage can be set to a higher value in the early stages of training, so that the simulated sample set occupies a larger proportion in the mixed training samples, and gradually decreases as the training rounds increase, so that the model can gradually incorporate the distribution characteristics of more real video samples.

[0209] After obtaining the proportion of simulated data, independent random sampling is performed on the simulation sample set and the video sample set respectively. During sampling, a portion of the total number of samples required for the current batch can be allocated to the simulation sample set and the other portion to the video sample set, and sample pairs can be selected from the corresponding sample sets using a random index. The random index can be generated by a random number generator and can be set to sampling without replacement to avoid too many duplicate samples within the same batch.

[0210] After extraction, the two types of samples are merged by batch dimension to form a mixed batch training sample set containing simulation job images, simulation image labels, video sample images, and video image labels. The following formula can be used to compose the mixed batch training sample set:

[0211]

[0212] In the formula, This is the mixed batch training sample set of the current training batch. The floor symbol is used for rounding down. The rounding up symbol; Image-label pairs sampled from a simulated sample set; These are image-label pairs sampled from a set of video samples.

[0213] When the image annotation model is trained on this mixed batch training sample set, the images in the mixed batch training sample set are first input into the network, and then the network outputs the corresponding label prediction results. The loss function is calculated with the sample labels, and then the model parameters are updated. Since the mixed batch training sample set has a different proportion of simulated data in different training rounds, the model can fully learn the basic structural information in the simulated samples in the early stage of training, and gradually adapt to the imaging characteristics and noise distribution of real video samples in subsequent training.

[0214] By adopting this method, the training process can dynamically adjust the mixing ratio of simulation and real data according to the training rounds, so that the model can gradually approach the distribution of real surgical field images while maintaining its ability to learn simulation features. This results in the trained image labeling model having a more stable label output capability for images collected during laparoscopic surgery, and improves the stability of model training convergence and the consistency of results.

[0215] In one possible implementation, the image annotation model is a generative adversarial network (GAN) model. The constructed image annotation model is trained using a simulation sample set and a video sample set, including: inputting the simulation task images and video sample images into the GAN model to generate a first predicted label corresponding to the simulation task images and a second predicted label corresponding to the video sample images; generating the expected distribution of simulated data and the expected distribution of real data based on the first predicted label and the corresponding simulation image label, and the second predicted label and the corresponding video image label, respectively; generating a domain discriminant loss function based on the expected distribution of simulated data and the expected distribution of real data; and training the constructed image annotation model using the domain discriminant loss function.

[0216] For example, a generative adversarial network model is a neural network model used to simultaneously perform label prediction and domain distribution alignment. It may include a feature extraction unit, a label generation unit, and a domain discrimination unit. The feature extraction unit is used to extract deep semantic features from the input image, the label generation unit is used to output the corresponding predicted label, and the domain discrimination unit is used to calculate discrimination information based on the distribution differences of different data sources.

[0217] In one example, after inputting simulated task images and video sample images into a generative adversarial network (GAN) model, the model encodes the images based on a shared backbone feature extraction structure and outputs predicted labels for corresponding categories, regions, or contours. Subsequently, the first predicted label is matched with the corresponding simulated image label, and the second predicted label is matched with the corresponding video image label, thus obtaining the expected distribution of the simulated data and the expected distribution of the real data, respectively. This expectation can be characterized by a weighted sum of label consistency probability, feature center distance, or distribution similarity. The two expectations are then input into the domain discriminant loss function calculation module to form the loss value used for backpropagation.

[0218] For example, setting a feature extractor Then a lightweight domain discriminator is connected in parallel. The latter takes feature maps as input and outputs domain labels. The training objective can be represented by the following function:

[0219]

[0220] In the formula, For feature extractor, the parameters are: Used to input image Mapping to feature map ; For the domain discriminator, the parameters are Taking the feature map as input, the output is the domain label prediction probability; Loss is determined by the domain; and These represent the expected distributions of the simulated and real data, respectively.

[0221] The aforementioned adversarial training can be achieved using a Gradient Reversal Layer (GRL): during forward propagation, the GRL performs an identity mapping; during backward propagation, the GRL multiplies the gradient by a negative factor. This approach causes the feature extractor to optimize towards increasing domain discrimination error (i.e., learning domain-invariant features) during parameter updates, while the domain discriminator optimizes towards decreasing domain discrimination error (i.e., accurately distinguishing feature sources). The result of this trade-off is that the feature representation learned by the feature extractor no longer contains domain-specific information, thus eliminating the feature distribution difference between simulation and reality.

[0222] By utilizing both simulated work images and video sample images in this way during training, the generative adversarial network model can maintain its label prediction ability while fulfilling inter-domain distribution constraints. Since the domain discriminant loss function is constructed from the expected distributions of both simulated and real data, the model can simultaneously absorb controllable annotation information from simulated data and data representation features from real videos during training. This results in a trained image annotation model with better adaptability and annotation consistency in real-world endoscopy scenarios.

[0223] In one possible implementation, the method further includes: constructing a cross-entropy loss function and a similarity coefficient loss function based on the first predicted label and the corresponding simulated image label, and the second predicted label and the corresponding video image label; and recording the weighted result of the cross-entropy loss function and the similarity coefficient loss function as the downstream task main loss function; calculating the loss function sum of the downstream task main loss function, the domain discrimination loss function, and the auxiliary regression loss based on preset coefficients; and training the constructed image annotation model based on the loss function sum.

[0224] For example, the first predicted label is used to characterize the output of the image annotation model for the simulated task image, and the simulated image label is used to characterize the real supervision information corresponding to the simulated task image; the second predicted label is used to characterize the output of the image annotation model for the video sample image, and the video image label is used to characterize the real supervision information corresponding to the video sample image. The cross-entropy loss function is used to characterize the classification bias between the predicted label and the corresponding label, and the similarity coefficient loss function is used to characterize the similarity between the two in spatial overlap or category matching. The downstream task main loss function is synthesized from the two according to preset weights to reflect the main optimization objective of the image annotation task. The corresponding calculation formula can be:

[0225]

[0226] In the formula, The segmentation result predicted by the model, i.e., the first predicted label or the second predicted label. These are the real labels, i.e., simulated image labels or video image labels. The first predicted label corresponds to the simulated image label, and the second predicted label corresponds to the video image label. These are the weighting coefficients for the Dice loss.

[0227] Correspondingly, the formulas for calculating the loss function and its value can be expressed as follows:

[0228]

[0229] In the formula, The main loss function for downstream tasks. For the adversarial loss of the domain discriminator, is the domain adaptive loss weight, and is a hyperparameter whose value can be set from 0.05 to 0.2 to control the importance of domain adaptive training in the overall optimization objective.

[0230] To assist the regression loss, the physical parameter labels naturally carried by the simulated data (smoke concentration, illumination parameters, deformation amplitude, etc.) are used as self-supervised signals to help the feature extractor establish the correlation between the scene's physical properties and visual appearance, thus assisting the regression loss. The calculation formula can be:

[0231]

[0232] In the formula, Enhance the set of parameters for the scene The true values ​​corresponding to each component in the middle. To assist in the return of the head For the The predicted value of each physical parameter, compared with the actual value. One-to-one correspondence, To the total number of auxiliary regression parameters, It is the square of the L2 norm (i.e., the mean square error). The auxiliary loss weight is a hyperparameter, and its value can be set from 0.02 to 0.10.

[0233] The purpose of setting the auxiliary regression loss is to force the feature extractor to predict the physical properties of the scene at the same time, so that the learned feature representation inherently encodes the information of "what visual interference conditions the current scene is under", thus enabling it to cope more robustly with various visual interferences in real surgical scenes that it has never seen before.

[0234] It should be noted that during the training process, a domain-adaptive training method can be used, which may include the following steps:

[0235] Step A: Initialize the feature extractor (Image Net pre-trained weights can be used as initial values), Task Header Domain discriminator Assisted return head ;

[0236] Step B: Conduct training and set training rounds. ( (Total training rounds), the specific training process may include:

[0237] - Calculate the mixing ratio for the current round using a progressive mixing ratio scheduling method. ;

[0238] - Mix batches from simulation sample set and video sample set Constructing mixed batches;

[0239] - Forward propagation: Output task prediction , Output domain discrimination result, Output physical parameter prediction ;

[0240] - Calculate loss using the multi-task joint loss function. Backpropagation updates all network parameters ( , , , );

[0241] Step C: Output the trained model .

[0242] This application embodiment constructs an anatomical structure model based on sample image data and generates corresponding organ digital models. These organ digital models are then input into a virtual reality simulation scene to perform laparoscopic surgical simulation operations on the organ digital models within the virtual reality simulation scene. Subsequently, according to a preset sampling period, a simulation sample set containing simulation operation images and their corresponding simulation image labels is acquired. This is combined with a video sample set containing video sample images and their corresponding video image labels to train the constructed image annotation model. The trained model can then output labels for images acquired during laparoscopic surgery to assist in the procedure. This approach reduces the pressure of acquiring real surgical data and the cost of manual annotation while providing training samples with clear structure, complete labels, and close resemblance to real surgical processes, thereby increasing the amount of sample data used for model training.

[0243] Figure 3A flowchart illustrating the image recognition method provided in this application is shown below. Figure 3 As shown, the image recognition method provided in this embodiment includes:

[0244] S301: Acquire images during laparoscopic surgery;

[0245] S302: Input the acquired image into the trained image annotation model to generate image labels for the acquired image. The trained image annotation model is a model trained according to the model training method applied to the annotation of laparoscopic surgery images described above.

[0246] In this embodiment, by acquiring images during laparoscopic surgery, the tissue structure, lesion morphology, and instrument status in the current surgical field can be used as objects to be identified and input into the subsequent identification process. The acquired images are then input into an image annotation model trained by a specific method to output corresponding image labels, so that the output image labels can be used to perform various intelligent auxiliary tasks for laparoscopic surgery, such as real-time segmentation and tracking of surgical instruments, real-time semantic segmentation of anatomical structures, automatic identification of surgical stages, and intelligent early warning of abnormal events.

[0247] During real-time segmentation and tracking of surgical instruments, real-time pixel-level segmentation and motion trajectory tracking of surgical instruments (such as ultrasonic scalpels, grasping forceps, electrocoagulation hooks, suction devices, etc.) in laparoscopic video frames can be performed based on image tags, providing a visual perception basis for intraoperative instrument collision warning and standardized operation assessment.

[0248] During real-time semantic segmentation of anatomical structures, key anatomical structures such as organ parenchyma, blood vessels, bile ducts, ureters, and nerve bundles in the laparoscopic field of view can be semantically segmented in real time based on image labels, which helps doctors clearly identify tissue layers and adjacent relationships and reduces the risk of misoperation.

[0249] During the automatic identification process of surgical stages, the current stage of the surgery (such as preparation, dissection, separation, hemostasis, suturing, and termination) can be automatically identified based on image tags, providing a temporal context for automatic generation of surgical records, key event marking, and process optimization analysis.

[0250] During the intelligent early warning process for abnormal events, risk events such as abnormal bleeding, accidental instrument contact with important anatomical structures, and accidental tissue perforation in the field of view can be detected based on image tags, and warnings can be issued to the surgeon in real time.

[0251] Figure 4 This is a schematic diagram of the model training device for laparoscopic surgical image annotation provided in this application, as shown below. Figure 4 As shown, the model training device 40 for laparoscopic surgical image annotation provided in this embodiment includes:

[0252] The organ generation module 401 is used to construct an anatomical structure model based on sample image data, and to perform biological simulation based on the anatomical structure model to generate a digital organ model; wherein the digital organ model includes at least one simulated organ.

[0253] Simulation module 402 is used to input digital organ models into a virtual reality simulation scene;

[0254] The data acquisition module 403 is used to acquire a simulation sample set according to a preset sampling period when the simulation device moves in the organ digital model in the virtual reality simulation scene, and / or when the simulation device performs laparoscopic surgery simulation operation in the organ digital model under the guidance of force feedback. The simulation sample set includes simulation operation images and simulation image labels corresponding to the simulation operation images.

[0255] The training module 404 is used to train the constructed image annotation model based on the simulation sample set and the video sample set to obtain the trained image annotation model. The video sample set includes video sample images and corresponding video image labels. The trained image annotation model is used to output the labels of the images acquired during the laparoscopic surgery.

[0256] The model training device 40 for laparoscopic surgical image annotation provided in this embodiment can execute the method provided in the above-described method embodiment. By constructing an anatomical structure model based on sample image data and performing biosimulation, a digital organ model with organ morphology, spatial relationships, and deformation characteristics can be formed, thereby providing a basic environment close to the real surgical field for subsequent simulation operations. The organ digital model is then imported into a virtual reality simulation scene, and the simulation instrument is controlled to perform laparoscopic surgical simulation along a preset path, which can stably reproduce multiple types of operation processes, so that the simulation operation images and corresponding labels can be automatically generated synchronously. By acquiring a simulation sample set according to a preset sampling period, a large-scale training data with high label consistency can be continuously accumulated, thereby reducing the dependence on real surgical video acquisition and manual frame-by-frame annotation. By combining the simulation sample set and the video sample set to train the image annotation model, the scale advantage of simulation data and the scene realism of real videos can be utilized simultaneously, thus helping to improve the model's adaptability to tissue deformation, lighting changes, and visual field interference, and improving the stability and accuracy of laparoscopic surgical image label output.

[0257] Figure 5 Schematic diagram of the model training device for laparoscopic surgical image annotation provided in this application Figure 2 ,like Figure 5 As shown, in this embodiment... Figure 4 Based on the embodiments, a model training device for laparoscopic surgical image annotation is described in detail. The model training device 50 for laparoscopic surgical image annotation includes:

[0258] The organ generation module 501 is used to construct an anatomical structure model based on sample image data, and to perform biological simulation based on the anatomical structure model to generate a digital organ model; wherein the digital organ model includes at least one simulated organ.

[0259] Simulation module 502 is used to input digital organ models into a virtual reality simulation scene;

[0260] The data acquisition module 503 is used to acquire a simulation sample set according to a preset sampling period when the simulation device moves in the organ digital model in the virtual reality simulation scene, and / or when the simulation device performs laparoscopic surgery simulation operation in the organ digital model under the guidance of force feedback. The simulation sample set includes simulation operation images and simulation image labels corresponding to the simulation operation images.

[0261] The training module 504 is used to train the constructed image annotation model based on the simulation sample set and the video sample set to obtain the trained image annotation model. The video sample set includes video sample images and corresponding video image labels. The trained image annotation model is used to output the labels of the images acquired during the laparoscopic surgery.

[0262] In one possible implementation, the device 50 further includes:

[0263] The noise module 505 is used to add noise to the simulation operation image according to the preset noise rules, and update the simulation sample set according to the simulation operation image after adding noise. The noise includes at least one of the following: electrosurgical smoke noise, lens contamination noise, and light fluctuation noise.

[0264] In one possible implementation, when the noise is electrosurgical smoke noise, the noise module 505 specifically includes:

[0265] The first noise submodule 5051 is used to time the trigger time in response to the received electric knife activation trigger command;

[0266] The first noise submodule 5051 is also used to generate an electric knife smoke noise layer according to the smoke particle density function when the timing duration is longer than the preset electric knife activation trigger time.

[0267] The first noise submodule 5051 is also used to add the electrosurgical smoke noise layer as a noise image to the simulation operation image.

[0268] In one possible implementation, when the noise is lens contamination noise, the noise module 505 specifically includes:

[0269] The second noise submodule 5052 is used to randomly sample from the texture library according to a preset time interval and a pollution event probability function to obtain a semi-transparent texture stain image.

[0270] The second noise submodule 5052 is also used to add semi-transparent textured stain images to the simulation job image in a weighted manner according to the transparency coefficient.

[0271] In one possible implementation, when the noise is light fluctuation noise, the noise module 505 specifically includes:

[0272] The third noise submodule 5053 is used to generate a light intensity function based on sinusoidal wave noise and random flicker noise;

[0273] The third noise submodule 5053 is also used to adjust the illumination intensity of the simulation image according to the illumination intensity function.

[0274] In one possible implementation, training module 504 includes:

[0275] The cumulative submodule 5041 is used to accumulate the training batches of the image annotation model, which is recorded as the training rounds;

[0276] The proportion submodule 5042 is used to determine the proportion of simulated data in the simulation sample set based on the training rounds;

[0277] The sampling module 5043 is used to randomly sample from the simulation sample set and the video sample set according to the proportion of the simulation data, and to form a mixed batch training sample set based on the random sampling results.

[0278] The first training submodule 5044 is used to train the constructed image annotation model based on the mixed batch training sample set.

[0279] In one possible implementation, the image annotation model is a generative adversarial network model, and the training module 504 includes:

[0280] The label submodule 5045 is used to input the simulation task image and the video sample image into the generative adversarial network model to generate the first predicted label corresponding to the simulation task image and the second predicted label corresponding to the video sample image.

[0281] The expectation submodule 5046 is used to generate the expected distribution of simulated data and the expected distribution of real data based on the first predicted label and the corresponding simulated image label, and the second predicted label and the corresponding video image label, respectively.

[0282] Loss submodule 5047 is used to generate a domain discrimination loss function based on the expected distribution of the simulated data and the expected distribution of the real data.

[0283] The second training submodule 5048 is used to train the constructed image annotation model according to the domain discrimination loss function.

[0284] In one possible implementation, the device 50 further includes:

[0285] The loss calculation module 506 is used to construct a cross-entropy loss function and a similarity coefficient loss function based on the first predicted label and the corresponding simulation image label, and the second predicted label and the corresponding video image label, and to record the weighted result of the cross-entropy loss function and the similarity coefficient loss function as the main loss function of the downstream task.

[0286] The loss calculation module 506 is also used to calculate the loss function and value of the downstream task's main loss function, domain discrimination loss function, and auxiliary regression loss based on preset coefficients;

[0287] The training module 504 is also used to train the constructed image annotation model based on the loss function and values.

[0288] The model training device 50 for laparoscopic surgical image annotation provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0289] Figure 6 A schematic diagram of the structure of the image recognition device provided in this application is shown below. Figure 6 As shown, the image recognition device 60 provided in this embodiment includes:

[0290] The acquisition module 601 is used to acquire images during the laparoscopic surgery process;

[0291] The generation module 602 is used to input the acquired image into the trained image annotation model to generate image labels for the acquired image. The trained image annotation model is a model trained according to the model training method applied to the annotation of laparoscopic surgery images described above.

[0292] The image recognition device 60 provided in this embodiment can execute the image recognition method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0293] Figure 7 A schematic diagram of the structure of the electronic device provided in this application. Figure 7 As shown, the electronic device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the electronic device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0294] In the specific implementation process, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to execute the above-mentioned model training method and / or image recognition method applied to laparoscopic surgical image annotation.

[0295] The specific implementation process of processor 701 can be found in the above method embodiments. By storing the above model training method and / or image recognition method in the memory as computer execution instructions and having them called and executed by the processor, the electronic device can undertake the task of training image annotation models based on simulation sample sets and video sample sets, and can also directly perform intraoperative image recognition and label output after training. This integrates laparoscopic imaging acquisition, sample processing, model training, and recognition deployment into a unified hardware platform, reducing the complexity of data transmission and calling caused by system dispersion. It allows model training results to be more efficiently transformed into intraoperative application capabilities, thereby improving adaptability to complex laparoscopic surgical scenarios, image annotation stability, and decision support efficiency. Therefore, it helps to reduce the burden of real sample acquisition and manual annotation and improves actual clinical auxiliary effects.

[0296] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0297] The memory may include random access memory (RAM) and non-volatile memory (NVM), such as at least one disk storage device.

[0298] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0299] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described model training method and / or image recognition method for laparoscopic surgical image annotation.

[0300] This application embodiment deploys a computer program product in a model training device, image processing terminal, or surgical assistance platform. It can directly call the aforementioned model training methods to generate a trained model, or call image recognition methods to label and output images acquired during laparoscopic surgery. This allows both the training and application phases to be implemented as software products. Since the computer program can be executed automatically by the processor, it reduces the burden of manual intervention in training process configuration and recognition process invocation. This enables the stable reuse of joint training capabilities based on simulation samples and video samples, thereby improving the model's adaptability to real intraoperative tissue deformation, lighting changes, and visual field interference. Therefore, this computer program product is easy to install and deploy on different hardware platforms and helps improve the accuracy, stability, and practicality of laparoscopic surgical image annotation and recognition.

[0301] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the aforementioned model training method and / or image recognition method for laparoscopic surgical image annotation.

[0302] This application embodiment pre-stores computer-executable instructions for model training methods or image recognition methods in a computer-readable storage medium. This allows the processor to directly call the corresponding program logic at the training or deployment end, thereby completing the construction of simulation samples, model training, and intraoperative image label output. This medium enables the relevant methods to be stably deployed as software products at the sample management or computing end, reducing the impact of manual configuration differences on the execution results. This results in better reproducibility and consistency in the model training and image recognition processes, facilitating their application in various laparoscopic surgical auxiliary devices. Therefore, it helps lower the system implementation threshold and enhances the practical value of intraoperative image annotation.

[0303] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0304] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0305] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0306] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0307] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0308] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0309] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0310] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A model training method for image annotation in laparoscopic surgery, characterized in that, include: Based on sample image data, an anatomical structure model is constructed, and biological simulation is performed based on the anatomical structure model to generate digital organ models; wherein, the digital organ model includes at least one simulated organ; The digital model of the organ is input into a virtual reality simulation scene; In the virtual reality simulation scene, the simulator moves in the digital model of the organ, and / or, when the simulator performs laparoscopic surgery simulation in the digital model of the organ under the guidance of force feedback, a simulation sample set is acquired according to a preset sampling period, wherein the simulation sample set includes simulation operation images and simulation image labels corresponding to the simulation operation images; The constructed image annotation model is trained based on the simulation sample set and the video sample set to obtain the trained image annotation model. The video sample set includes video sample images and corresponding video image labels. The trained image annotation model is used to output the labels of images acquired during laparoscopic surgery.

2. The method according to claim 1, characterized in that, The method further includes: According to a preset noise rule, noise is added to the simulated work image, and the simulated work image with added noise is updated. The noise includes at least one of the following: electrosurgical smoke noise, lens contamination noise, and light fluctuation noise.

3. The method according to claim 2, characterized in that, When the noise is electrosurgical smoke noise, adding noise to the simulated operation image according to preset noise rules specifically includes: In response to the received electrosurgical activation trigger command, the trigger time is timed; When the timing duration exceeds the preset electrosurgical activation trigger time, an electrosurgical smoke noise layer is generated based on the smoke particle density function; The electrosurgical smoke noise layer is added as a noise image to the simulation operation image.

4. The method according to claim 2, characterized in that, When the noise is lens contamination noise, adding noise to the simulated image according to preset noise rules specifically includes: Based on a preset time interval and a pollution event probability function, random sampling is performed from the texture library to obtain semi-transparent texture stain images; Based on the transparency coefficient, the semi-transparent textured stain image is added to the simulation job image in a weighted manner.

5. The method according to any one of claims 1-4, characterized in that, The image annotation model is a generative adversarial network model. The training of the constructed image annotation model based on the simulation sample set and the video sample set includes: The simulated task image and the video sample image are input into the generative adversarial network model to generate a first predicted label corresponding to the simulated task image and a second predicted label corresponding to the video sample image; Based on the first predicted label and the corresponding simulated image label, and the second predicted label and the corresponding video image label, the expected distribution of simulated data and the expected distribution of real data are generated respectively. Based on the expected distribution of the simulated data and the expected distribution of the real data, a domain discrimination loss function is generated; The constructed image annotation model is trained based on the domain discrimination loss function.

6. The method according to claim 5, characterized in that, The method further includes: Based on the first predicted label and the corresponding simulated image label, and the second predicted label and the corresponding video image label, a cross-entropy loss function and a similarity coefficient loss function are constructed, and the weighted result of the cross-entropy loss function and the similarity coefficient loss function is denoted as the downstream task main loss function; Based on preset coefficients, calculate the loss functions and values ​​of the downstream task main loss function, the domain discrimination loss function, and the auxiliary regression loss; The constructed image annotation model is trained based on the loss function and its value.

7. An image recognition method, characterized in that, include: Acquire images during laparoscopic surgery; The acquired image is input into the trained image annotation model to generate image labels for the acquired image, wherein the trained image annotation model is a model trained according to the model training method for laparoscopic surgery image annotation as described in any one of claims 1-6.

8. A model training device for image annotation in laparoscopic surgery, characterized in that, include: An organ generation module is used to construct an anatomical structure model based on sample image data, and to perform biological simulation based on the anatomical structure model to generate a digital organ model; wherein the digital organ model includes at least one simulated organ. The simulation module is used to input the digital model of the organ into a virtual reality simulation scene; The data acquisition module is used to move the simulation device in the virtual reality simulation scene in the digital model of the organ, and / or, when the simulation device performs laparoscopic surgery simulation operation in the digital model of the organ under the guidance of force feedback, acquire a simulation sample set according to a preset sampling period, wherein the simulation sample set includes simulation operation images and simulation image labels corresponding to the simulation operation images; The training module is used to train the constructed image annotation model based on the simulation sample set and the video sample set to obtain the trained image annotation model. The video sample set includes video sample images and corresponding video image labels. The trained image annotation model is used to output the labels of images acquired during laparoscopic surgery.

9. An image recognition device, characterized in that, include: The acquisition module is used to acquire images during laparoscopic surgery. The generation module is used to input the acquired image into the trained image annotation model to generate image labels for the acquired image, wherein the trained image annotation model is a model trained according to the model training method for laparoscopic surgery image annotation as described in any one of claims 1-6.

10. An electronic device / computer-readable storage medium / computer program product, characterized in that, The electronic device includes: a memory and a processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the model training method for laparoscopic surgical image annotation as described in any one of claims 1-6, or to perform the image recognition method as described in claim 7; and / or, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the model training method for laparoscopic surgical image annotation as described in any one of claims 1-6, or to execute the image recognition method as described in claim 7; and / or, The computer program product includes a computer program that, when executed by a processor, implements the model training method for laparoscopic surgical image annotation as described in any one of claims 1-6, or performs the image recognition method as described in claim 7.

Images