Image processing method, data processing method, medical system, equipment and medium

Abstract

This specification provides an image processing method, a data processing method, a medical system, a device and a medium. Based on this method, before specific implementation, a target weighted loss function can be constructed by introducing and utilizing a sample disparity map containing depth information; then, based on the target weighted loss function, an image processing model is trained to obtain a better ability to segment image regions at relatively deep positions at the far end of the field of view, and is suitable for processing images collected in scenes with more complex environments such as surgical scenes; during specific implementation, after obtaining the target image to be processed, the target image can be processed by calling the above-mentioned image processing model to accurately and efficiently segment the foreground area and background area in the target image, and obtain a target segmentation result with smaller errors and higher accuracy.

Description

Technical Field

[0001] This specification belongs to the field of medical device technology, and in particular relates to image processing methods, data processing methods, medical systems, equipment and media. Background Art

[0002] Usually, after a medical system acquires an endoscopic image of a surgical area, it is necessary to first perform image segmentation on the endoscopic image to separate a foreground area and a background area.

[0003] However, due to the complex scene environment in the surgical scene, for example, there are often interferences from environmental factors such as blood and flesh, smoke obstruction, reflection, uneven brightness, etc., which causes the endoscopic images collected in this scene to be different from the images collected in other conventional environments. The image quality is often relatively poor, affecting the accuracy of image segmentation.

[0004] At present, there is an urgent need for a method that can efficiently and accurately perform image segmentation processing on images collected in complex environments. Summary of the invention

[0005] This specification provides an image processing method, a data processing method, a medical system, a device and a medium, which can accurately and efficiently perform image segmentation processing on images collected in relatively complex environments such as surgical scenes, and obtain segmentation results with high accuracy and small errors.

[0006] An embodiment of the present specification provides an image processing method, comprising: acquiring a target image; calling an image processing model to process the target image to obtain a corresponding target segmentation result; wherein the image processing model is trained based on a target weighted loss function, and the target weighted loss function at least includes a segmentation loss term related to depth information.

[0007] An embodiment of the present specification also provides a data processing method, including: obtaining a sample image; generating a sample disparity map and a sample segmentation map based on the sample image; constructing a target weighted loss function using the sample disparity map; based on the target weighted loss function, training a network model using the sample image, the sample disparity map and the sample segmentation map to obtain an image processing model.

[0008] The embodiment of the present specification also provides a medical system, including an endoscope and an image processing device, wherein the image processing device is used to process the image collected by the endoscope using the image processing method.

[0009] The embodiment of the present specification also provides a computer device, including a processor and a memory for storing processor executable instructions, and the processor implements the relevant steps of the image processing method or the data processing method when executing the instructions.

[0010] The embodiments of the present specification also provide a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed, the relevant steps of the image processing method are implemented.

[0011] Based on the image processing method, data processing method, medical system, equipment and medium provided in this specification, before specific implementation, a target weighted loss function that can guide the model to strengthen the segmentation learning of image areas at the far end of the field of view and at deeper positions can be constructed by introducing and utilizing a sample disparity map containing depth information; then, based on the target weighted loss function, a network model is trained to obtain an image processing model that has good segmentation processing capabilities for image areas at relatively deep positions at the far end of the field of view and is suitable for processing endoscopic images collected in more complex environments such as surgical scenes.

[0012] During specific implementation, after obtaining the target image to be processed, the target image is processed by calling the above-mentioned image processing model, so as to accurately and efficiently identify and segment the foreground area and background area in the target image, and obtain a target segmentation result with smaller error and higher precision. Subsequently, the above-mentioned target segmentation result can be used as a guide to perform specific surgical operations more accurately.

[0013] Furthermore, before specific implementation, in the process of training the image processing model, the corrected sample disparity map can be introduced and used at the same time, so as to train the model's image segmentation ability while training the model's ability to correct the disparity map in parallel, thereby obtaining an image processing model that can perform image segmentation and disparity map correction at the same time, thereby improving the generalization ability of the model.

[0014] Accordingly, in a specific implementation, the above image processing model can be called to process the target image, and while obtaining the target segmentation result of the target image, the corrected disparity map of the target image can also be obtained. Subsequently, based on the above target segmentation result and the corrected disparity map, the three-dimensional information can be effectively used to perform specific surgical operations more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of this specification, the drawings required for use in the embodiments will be briefly introduced below. The drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 is a flowchart of an image processing method provided by an embodiment of this specification;

[0017] Figure 2 is a schematic diagram of an embodiment of an image processing method provided by an embodiment of this specification, in a scenario example;

[0018] Figure 3 is a schematic diagram of an embodiment of an image processing method provided by an embodiment of this specification, in a scenario example;

[0019] Figure 4 is a schematic diagram of an embodiment of an image processing method provided by an embodiment of this specification, in a scenario example;

[0020] Figure 5 is a schematic diagram of an embodiment of an image processing method provided by an embodiment of this specification, in a scenario example;

[0021] Figure 6 is a schematic diagram of an embodiment of an image processing method provided by an embodiment of this specification, in a scenario example;

[0022] Figure 7 is a schematic diagram of an embodiment of an image processing method provided by an embodiment of this specification, in a scenario example;

[0023] Figure 8 is a schematic diagram of an embodiment of an image processing method provided by an embodiment of this specification, in a scenario example;

[0024] Fig. 9 is a schematic diagram of an embodiment of an image processing method provided by an embodiment of this specification, in a scenario example;

[0025] Fig.10 is a schematic diagram of an embodiment of an image processing method provided by an embodiment of this specification, in a scenario example;

[0026] Fig.11 is a schematic diagram of an embodiment of an image processing method provided by an embodiment of this specification, in a scenario example;

[0027] Fig.12 is a schematic diagram of an embodiment of an image processing method provided by an embodiment of this specification, in a scenario example;

[0028] Fig.13 is a schematic diagram of an embodiment of an image processing method provided by an embodiment of this specification, in a scenario example;

[0029] Fig.14 is a flowchart of a data processing method provided by an embodiment of this specification;

[0030] Fig.15 is a schematic diagram of the structure of a computer device provided by an embodiment of this specification;

[0031] Fig.16 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present specification;

[0032] Fig.17 It is a schematic diagram of the structural composition of a medical system provided by an embodiment of this specification. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0034] See also Figure 1 As shown, the embodiment of this specification provides an image processing method. Among them, the method is specifically applied to the medical system side. When implemented specifically, the method may include the following contents.

[0035] S101: Acquire a target image.

[0036] S102: Calling an image processing model to process the target image to obtain a corresponding target segmentation result; wherein the image processing model is trained based on a target weighted loss function, and the target weighted loss function at least includes a segmentation loss term related to depth information.

[0037] In some embodiments, the target image may be an image that at least includes depth information. For example, the target image may be an image acquired by a binocular camera or other device, including a left image and a right image.

[0038] Furthermore, the target image may be an image containing depth information acquired in a scene with a relatively complex environment. For example, the target image may be an endoscopic image acquired in a scene with a relatively complex environment.

[0039] Of course, it should be noted that the target images listed above are only schematic illustrations. In specific implementations, according to specific processing requirements, the target images may also include images containing depth information collected in other application scenarios. For example, the target images may also be images collected by binocular surveillance cameras in traffic management scenarios. This specification does not limit this.

[0040] The following takes the target image as an endoscopic image collected in a surgical scene as an example to specifically describe the image processing method provided in this specification. For other types of target images, the following embodiments for processing endoscopic images collected in a surgical scene can be referred to.

[0041] In some embodiments, the above-mentioned image processing method can be specifically applied to a medical system (eg, a surgical robot, etc.). Figure 2 and Figure 3 .

[0042] in, Figure 2 Specifically, it is a control console of the medical system facing the doctor and operated and controlled by the doctor. Figure 3 The operating table of the medical system is a table facing the patient, responding to the operation instructions initiated by the doctor through the console, and performing specific surgical operations on the patient. The above-mentioned operating table can be specifically configured with relevant equipment such as an endoscope, a robotic arm, an image processing device, a robotic arm control device, etc.

[0043] The endoscope may specifically include a rigid endoscope and a flexible endoscope. The endoscope has components such as a camera (e.g., a binocular camera) and an image sensor. The camera is used to image the object to be photographed on the image sensor, and the image sensor is used to output an electrical signal according to the imaging of the camera, and the electrical signal is used to generate an image containing the object to be photographed.

[0044] When doctors use this medical system to perform surgery on patients, please refer to Figure 4 As shown, first, the target area A1-1 where the surgical operation is currently being performed under the endoscopic field of view can be photographed through the endoscope of the operating table to obtain the corresponding endoscopic image A1-2. Since the endoscope uses a binocular endoscope (including a left camera and a right camera), the collected endoscopic image includes two images, a left image and a right image. Among them, the left image is an image taken by the left camera, and the right image is an image taken by the right camera. Since the binocular cameras are parallel in spatial structure and maintain a certain distance (which can be recorded as the baseline distance), the two images contained in the collected endoscopic image have slightly different shooting angles, but the images are generally similar.

[0045] Then, the doctor can control the instruments and equipment configured on the operating table through the console according to the endoscopic image A1-2 to accurately perform specific surgical operations on the patient.

[0046] It should be noted that the target area A1-1 is an area under the surgical scene. Different from the conventional scene environment, it is relatively complex. It not only contains the patient's organs (for example, the target organs targeted by the current surgery), surgical instruments (for example, scissors, duckbill forceps, etc.) and other target objects of different types and sizes; but also in the process of surgery, it is easy to appear complex and changeable interference factors such as smoke, bleeding, reflection, uneven light and dark, making the environment more complex. For example, during surgery, the operation of surgical instruments may cause blood and flesh tissue to be blurred. For another example, electrocoagulation instruments may produce smoke, interfering with the surgical field of view. For another example, liquids and metal instrument heads of surgical instruments will produce local reflections. Moreover, the above situation often occurs at a position far from the field of view of the endoscope lens. This will cause the image quality of the endoscopic image collected by the endoscope to be relatively poor and unstable, especially the image area at the far end of the endoscope field of view and relatively deep position will become more blurred and difficult to identify, which increases the difficulty of machine segmentation of images and affects the surgical accuracy of the medical system.

[0047] In some embodiments, the target image (eg, endoscopic binocular image) may specifically be an image to be processed that is acquired through an endoscope before or during a surgical operation and contains the target area targeted by the surgical operation.

[0048] In some embodiments, the above-mentioned image processing model can be specifically understood as a network model that is trained using a target weighted loss function constructed based on a sample disparity map, and is at least capable of performing image segmentation processing on the foreground area and the background area of ​​the input image. The sample disparity map contains depth information. Accordingly, by training the network model using a target weighted loss function constructed based on the sample disparity map, the network model's deep learning of the segmentation processing of image areas that are far away from the end of the field of view and relatively deep can be strengthened in a targeted manner, thereby obtaining an image processing model with a better segmentation processing effect that is suitable for processing images collected in scenes with more complex environments such as surgical scenes. The target weighted loss function at least contains a segmentation loss term related to depth information.

[0049] In some embodiments, the above-mentioned image processing model may support image segmentation processing of the input image, and / or image processing of generating a corresponding corrected disparity map with higher accuracy based on the input image.

[0050] In some embodiments, during specific implementation, the target image can be used as a model input, input into an image processing model, and the model is run. The image processing model automatically identifies and segments the foreground area and the background area in the target image by processing the target image, and obtains and outputs the corresponding target segmentation result.

[0051] See also Figure 5 As shown, it is the segmentation result corresponding to the endoscopic image A1-2. Among them, the foreground area can be specifically understood as the area in the target area where the current surgical operation is relatively concerned, for example, the area within the white circle. The background area can be understood as other areas in the target area except the foreground area, for example, the area outside the white circle.

[0052] In some embodiments, after obtaining the target segmentation result, the method may further include, when implemented, controlling the robotic arm to perform corresponding surgical operations through a robotic arm control device according to the target segmentation result.

[0053] Specifically, for example, based on the target segmentation result, the image area in the target image that is segmented and marked as the foreground area can be further and more precisely recognized; and then based on the image recognition result, the robotic arm of the operating table can be controlled by the robotic arm control device to perform more precise surgical operations on the patient.

[0054] In some embodiments, before implementation, the image processing model may be trained in the following manner:

[0055] S1: Get sample image;

[0056] S2: Generate a sample disparity map and a sample segmentation map according to the sample image;

[0057] S3: Based on the target weighted loss function, the network model is trained using the sample image, the sample disparity map and the sample segmentation map to obtain an image processing model that meets the requirements.

[0058] Wherein, the target weighted loss function is constructed using a sample disparity map containing depth information. The target weighted loss function at least contains a segmentation loss term related to the depth information. The depth information can be specifically understood as the depth information obtained through the sample disparity map. Correspondingly, the target weighted loss function can also be understood as a loss function constructed using the sample disparity map. Since the sample disparity map is generated based on the sample image, the depth information can also be understood as the depth information in the sample image.

[0059] Through the above embodiments, an image processing model that supports image segmentation processing of an input image can be trained.

[0060] In some embodiments, considering that the disparity map directly obtained may often contain interference errors such as noise, missing values, outliers, etc., which may affect the subsequent use of the disparity map. In order to obtain a disparity map with smaller interference errors and higher precision, an image processing model that can simultaneously support the generation of a corrected disparity map with higher precision can also be trained.

[0061] In some embodiments, after generating a sample disparity map and a sample segmentation map according to a sample image, the method may further include: performing correction processing on the sample disparity map to obtain a corrected sample disparity map; accordingly, based on a target weighted loss function, training a network model using the sample image, the sample disparity map, the corrected sample disparity map and the sample segmentation map to obtain an image processing model that meets the requirements.

[0062] Through the above embodiments, an image processing model that supports both image segmentation processing of an input image and generation of a corrected disparity map based on the input image can be trained.

[0063] In some embodiments, the sample images may specifically be endoscopic images collected in a surgical scenario and used for model training.

[0064] In some embodiments, during implementation, an endoscope with a depth function (eg, structured light or time-of-flight method, etc.) may be used to acquire sample images with better effects.

[0065] In some embodiments, the above-mentioned generation of a sample disparity map according to a sample image may include the following when implemented: calling a preset image processing algorithm to process the sample image to obtain a sample disparity map.

[0066] Among them, the preset image processing algorithm includes at least one of the following: BM, SGBM, GC and other algorithms. Of course, it should be noted that the preset image processing algorithm listed above is only a schematic illustration. During specific implementation, other suitable algorithms can also be introduced and used as preset image processing algorithms according to specific application scenarios and processing requirements. This specification does not limit this.

[0067] In some embodiments, during specific implementation, a sample depth map and camera parameters of the endoscope may be first acquired; and then, according to the camera parameters, the sample depth map may be converted into a corresponding sample disparity map.

[0068] In some embodiments, the sample segmentation map may also be referred to as a real sample segmentation image. The above-mentioned generation of the sample segmentation map based on the sample image may include: detecting the sample image according to the annotation rule, identifying the ROI (Region of Interest) area (e.g., foreground area, background area, etc.) in the sample image; and annotating the ROI area to obtain the corresponding sample segmentation map. For example, Figure 5 .

[0069] In some embodiments, the sample disparity map is corrected to obtain a corrected sample disparity map. When implemented specifically, the following contents may be included: filtering the sample disparity map, and then detecting and correcting outliers in the filtered sample disparity map to obtain a corrected sample disparity map.

[0070] See also Figure 6 As shown, the modified sample disparity map (or modified sample disparity map) obtained by the above processing has significantly suppressed interference errors such as noise, missing values, outliers, etc., compared with the original uncorrected sample disparity map, and the image accuracy is improved.

[0071] In some embodiments, based on the target weighted loss function, the network model is trained using the sample disparity map, the corrected sample disparity map and the sample segmentation map. The specific implementation may include the following:

[0072] S1: Call the network model to process the sample segmentation map to obtain the predicted sample disparity map, the predicted sample segmentation map, and the predicted corrected sample disparity map;

[0073] S2: Based on the target weighted loss function, the target weighted loss value is calculated using the sample image, the sample disparity map, the corrected sample disparity map, the sample segmentation map, the predicted sample disparity map, the predicted sample segmentation map and the predicted corrected sample disparity map;

[0074] S3: Update the network model according to the target weighted loss value.

[0075] In some embodiments, the network model may be a basic model including a feature extraction network. The feature extraction network may be a feature extraction network based on a Unet structure. Figure 7 As shown in the figure, the left half is the downsampling process. After each downsampling, the number of feature channels doubles and the size of the feature map is halved; the right half is the upsampling process. After each upsampling, the number of feature channels is halved and the size of the feature map doubles, and then stacked with the same layer channel of the downsampling. In the figure, the icon Represents the convolution activation operation, icon Indicates downsampling operation, icon Represents an upsampling operation.

[0076] The above feature extraction network can also be a feature extraction network based on the Deeplab structure. Figure 8 As shown, the icon Represents a convolution activation operation. When extracting image features, a ResNet or VGG network can be used. Based on this feature extraction network, the extracted features can be pooled at different image scales to obtain semantic representations at different scales. Based on this feature extraction network, the number of channels of different pooled features can be changed through a 1×1 convolution operation. Based on this feature extraction network, the pooled features of each scale are upsampled to the original image feature size and channel splicing is performed. Among them, upsampling can be specifically implemented using the difference method.

[0077] The above feature extraction network can also be a feature extraction network based on the SPNet structure. Fig. 9 As shown in the figure, CNN is the backbone network of the network feature extraction network, which can be constructed by using networks such as ResNet or VGG. The ASPP module is mainly constructed by structures such as 1×1 convolution, dilated convolution, and pooling, and can extract semantic information of different receptive fields of feature maps. The above-mentioned dilated convolution can specifically improve the receptive field range of the convolution kernel through the same parameter amount, where R represents the dilation rate. Based on this feature extraction network, shallow semantic information and deep semantic information can be stacked on the channel through splicing operations to obtain features at different resolutions and increase the richness of semantics. When performing dilated convolution, the gray part of the convolution kernel used is the parameter, and the white part between the gray is filled with 0 values. In this way, the receptive field of the 5×5 convolution kernel can be achieved through the 3×3 convolution kernel, and the parameters that need to be learned are greatly reduced. The above pooling operation can be specifically a downsampling process, in which the output image is calculated by averaging, finding the maximum value, finding a random value, etc. in a certain area (gray) of the input image. In this way, the image can be downsampled without losing the image features as much as possible, thereby reducing the amount of calculation parameters.

[0078] Of course, it should be noted that the feature extraction network listed above is only a schematic illustration. In specific implementation, according to specific application scenarios and processing requirements, feature extraction networks with other suitable structures can also be selected and used.

[0079] In some embodiments, the calling network model obtains a predicted sample disparity map, a predicted sample segmentation map, and a predicted corrected sample disparity map by processing the sample image. When implemented specifically, the following contents may be included:

[0080] S1: Call the network model to extract features from the sample image to obtain a sample disparity feature map, a sample segmentation feature map, and a predicted and corrected sample disparity feature map;

[0081] S2: Generate a predicted sample disparity map, a predicted sample segmentation map, and a predicted corrected sample disparity map according to the sample disparity feature map, the sample segmentation feature map, and the predicted corrected sample disparity feature map.

[0082] The predicted sample disparity map, the preset sample segmentation map, and the predicted corrected sample disparity map generated by the network model based on the above method can be specifically understood as the disparity map, the segmentation map, and the corrected sample disparity map obtained after being processed by the network model, respectively.

[0083] It should be noted that, for an image processing model that only supports image segmentation processing of an input image, the predicted corrected sample disparity map may not be generated or used during the training process.

[0084] In some embodiments, further, the calling network model extracts features from the sample segmentation map to obtain a sample disparity feature map, a sample segmentation feature map, and a predicted corrected sample disparity feature map. When specifically implemented, the following may be included:

[0085] S1: Calling the network model to extract image features from the sample segmentation map to obtain a sample feature map;

[0086] S2: generating a sample disparity feature map and a sample segmentation feature map according to the sample feature map;

[0087] S3: splicing the sample disparity feature map and the sample segmentation feature map to obtain a spliced ​​sample feature map;

[0088] S4: performing a convolution operation on the spliced ​​sample feature map to obtain a predicted and corrected sample disparity feature map.

[0089] For details, see Fig.10 As shown, the feature extraction network of the network model can be called to extract features from the sample image to obtain corresponding sample feature maps; then different network layers are called to perform different multi-layer convolution operations on the sample feature maps to obtain corresponding sample disparity feature maps and sample segmentation feature maps. The multi-layer convolution operation includes: using a structure formed by stacking multiple convolution layers and activation functions to form the output layer of each branch to process and output corresponding data.

[0090] Furthermore, the sample disparity feature map may be processed to generate a corresponding predicted sample disparity map; and the sample segmentation feature map may be processed to generate a corresponding predicted sample segmentation map.

[0091] In addition, the sample disparity feature map and the sample segmentation feature map can be spliced ​​to obtain a spliced ​​sample feature map. Specifically, the sample disparity feature map and the sample segmentation feature map can be spliced ​​in the channel dimension between different feature channels (which can be recorded as ) to obtain the sample feature map after splicing that meets the requirements.

[0092] Then, a multi-layer convolution operation is performed on the above-mentioned spliced ​​sample feature map to obtain a predicted and corrected sample disparity feature map.

[0093] Furthermore, the corresponding predicted corrected sample disparity map may be generated by processing the predicted corrected sample disparity feature map.

[0094] In some embodiments, the target weighted loss value is calculated based on the target weighted loss function using the sample image, the actual disparity map, the corrected sample disparity map, the sample segmentation map, the predicted sample disparity map, the predicted sample segmentation map and the predicted corrected sample disparity map. When implemented specifically, the following contents may be included:

[0095] S1: Calculate a disparity loss value using the sample disparity map and the predicted sample disparity map;

[0096] S2: Calculate a segmentation loss value according to the sample disparity feature map, the predicted sample segmentation map and the sample segmentation map;

[0097] S3: Calculate the corrected loss value using the predicted corrected sample disparity map and the corrected sample disparity map;

[0098] S4: Based on the target weighted loss function, the target weighted loss value is calculated using the disparity loss value, the segmentation loss value, and the correction loss value.

[0099] In some embodiments, the above target weighted loss function may further include: a correction loss term and a parallax loss term.

[0100] In some embodiments, the target weighted loss function may specifically be a weighted sum function of a disparity loss term, a correction loss term, and a segmentation loss term.

[0101] For example, see Fig.10As shown, the corresponding Loss1 calculated by combining the sample disparity map and the predicted sample disparity map can be used as the disparity loss value; the corresponding Loss2 calculated by combining the predicted sample segmentation map and the sample segmentation map can be used as the segmentation loss value; the corresponding Loss3 calculated by combining the corrected sample disparity map and the sample disparity map can be used as the corrected loss value. Then, by using the weight coefficient that takes into account the depth information in the disparity map, the disparity loss value, the segmentation loss value and the corrected loss value are calculated in combination, and the total target weighted loss value Loss that can specifically strengthen the image segmentation learning of the relatively deep position at the far end of the field of view can be obtained.

[0102] In some embodiments, the segmentation loss value is calculated based on the sample disparity feature map, the predicted sample segmentation map and the sample segmentation map. When implemented specifically, the following contents may be included:

[0103] S1: using an activation function, mapping the pixel values ​​of each pixel in the sample disparity feature map to obtain a corresponding sample weight map;

[0104] S2: Calculate the segmentation loss value based on the sample weight map, the predicted sample segmentation map and the sample segmentation map.

[0105] Through the above embodiment, using the sample weight map (which can be simply referred to as the weight map), the weight value of the pixel point segmentation loss at a relatively deep position at the far end of the field of view can be increased in a targeted manner to strengthen the learning of the segmentation processing of the image area relatively deep at the far end of the field of view.

[0106] In this embodiment, the activation function used may specifically be a softmax function.

[0107] In this embodiment, during specific implementation, the above activation function may be used to map the pixel value of each pixel point in the sample disparity feature map containing depth information to a value range of (0, 1).

[0108] Specifically, the mapped pixel value (or softmax value) of any pixel numbered i can be expressed as follows:

[0109]

[0110] Among them, i, j are the numbers (or indices) of the pixel points, and n is the total number of pixel points.

[0111] The sample weight map obtained by the above mapping process can represent the probability map of the pixel values ​​in the (0,1) interval after the sample disparity feature map is mapped by the softmax function. Each pixel value in the sample weight map represents the weight value of the corresponding pixel point used to calculate the segmentation loss value. Based on the above sample weight map, the network model can prioritize the segmentation loss value of the pixel points with large weight values.

[0112] Specifically, when calculating the segmentation loss value, for example, the sample weight map used is a 5×5 matrix, and the predicted sample segmentation map and the sample segmentation map (i.e., the actual sample segmentation map) are also 5×5 matrices. At this time, the elements at the same position of the above three matrices can be multiplied first, that is, the weight value is weighted on the segmentation loss; then the elements contained in the obtained matrix are summed to obtain the final weighted segmentation loss value Loss2 (the entire operation process can be recorded as ).

[0113] In some embodiments, during specific implementation, the following formula may be used as the target weighted loss function:

[0114]

[0115] α+β+γ=1

[0116] Among them, Loss is the target weighted loss value, Loss1 is the parallax loss value, Loss2 is the segmentation loss value, Loss3 is the correction loss value, α is the first weight coefficient, β is the second weight coefficient, γ is the third weight coefficient, N is the number of pixels, i is the number of pixels, d is the pixel number, i ′ is the pixel value of the pixel numbered i in the predicted sample disparity map, d i is the pixel value of the pixel numbered i in the sample disparity map, w i is the weight value of pixel number i in the weight map, y i is the pixel value of the pixel numbered i in the sample segmentation map, p i To predict the pixel value of the pixel numbered i in the sample segmentation map, t i ′ is the pixel value of the pixel numbered i in the predicted corrected sample disparity map, t i To correct the pixel value of the pixel numbered i in the sample disparity map.

[0117] Based on the above target loss function, the corresponding segmentation loss value, disparity loss value, and correction loss value can be calculated first; and then the total target weighted loss value can be calculated.

[0118] It should be noted that, for an image processing model that only supports image segmentation processing of an input image, the corrected loss value may not be calculated when calculating the target weighted loss value.

[0119] In some embodiments, in the process of training the network model based on the target weighted loss function and using the sample image, sample disparity map, corrected sample disparity map and sample segmentation map, the method may also include the following contents when implemented: monitoring whether the target weighted loss value converges for the first time; when monitoring that the target weighted loss value converges for the first time, according to the preset weight coefficient adjustment rule, reducing the values ​​of the first weight coefficient and the second weight coefficient, and increasing the value of the third weight coefficient, to obtain an updated target weighted loss function; accordingly, based on the updated target weighted loss function, training the network model using the sample image, sample disparity map, corrected sample disparity map and sample segmentation map.

[0120] In this embodiment, when training the network model, multiple iterative training can be performed using sample images, sample disparity maps, corrected sample disparity maps, and sample segmentation maps. After each iterative training, the average value of the target weighted loss function value during the iterative training can be calculated based on the target weighted loss function; and the average value of the target weighted loss function value of this time can be compared with the average value of the target weighted loss function of the previous time to obtain a comparison result; and then, based on the comparison result, it is determined whether the target weighted loss value has converged for the first time.

[0121] In this embodiment, before the first convergence occurs, an initial coefficient group (including: an initial first weight coefficient, an initial second weight coefficient, and an initial third weight coefficient) can be used to construct a target weighted loss function; and based on the target weighted loss function, iterative training is performed using sample images, sample disparity maps, corrected sample disparity maps, and sample segmentation maps.

[0122] When setting the initial coefficient group, the values ​​of the first weight coefficient and the second weight coefficient can be set relatively large, and the value of the third weight coefficient can be set relatively small. For example, in the initial coefficient group, the value range of the initial first weight coefficient can be [0.4, 0.45], the value range of the initial second weight coefficient can be [0.4, 0.45], and the value range of the initial third weight coefficient can be [0.1, 0.2]. In addition, the sum of the initial first weight coefficient, the initial second weight coefficient and the initial third weight coefficient is equal to 1.

[0123] By using the above initial coefficient group for iterative training in the early stage of model training before the first convergence, the network model can be guided to focus on learning sub-processing and disparity map generation in a targeted manner.

[0124] In this embodiment, specifically, the comparison result can be compared with a preset threshold parameter. When it is determined that the absolute value of the comparison result is equal to the preset threshold parameter and the target weighted loss value has not yet stabilized and the first convergence has not occurred, the initial coefficient group can continue to be used for iterative training.

[0125] When it is determined that the absolute value of the comparison result is less than or equal to the preset threshold parameter, it is determined that the target weighted loss value has stabilized. When the first convergence occurs, the initial coefficient group can be adjusted according to the preset weight coefficient adjustment rules, the values ​​of the first weight coefficient and the second weight coefficient can be reduced, and the value of the third weight coefficient can be increased to obtain an updated coefficient group; based on the updated coefficient group, an updated target weighted loss function can be obtained.

[0126] Furthermore, the updated coefficient group can be used to replace the initial coefficient group to continue iterative training.

[0127] Among them, in the updated coefficient group, the updated first weight coefficient may be in the range of [0.1, 0.2], the updated second weight coefficient may be in the range of [0.1, 0.2], and the updated third weight coefficient may be in the range of [0.6, 0.8]. Moreover, the sum of the updated first weight coefficient, the updated second weight coefficient and the updated third weight coefficient is equal to 1.

[0128] By using the updated coefficient group instead of the initial coefficient group for iterative training in the later stage of model training after the first convergence, the network model can be guided to focus on learning the fine-tuning and correction of the disparity map in a targeted manner.

[0129] In some embodiments, in the process of training the network model based on the updated target weighted loss function and using the sample image, sample disparity map, corrected sample disparity map and sample segmentation map, the method may also include the following contents when implemented: monitoring whether the target weighted loss value converges for the second time; when monitoring that the target weighted loss value converges for the second time, stopping training to obtain the image processing model.

[0130] In this embodiment, in the iterative training after the first convergence, if the target weighted loss value is monitored to converge again (i.e., the second convergence), it can be determined that the current training has been completed, and the training can be stopped, and the current network model is used as an image processing model that meets the requirements. On the contrary, if the target weighted loss value is not monitored to converge for the second time, the iterative training can continue.

[0131] Through the above embodiments, the required image processing model can be trained more accurately.

[0132] In some embodiments, when it is monitored that the target weighted loss value has not converged, the gradient parameters can be calculated first; then, based on the gradient parameters, the network parameters and weight parameters of the network model are updated, and the network model is forward propagated for iterative training.

[0133] In some embodiments, after the image processing model is obtained, the method may further include the following when implemented:

[0134] S1: Call the image processing model to process the test image and obtain the corresponding test result;

[0135] S2: Calculate evaluation parameters for the image processing model based on the test results;

[0136] S3: Determine whether the image processing model meets the preset accuracy requirements based on the evaluation parameters.

[0137] Through the above-mentioned embodiments, the converged image processing model can be tested using the test image, and whether the accuracy of the currently preset processing model meets the preset accuracy requirement can be accurately determined based on the test result.

[0138] In some embodiments, the above evaluation parameters include: Dice coefficient (dice similarity coefficient) and / or IoU (intersection over union) coefficient. Among them, Dice coefficient and IoU coefficient are indicator parameters used to evaluate the image segmentation effect. Specifically, the value range of Dice coefficient and IoU coefficient is (0,1). The larger the value of Dice coefficient and IoU coefficient, the better the image segmentation effect.

[0139] In some embodiments, during specific implementation, the Dice coefficient can be calculated according to the following formula:

[0140]

[0141] Among them, TP (True Positive) represents the area of ​​the foreground area in the actual annotation result and is correctly marked as the foreground area in the segmentation result (or prediction result). For example, Fig.11 The area enclosed by the solid line in the middle; FN (False Negative) indicates the area of ​​the area that is actually the foreground area in the true annotation result but is incorrectly marked as the foreground area in the segmentation result, for example, Fig.11 The area enclosed by the dotted line in the middle; FP (False Positive) indicates the area of ​​the area that is actually the background area in the true annotation result but is mistakenly marked as the foreground area in the segmentation result, for example, Fig.11 The area enclosed by the midpoint dashed line.

[0142] In some embodiments, during specific implementation, the above IoU coefficient can be calculated according to the following formula:

[0143]

[0144] In some embodiments, during specific implementation, the evaluation parameter can be compared with a preset evaluation parameter threshold. If the evaluation parameter is greater than or equal to the preset evaluation parameter threshold, it is determined that the current image processing model has met the preset accuracy requirement. On the contrary, if the evaluation parameter is less than the preset evaluation parameter threshold, it is determined that the current image processing model cannot meet the preset accuracy requirement and further training is required.

[0145] Specifically, for example, when it is determined through numerical comparison that the obtained Dice coefficient is less than a preset evaluation parameter threshold (0.95), it can be determined that the current image processing model does not meet the preset accuracy requirement.

[0146] In some embodiments, after determining whether the image processing model meets the preset accuracy requirement according to the evaluation parameters, the method may further include the following when implemented:

[0147] S1: Collect error samples based on test results;

[0148] S2: Determine the error type of the image processing model according to the error sample;

[0149] S3: Determine the cause of the error according to the error type;

[0150] S4: Determine a matching target adjustment strategy according to the error type and error cause;

[0151] S5: Train the image processing model based on the target adjustment strategy.

[0152] In some embodiments, the above-mentioned erroneous samples can be specifically understood as sample images with large differences in segmentation and labeling between the predicted sample segmentation map (or predicted sample segmentation result) output by the model and the original sample segmentation map.

[0153] In some embodiments, the error type of the erroneous sample may be determined according to the difference type between the predicted sample segmentation map and the original sample segmentation map.

[0154] In some embodiments, error analysis can be performed based on error samples to determine the error type of the image processing model. Specifically, the error sample statistics can be used to calculate the sample proportions corresponding to various error types, and the error type with the highest sample proportion is determined as the error type of the image processing model.

[0155] The above error types may specifically include: over-segmentation and under-segmentation.

[0156] See also Fig.12 As shown, the above over-segmentation can be specifically understood as an error of segmenting an image area that originally belongs to the background area into the foreground area. The above under-segmentation can be specifically understood as an error of not segmenting an image area that originally belongs to the foreground area into the foreground area.

[0157] In some embodiments, after the error type is determined, a targeted error cause analysis may be performed based on the error type to determine the error cause that currently causes the error of the image processing model.

[0158] For example, different error types can be distinguished, and the cause of the error can be found by determining whether the image processing model has incorrect labeling during training, whether the number of samples is too small, whether the sample categories are unbalanced, whether the hyperparameter settings are unreasonable, etc.

[0159] In some embodiments, based on the error type and error cause, a preset adjustment strategy that matches the error type and error cause can be screened out from a plurality of pre-configured preset adjustment strategies as a target adjustment strategy; then, based on the target adjustment strategy, after performing corresponding adjustment processing, the image processing model is further trained, or the image processing model is retrained to finally obtain an image processing model that meets the preset accuracy requirements.

[0160] For example, if the segmentation results of sample images are wrong due to incorrect annotation, the annotation content can be manually modified based on the matching preset adjustment strategy, and the image processing model can be retrained using the modified data. For another example, if the test results and training results differ greatly due to the small number of sample images, more sample images can be collected based on the matching preset adjustment strategy, and the existing image processing model can be further trained using the more sample images.

[0161] In some embodiments, before implementation, a large amount of historical data may be learned in advance to configure multiple preset adjustment strategies, wherein each preset adjustment strategy may correspond to a combination of an error type and an error cause.

[0162] In some embodiments, after obtaining the target segmentation result, the method further includes: controlling the robotic arm to perform corresponding surgical operations through a robotic arm control device according to the target segmentation result.

[0163] In some embodiments, when the method is implemented, it may also include: calling an image processing model to process the target image to obtain a target segmentation result and a corrected disparity map; accordingly, according to the target segmentation result and the corrected disparity map, controlling the robotic arm through the robotic arm control device to perform corresponding surgical operations.

[0164] In this embodiment, the corrected disparity map and the target segmentation result can be used simultaneously to more finely analyze and identify the specific objects in the target area of ​​the current operation based on the three-dimensional dimension, so that the robotic arm can be more finely controlled to perform the corresponding operation and reduce surgical errors.

[0165] As can be seen from the above, based on the image processing method provided by the embodiment of this specification, before specific implementation, a target weighted loss function containing at least a segmentation loss term related to depth information can be constructed by introducing and using a sample disparity map containing depth information; then based on the target weighted loss function, the segmentation learning of the image area with poor image quality at the far end of the field of view can be improved, so that an image processing model with good segmentation processing ability for the image area at a relatively deep position at the far end of the field of view can be trained and obtained, which is suitable for processing images collected in scenes with complex environments such as surgical scenes. In specific implementation, after obtaining the target image to be processed, the target image can be processed by calling the above image processing model, and the foreground area and background area in the target image can be accurately and efficiently segmented to obtain a target segmentation result with small error and high accuracy; the above target segmentation result can be used later to more accurately guide specific surgical operations. Further, before specific implementation, in the process of training the image processing model, the corrected sample disparity map can also be introduced and used at the same time, so as to train the image segmentation ability of the model while training the corrected disparity map ability of the model in parallel, so that an image processing model that can perform image segmentation and disparity map correction at the same time can be obtained. Accordingly, in a specific implementation, the target image can be processed by calling the above-mentioned image processing model, and a target segmentation result and a corrected disparity map about the target image can be obtained at the same time; then, the above-mentioned target segmentation result and the corrected disparity map can be simultaneously utilized to more accurately guide specific surgical operations based on three-dimensional stereo information.

[0166] In a specific scenario example, endoscopic images at different time points during the operation can be collected to obtain multiple endoscopic images (as target images). Fig.13 The above-mentioned multiple endoscopic images correspond to the complex situations in which interference factors such as smoke, bleeding, reflection, and uneven brightness appear in the surgical scene.

[0167] Based on the image processing method provided in the embodiments of this specification, the above-mentioned image processing model can be called to process the above-mentioned multiple endoscopic images respectively to obtain corresponding multiple segmentation results (or segmentation result maps). At the same time, the technicians annotate the multiple endoscopic images respectively through manual annotation to obtain corresponding multiple annotation results (or annotation result maps).

[0168] See also Fig.13 , comparing the above segmentation results with the annotation results, it can be seen that the image processing method provided in the embodiment of this specification has good accuracy in segmenting and processing endoscopic images collected under various complex situations in surgical scenes, and is not affected by interference factors such as smoke, bleeding, reflections, uneven brightness, etc. in surgical scenes. Even for image areas that are far away from the lens field of view and located deeper, it has a good segmentation processing effect. It verifies that the image processing method provided in the embodiment of this specification can indeed accurately and efficiently segment the foreground area and background area in the target image, and obtain a target segmentation result with small errors and high accuracy.

[0169] See also Fig.14 As shown, the embodiment of this specification also provides a data processing method. Wherein, when the method is specifically implemented, it may include the following contents:

[0170] S1401: Acquire a sample image;

[0171] S1402: Generate a sample disparity map and a sample segmentation map according to the sample image;

[0172] S1403: constructing a target weighted loss function using the sample disparity map;

[0173] S1404: Based on the target weighted loss function, the network model is trained using the sample image, the sample disparity map and the sample segmentation map to obtain an image processing model that meets the requirements.

[0174] Among them, the above-mentioned target weighted loss function may at least include a parallax loss vector related to depth information. The above-mentioned sample image may specifically be an image acquired in the corresponding application scenario. Specifically, the above-mentioned sample image may be an endoscopic image acquired in a surgical scenario, or a monitoring image acquired in a traffic maintenance scenario, or a product image acquired in a product inspection scenario, and so on. Of course, it should be noted that the sample images listed above are only a schematic illustration. During specific implementation, according to the specific application scenario and processing requirements, images in other related scenarios may also be acquired as the above-mentioned sample images.

[0175] Through the above embodiments, an image processing model can be efficiently trained to have good segmentation processing capabilities for image regions that are relatively deep at the far end of the field of view and at least be able to achieve accurate image segmentation processing.

[0176] See also Fig.17 As shown, an embodiment of this specification also provides a medical system, which may specifically include an endoscope 1701 and an image processing device 1702, wherein the image processing device 1702 is used to process the image captured by the endoscope 1701 using the image processing method provided in the embodiment of this specification.

[0177] The endoscope 1701 may be a binocular endoscope. The image collected by the endoscope 1701 may be an endoscope image in a surgical scene.

[0178] The image processing device 1702 processes the image collected by the endoscope 1701 using the image processing method provided in the embodiment of this specification to obtain a corresponding processing result. The processing result may be a segmentation result for the image collected by the endoscope 1701, or a combination of a segmentation result and a corrected disparity map.

[0179] In some embodiments, the medical system may further include a robotic arm 1703 and a robotic arm control device 1704, and the robotic arm control device 1704 is used to control the robotic arm 1703 to perform corresponding surgical operations according to the processing results of the image processing device 1702.

[0180] An embodiment of the present specification also provides a computing device, including a processor and a memory for storing processor executable instructions. When the processor is implemented, it can perform the following steps according to the instructions: obtain a target image; call an image processing model to process the target image to obtain a corresponding target segmentation result; wherein the image processing model is trained based on a target weighted loss function, and the target weighted loss function at least includes a segmentation loss term related to depth information.

[0181] In order to complete the above instructions more accurately, refer to Fig.15 As shown, the embodiment of this specification also provides another specific computer device, wherein the computer device at least includes a network communication port 1501, a processor 1502 and a memory 1503, and the above structures are connected through internal cables so that each structure can perform specific data interaction.

[0182] The network communication port 1501 can be specifically used to obtain a target image.

[0183] The processor 1502 can be specifically used to call an image processing model to process the target image to obtain a corresponding target segmentation result; wherein the image processing model is trained based on a target weighted loss function, and the target weighted loss function at least includes a segmentation loss term related to depth information.

[0184] The memory 1503 may be specifically used to store corresponding instruction programs.

[0185] In this embodiment, the network communication port 1501 can be a virtual port that is bound to different communication protocols so that different data can be sent or received. For example, the network communication port can be a port responsible for web data communication, a port responsible for FTP data communication, or a port responsible for email data communication. In addition, the network communication port can also be a physical communication interface or communication chip. For example, it can be a wireless mobile network communication chip, such as GSM, CDMA, etc.; it can also be a Wifi chip; it can also be a Bluetooth chip.

[0186] In this embodiment, the processor 1502 may be implemented in any appropriate manner. For example, the processor may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) executable by the (micro)processor, a logic gate, a switch, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc. This specification does not limit this.

[0187] In this embodiment, the memory 1503 may include multiple levels. In a digital system, anything that can store binary data can be a memory; in an integrated circuit, a circuit with a storage function that has no physical form is also called a memory, such as RAM, FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, TF card, etc.

[0188] An embodiment of the present specification also provides a computer storage medium based on the above-mentioned image processing method, wherein the computer storage medium stores computer program instructions, which, when executed, implement: acquiring a target image; calling an image processing model to process the target image to obtain a corresponding target segmentation result; wherein the image processing model is trained based on a target weighted loss function, and the target weighted loss function at least includes a segmentation loss term related to depth information.

[0189] In this embodiment, the storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk (HDD), or a memory card. The memory may be used to store computer program instructions. The network communication unit may be an interface for network connection communication set in accordance with the standard specified by the communication protocol.

[0190] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer storage medium can be explained in comparison with other implementations and will not be described in detail here.

[0191] See also Fig.16 As shown, at the software level, the embodiment of this specification also provides an image processing device, which may specifically include the following structural modules:

[0192] An acquisition module 1601 may be specifically used to acquire a target image;

[0193] The processing module 1602 can be specifically used to call the image processing model to process the target image to obtain a corresponding target segmentation result; wherein the image processing model is trained based on a target weighted loss function, and the target weighted loss function at least includes a segmentation loss term related to depth information.

[0194] The embodiment of this specification also provides a data processing device, which includes at least the following structural modules:

[0195] An acquisition module, which can be used to acquire sample images;

[0196] A generation module, which can be specifically used to generate a sample disparity map and a sample segmentation map according to a sample image;

[0197] The training module can be specifically used to train the network model based on the target weighted loss function using the sample image, the sample disparity map and the sample segmentation map to obtain an image processing model that meets the requirements.

[0198] It should be noted that the units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0199] It can be seen from the above that the image processing device provided based on the embodiments of this specification can accurately and efficiently segment the foreground area and the background area in the target image, and obtain a target segmentation result with smaller error and higher precision, so that the robotic arm can be accurately controlled to perform specific surgical operations based on the above target segmentation results to reduce surgical errors.

[0200] Although the present specification provides method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent a unique execution order. When the device or client product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. The first, second, etc. words are used to represent the name, and do not represent any particular order.

[0201] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0202] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0203] Through the description of the above embodiments, it can be known that those skilled in the art can clearly understand that the present specification can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present specification can essentially be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in each embodiment of the present specification or some parts of the embodiments.

[0204] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. This specification can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0205] Although the present specification is described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present specification without departing from the spirit of the present specification, and it is intended that the appended claims include these modifications and changes without departing from the spirit of the present specification.

Claims

1. A method for processing an image, characterized in that: include: Acquire a target image; wherein the target image includes an endoscopic image acquired in a surgical scene with a complex environment; Calling an image processing model to process the target image to obtain a corresponding target segmentation result; wherein the image processing model is trained based on a target weighted loss function, the target weighted loss function at least includes a segmentation loss term related to depth information, the target weighted loss function is used to guide the model to strengthen the segmentation learning of image areas at the far end of the field of view and at a deep position, and the target segmentation result includes a foreground area and a background area identified and segmented in the target image; the target weighted loss function also includes: a correction loss term and a parallax loss term; The method also includes: in the process of training the network model based on the target weighted loss function, monitoring whether the target weighted loss value converges for the first time; when monitoring that the target weighted loss value converges for the first time, according to the preset weight coefficient adjustment rule, reducing the values ​​of the first weight coefficient corresponding to the disparity loss term and the second weight coefficient corresponding to the segmentation loss term, and increasing the value of the third weight coefficient corresponding to the correction loss term, to obtain an updated target weighted loss function; the first weight coefficient before adjustment and the second weight coefficient before adjustment are respectively greater than the third weight coefficient before adjustment; based on the updated target weighted loss function, continuing to train the network model to obtain the image processing model.

2. The image processing method according to claim 1, characterized in that: The method further comprises: Get a sample image; According to the sample image, generating a corresponding sample disparity map and a sample segmentation map; Based on the target weighted loss function, a network model is trained using the sample image, the sample disparity map and the sample segmentation map to obtain the image processing model.

3. The image processing method according to claim 2, characterized in that: After generating the corresponding sample disparity map and sample segmentation map according to the sample image, the method further includes: Performing correction processing on the sample disparity map to obtain a corrected sample disparity map; Accordingly, Based on the target weighted loss function, the network model is trained using the sample image, the sample disparity map, the corrected sample disparity map and the sample segmentation map to obtain the image processing model.

4. The image processing method according to claim 3, characterized in that: The target weighted loss function is based on the sample image, the sample disparity map, the corrected sample disparity map and the sample segmentation map to train the network model, including: Calling the network model to process the sample image to obtain a corresponding predicted sample disparity map, a predicted sample segmentation map, and a predicted corrected sample disparity map; Based on the target weighted loss function, a target weighted loss value is calculated using the sample disparity map, the corrected sample disparity map, the sample segmentation map, the predicted sample disparity map, the predicted sample segmentation map and the predicted corrected sample disparity map corresponding to the sample image; The network model is updated according to the target weighted loss value.

5. The image processing method according to claim 4, characterized in that: The calling of the network model to process the sample image to obtain a corresponding predicted sample disparity map, a predicted sample segmentation map, and a predicted corrected sample disparity map includes: Calling the network model to perform feature extraction on the sample image to obtain a sample disparity feature map, a sample segmentation feature map and a corrected sample disparity feature map; A predicted sample disparity map, a predicted sample segmentation map, and a predicted corrected sample disparity map are generated according to the sample disparity feature map, the sample segmentation feature map, and the corrected sample disparity feature map.

6. The image processing method according to claim 5, characterized in that: The calling of the network model to extract features from the sample image to obtain a sample disparity feature map, a sample segmentation feature map and a corrected sample disparity feature map includes: Call the network model to extract image features from the sample image to obtain a sample feature map; Generating the sample disparity feature map and the sample segmentation feature map according to the sample feature map; splicing the sample disparity feature map and the sample segmentation feature map to obtain a spliced ​​sample feature map; A convolution operation is performed on the spliced ​​sample feature map to obtain a modified sample disparity feature map.

7. The image processing method according to claim 4, characterized in that: Based on the target weighted loss function, a target weighted loss value is calculated using the sample disparity map, the corrected sample disparity map, the sample segmentation map, the predicted sample disparity map, the predicted sample segmentation map and the predicted corrected sample disparity map corresponding to the sample image, including: Calculating a disparity loss value using the sample disparity map and the predicted sample disparity map; Calculating a segmentation loss value according to the sample disparity feature map, the predicted sample segmentation map and the sample segmentation map; The predicted corrected sample disparity map and the corrected sample disparity map are used to calculate the corrected loss value; Based on the target weighted loss function, the target weighted loss value is calculated using the disparity loss value, segmentation loss value, and correction loss value.

8. The image processing method according to claim 7, characterized in that: The segmentation loss value is calculated according to the sample disparity feature map, the predicted sample segmentation map and the sample segmentation map, including: Using an activation function, mapping the pixel values ​​of each pixel in the sample disparity feature map to obtain a corresponding sample weight map; The segmentation loss value is calculated based on the sample weight map, the predicted sample segmentation map and the sample segmentation map.

9. The image processing method according to claim 7, characterized in that: The following formula is used as the objective weighted loss function: α+β+γ=1 Among them, Loss is the target weighted loss value, Loss1 is the parallax loss value, Loss2 is the segmentation loss value, Loss3 is the correction loss value, α is the first weight coefficient, β is the second weight coefficient, γ is the third weight coefficient, N is the number of pixels, i is the number of pixels, d is the pixel number, i ′ is the pixel value of the pixel numbered i in the predicted sample disparity map, d i is the pixel value of the pixel numbered i in the sample disparity map, w i is the weight value of the pixel numbered i in the sample weight map, y i is the pixel value of the pixel numbered i in the sample segmentation map, p i To predict the pixel value of the pixel numbered i in the sample segmentation map, t i ′ is the pixel value of the pixel numbered i in the predicted corrected sample disparity map, t i To correct the pixel value of the pixel numbered i in the sample disparity map.

10. The image processing method according to claim 9, characterized in that: In the process of training the network model based on the updated target weighted loss function and using the sample image, the sample disparity map, the corrected sample disparity map and the sample segmentation map, the method further includes: Monitoring whether the target weighted loss value converges for a second time; When it is monitored that the target weighted loss value converges for the second time, the training is stopped to obtain the image processing model.

11. The image processing method according to claim 2, characterized in that: After obtaining the image processing model, the method further includes: Calling the image processing model to process the test image and obtain the corresponding test result; Calculating evaluation parameters for the image processing model according to the test results; According to the evaluation parameters, it is determined whether the image processing model meets the preset accuracy requirements.

12. The image processing method according to claim 11, characterized in that: In the case where it is determined that the processing model does not meet the preset accuracy requirement, the method further includes: Collect error samples based on the test results; Determining the error type of the image processing model according to the error sample; According to the error type, determine the cause of the error; Determine a matching target adjustment strategy based on the error type and error cause; Based on the target adjustment strategy, the image processing model is trained accordingly.

13. The image processing method according to claim 3, characterized in that: The method further includes: calling an image processing model to process the target image to obtain a target segmentation result and a corrected disparity map.

14. A data processing method, characterized in that: include: Get a sample image; According to the sample image, a sample disparity map and a sample segmentation map are generated; Using the sample disparity map, a target weighted loss function is constructed; wherein the target weighted loss function is used to guide the model to strengthen the segmentation learning of image areas at the far end of the field of view and at a deep position; the target weighted loss function also includes: a correction loss term and a disparity loss term; Based on the target weighted loss function, the network model is trained using the sample image, the sample disparity map and the sample segmentation map to obtain an image processing model; The method also includes: in the process of training the network model based on the target weighted loss function and using the sample image, the sample disparity map and the sample segmentation map, monitoring whether the target weighted loss value converges for the first time; when monitoring that the target weighted loss value converges for the first time, according to a preset weight coefficient adjustment rule, reducing the values ​​of the first weight coefficient corresponding to the disparity loss term and the second weight coefficient corresponding to the segmentation loss term, and increasing the value of the third weight coefficient corresponding to the correction loss term, to obtain an updated target weighted loss function; the first weight coefficient before adjustment and the second weight coefficient before adjustment are respectively greater than the third weight coefficient before adjustment; based on the updated target weighted loss function, continuing to train the network model to obtain an image processing model.

15. A medical system comprising an endoscope and an image processing device, characterized in that: The image processing device is used to process the image collected by the endoscope using the image processing method described in any one of claims 1 to 13.

16. The medical system according to claim 15, characterized in that The medical system further comprises a robotic arm and a robotic arm control device, wherein the robotic arm control device is used to control the robotic arm to perform corresponding surgical operations according to the processing result of the image processing device.

17. A computer device, characterized in that: It comprises a processor and a memory for storing processor executable instructions, and the processor implements the relevant steps of the image processing method according to any one of claims 1 to 13 when executing the instructions.

18. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed, the relevant steps of the method described in any one of claims 1 to 13 or 14 are implemented.

Images