Focus size evaluation method and device, electronic equipment, storage medium and computer program product

Abstract

The invention relates to the technical field of image processing, and provides a lesion size evaluation method and device, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: inputting a to-be-evaluated image into a lesion recognition model, and obtaining a lesion depth image and a lesion edge contour image outputted by the lesion recognition model; based on the focus depth image and the focus edge contour image, evaluating the focus size; wherein the focus identification model is obtained by performing knowledge distillation based on unlabeled sample data with pseudo labels and labeled sample data through a teacher model; the teacher model is obtained by training based on a composite image and a label corresponding to the composite image; the label corresponding to the composite image comprises a focus depth annotation image and a focus edge contour annotation image; the pseudo label is generated based on the unlabeled sample data through the teacher model. According to the invention, the focus size can be accurately evaluated.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a lesion size assessment method, device, electronic device, storage medium and computer program product. Background Art

[0002] As a non-invasive, anesthesia-free endoscopic examination technology, capsule endoscopy is often used to examine various lesions of the digestive tract. The images collected by current capsule endoscopy are still two-dimensional images, and doctors need to identify the location of the two-dimensional image in the stomach based on their own knowledge and experience.

[0003] The development of artificial intelligence-assisted recognition technology for capsule endoscopes is a major breakthrough in recent years. However, although lesion detection technology has made significant progress, there are still few studies on endoscopic lesion size measurement. Accurate measurement of lesion size is important for evaluating the condition, formulating treatment plans, and monitoring treatment effects. However, existing capsule endoscopes are mainly monocular cameras, and the images they capture are all two-dimensional images, which makes it difficult to accurately assess the size of the lesion. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a lesion size assessment method, device, electronic device, storage medium and computer program product to solve the problem that capsule endoscopes are mainly monocular cameras, and the images captured are all two-dimensional images, which makes it difficult to accurately assess the size of lesions, and to achieve accurate assessment of the size of lesions.

[0005] The method for evaluating lesion size according to the first aspect of the present application includes: Inputting the image to be evaluated into a lesion recognition model to obtain a lesion depth image and a lesion edge contour image output by the lesion recognition model; Based on the lesion depth image and the lesion edge contour image, evaluating the lesion size; Among them, the lesion recognition model is obtained by performing knowledge distillation on the teacher model based on unlabeled sample data and labeled sample data with pseudo labels; the teacher model is trained based on synthetic images and labels corresponding to the synthetic images; the labels corresponding to the synthetic images include lesion depth labeled images and lesion edge contour labeled images; the pseudo labels are generated by the teacher model based on the unlabeled sample data.

[0006] According to one embodiment of the present application, the step of evaluating the lesion size based on the lesion depth image and the lesion edge contour image includes: Determine the size conversion factor between the under-the-mirror size and the actual size; Determine the disparity value of each pixel in the lesion depth image; Determining the actual physical coordinates of each pixel point of the lesion edge contour in the lesion edge contour image based on each of the disparity values ​​and the size conversion coefficient; The lesion size is evaluated according to the actual physical coordinates of each pixel point in the lesion edge contour.

[0007] According to an embodiment of the present application, before performing knowledge distillation based on unlabeled sample data and labeled sample data with pseudo labels, the method further includes: Perform data augmentation on unlabeled sample data with pseudo labels.

[0008] According to one embodiment of the present application, the synthetic image is generated based on a preset three-dimensional physical model; the preset three-dimensional physical model is constructed based on a virtual image engine.

[0009] According to one embodiment of the present application, the loss function of the teacher model includes depth estimation loss, image segmentation loss and affine invariant loss.

[0010] According to one embodiment of the present application, the loss function of the lesion recognition model includes depth estimation loss and image segmentation loss.

[0011] According to the second aspect of the present application, the lesion size assessment device includes: A recognition module, used for inputting the image to be evaluated into a lesion recognition model to obtain a lesion depth image and a lesion edge contour image output by the lesion recognition model; A determination module, configured to evaluate the size of a lesion based on the lesion depth image and the lesion edge contour image; Among them, the lesion recognition model is obtained by performing knowledge distillation on the teacher model based on unlabeled sample data and labeled sample data with pseudo labels; the teacher model is trained based on synthetic images and labels corresponding to the synthetic images; the labels corresponding to the synthetic images include lesion depth labeled images and lesion edge contour labeled images; the pseudo labels are generated by the teacher model based on the unlabeled sample data.

[0012] According to an electronic device of an embodiment of the third aspect of the present application, the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the lesion size assessment method described above is implemented.

[0013] According to the storage medium of the fourth aspect embodiment of the present application, the storage medium is a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements any of the lesion size assessment methods described above.

[0014] The computer program product according to the fifth aspect of the present application includes a computer program, which, when executed by a processor, implements any of the lesion size assessment methods described above.

[0015] The above one or more technical solutions in the embodiments of the present application have at least the following technical effects: A teacher model is obtained by pre-training based on a synthetic image and a lesion depth annotated image and a lesion edge contour annotated image corresponding to the synthetic image, and after the teacher model generates pseudo labels based on unlabeled sample data, the teacher model performs knowledge distillation based on the unlabeled sample data and the labeled sample data with pseudo labels to obtain a lesion recognition model. Therefore, when lesion recognition is required, the image to be evaluated is input into the lesion recognition model to obtain the lesion depth image and lesion edge contour image output by the lesion recognition model. Thus, the lesion size can be accurately evaluated through the lesion depth image and the lesion edge contour image.

[0016] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the lesion size assessment method provided in the embodiment of the present application.

[0019] Figure 2 It is a schematic diagram of a multi-task learning framework and knowledge distillation in the lesion size assessment method provided in an embodiment of the present application.

[0020] Figure 3 It is a schematic diagram of the pseudo-label generation and self-training process in the lesion size assessment method provided in an embodiment of the present application.

[0021] Figure 4 It is a schematic diagram of the input and output structure of lesion identification in the lesion size assessment method provided in an embodiment of the present application.

[0022] Figure 5 It is a schematic diagram of a lesion size assessment device provided in an embodiment of the present application.

[0023] Figure 6 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION

[0024] The following is a further detailed description of the implementation of the present application in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but cannot be used to limit the scope of the present application.

[0025] In the description of the embodiments of the present application, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the embodiments of the present application. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0026] In the description of the embodiments of the present application, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present application can be understood according to specific circumstances.

[0027] In the embodiments of the present application, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0028] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0029] It should be noted that traditional lesion detection and size measurement technologies mainly rely on two-dimensional images, using image processing and analysis algorithms to identify lesion areas and perform size measurements. These technologies are widely used in medical imaging, but have the following limitations: 1. Lack of depth information: Due to the limitations of two-dimensional images, it is difficult to accurately obtain three-dimensional information of the lesion, resulting in errors in size measurement. Many existing technologies rely on planar images for measurement and lack effective calibration of the lesion depth. The capsule endoscope moves in the digestive tract and is limited by narrow and curved spaces, making it difficult to obtain sufficient viewing angles for depth reconstruction.

[0030] 2. Lighting and texture effects: Lighting changes and tissue texture complexity in the image will affect the accuracy of lesion detection and measurement, leading to false detection and missed detection. The quality of endoscopic images is often limited by the interference of light source intensity and shooting angle.

[0031] 3. Data dependence: Lesion detection and measurement algorithms usually require a large amount of high-quality labeled data for training. However, these data are difficult to obtain in clinical environments. The image quality is limited by the cleanliness of the stomach, the capsule hardware configuration, and the image compression ratio, resulting in a lot of noise in the data set, which affects the training results.

[0032] 4. Computational complexity and real-time performance: The multi-view imaging and 3D reconstruction process is complex and computationally intensive, making it difficult to perform real-time 3D reconstruction. During real-time detection, the limitations of processing speed and computing power make it difficult for the system to provide accurate lesion detection and size measurement results in a short period of time.

[0033] Based on this, the present application proposes a lesion size assessment method, device, electronic device, storage medium and computer program product.

[0034] Figure 1 is a flow chart of the lesion size assessment method provided in the embodiment of the present application, such as Figure 1 As shown, the lesion size assessment method includes: Step 110 , input the image to be evaluated into a lesion recognition model, and obtain a lesion depth image and a lesion edge contour image output by the lesion recognition model.

[0035] Step 120 , evaluating the lesion size based on the lesion depth image and the lesion edge contour image.

[0036] Among them, the lesion recognition model is obtained by performing knowledge distillation on the teacher model based on unlabeled sample data and labeled sample data with pseudo labels; the teacher model is trained based on synthetic images and labels corresponding to synthetic images; the labels corresponding to synthetic images include lesion depth labeled images and lesion edge contour labeled images; pseudo labels are generated by the teacher model based on unlabeled sample data.

[0037] Furthermore, the synthetic image is generated based on a preset three-dimensional physical model; the preset three-dimensional physical model is constructed based on a virtual image engine.

[0038] The loss function of the teacher model includes depth estimation loss, image segmentation loss and affine invariance loss.

[0039] The loss function of the lesion recognition model includes depth estimation loss and image segmentation loss.

[0040] It should be noted that the executor of the lesion size assessment method provided in the embodiment of the present application can be a server, a computer device, etc., such as a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc.

[0041] The server or computer device of the present application may be provided with or connected to a lesion size assessment device, thereby controlling the lesion size assessment device to execute the lesion size assessment method of the present application.

[0042] This application can use virtual image engines such as Unreal Engine to build 3D models of objects such as the esophagus and stomach cavity, and further generate high-quality synthetic images with accurate depth information based on the 3D models. These synthetic images can make up for the lack of real medical image data, provide a large amount of high-quality data for depth estimation training, and also reduce image noise caused by image quality differences, capsule hardware differences, or compression ratio differences in real images.

[0043] Specifically, the present application can use Unreal Engine to build a three-dimensional model to simulate the anatomical structure of the esophagus and stomach cavity. Material parameters are set using a physically based rendering model, including reflectivity, roughness, and subsurface scattering.

[0044] Use Unreal Engine's Depth Pass rendering function to export the depth map, ensuring that the value range of the depth map is proportional to the actual distance.

[0045] Normalize the depth map values ​​to Range, calculated as: ; in, is the original depth value, and are the minimum and maximum depth values, respectively.

[0046] Therefore, when constructing a dataset, a virtual engine (such as Unreal Engine) can be used to build a three-dimensional model to simulate the structure of the esophagus and gastric cavity, generate high-quality color images and depth maps, and use these images to train models for depth estimation and lesion segmentation.

[0047] Furthermore, the composite image may be annotated manually or in other ways to generate a label of the composite image. The information annotation may include a lesion depth annotation and a lesion edge contour annotation, so the label of the composite image may include a lesion depth annotation image and a lesion edge contour annotation image.

[0048] At the same time, you can choose a complex and powerful model architecture (such as Vision Transformer or ResNet-50), use DINOv2 pre-trained weights to initialize the encoder, and use a dataset consisting of synthetic images and their corresponding labels to train the model to generate a teacher model that can accurately identify lesion depth and accurately segment lesions. Vision Transformer is a deep learning model designed specifically for computer vision tasks. ResNet-50 is a convolutional neural network, a model in the residual network family. DINOv2 is a computer vision model training method that uses self-supervised learning.

[0049] Specifically, during the pre-training process, the teacher model uses the pre-trained semantic segmentation model to detect the farthest region in the stomach image under the wide-angle field of view. The specific method can be to set the disparity value of the farthest region segmented as , as a reference benchmark for depth estimation.

[0050] The loss function of the teacher model is as follows: ; in, is the depth estimation loss, is the image segmentation loss, is the affine-invariant loss, are the weight coefficients of the corresponding losses respectively; The specific affine-invariant loss is: ; in, ; ; ; ; HW represents the total number of pixels in the image, which is the product of the image width (W) and height (H). This is used to normalize the loss value so that the loss calculation is not affected by the image resolution. The purpose of this is to get a consistent loss evaluation on images of different sizes. Represents the true depth value of the i-th pixel in the image, that is, the target depth value of the pixel (Ground Truth Depth). Represents the depth value (Predicted Depth) predicted by the model for the i-th pixel. In the loss calculation, by comparing and The difference between the model prediction and the true value is calculated to optimize the deep prediction ability of the model.

[0051] Furthermore, the present application can use the trained teacher model to generate pseudo labels for unlabeled real medical images. These pseudo labels serve as additional supervisory information for further training of the student model.

[0052] Furthermore, the present application may select a deep convolutional neural network model as the student model, for example, EfficientNet-B2. EfficientNet-B2 is a deep convolutional neural network model, which is mainly constructed based on a composite scaling strategy.

[0053] Furthermore, the unlabeled data with pseudo labels (i.e., the real medical images with pseudo labels) are mixed with the labeled data (synthetic images and their labels) to construct an enhanced training set, and the student model is self-trained through knowledge distillation technology. The self-training continuously optimizes the generalization ability and robustness of the student model through multiple iterations, and finally obtains a lesion recognition model that can output a lesion depth image (hereinafter referred to as a depth map) and a lesion edge contour image (hereinafter referred to as an edge contour map) based on the input image.

[0054] Specifically, when performing knowledge distillation, the distillation loss function can be: ; in and are the outputs of the teacher model and the student model respectively, is the softmax function, is the distillation temperature, is the loss weight, is the hard label loss.

[0055] More specifically, and They are the output logits of the teacher model and the student model respectively; represents the softmax function, which is used to convert logits into probability distribution; is the distillation temperature, which is used to smooth the logits so that the model can learn more subtle probability distributions from the soft labels of the teacher model. .

[0056] The loss weight determines the contribution ratio of the soft label (distilled part) and the hard label (true label) to the total loss. It can be adjusted according to the specific task. The value of KL is the divergence, is a measure of two probability distributions and Function of the measure of difference between: ; During the distillation process, .

[0057] It should be noted that during the training process, the data enhancement technology CutMix is ​​used.

[0058] The specific implementation is: the disparity space of the real image with pseudo labels is regionally mixed according to the randomly generated binary mask, and then the affine-invariant loss is calculated: ; in, Output logits for the student model; and They are the output logits of the teacher model respectively; is a binary mask used to blend different image regions; for , usually Smooth L1 Loss, is used to measure the difference between the output of the student model and the teacher model; Smooth L1 Loss is a loss function; is the spatial resolution of the image (i.e., the product of height and width), used to normalize the loss.

[0059] Figure 2 Schematic diagram of a multi-task learning framework and knowledge distillation in the lesion size assessment method provided in the embodiment of the present application, such as Figure 2 As shown, the present application provides a multi-task learning framework: supporting lesion segmentation and depth estimation tasks simultaneously. A dual decoder structure is used to process depth map generation and lesion area segmentation respectively, ensuring high efficiency and accuracy in multi-task situations.

[0060] EfficientNet-B2, which is trained and lightweight using knowledge distillation technology, is used as a shared encoder. After knowledge distillation, EfficientNet-B2 has the depth estimation and lesion segmentation capabilities inherited from the teacher model, and is more efficient in terms of inference speed and model size. EfficientNet-B2 is initialized with weights distilled from the teacher model. These weights combine the efficient feature extraction capabilities of the teacher model while also being optimized in terms of computational efficiency.

[0061] Dual decoder architecture design: including depth estimation decoder and lesion segmentation decoder, where the depth estimation decoder includes: multi-scale upsampling: from the multi-scale feature map extracted and output by the EfficientNet-B2 encoder, upsampling is performed step by step through the task execution path to restore the spatial resolution of the image and generate and output the depth map; skip connection: during the upsampling process, skip connections are used to pass the high-resolution features in the EfficientNet-B2 encoder directly to the decoder to improve the detail performance of depth estimation; loss function: use Smooth L1 Loss to optimize depth prediction: ; in, is the depth prediction generated by the student model, is the depth in pseudo-label or real annotated data.

[0062] Thus, the depth estimation decoder can complete the depth estimation task and output the lesion depth image D1 for three-dimensional reconstruction and lesion size estimation.

[0063] The lesion segmentation decoder includes: convolution and deconvolution layers: from the multi-scale feature map extracted and output by the EfficientNet-B2 encoder, convolution and deconvolution operations are performed through the task execution path to gradually restore the spatial resolution of the image, generate and output the lesion segmentation map; contextual attention mechanism: the contextual attention mechanism is introduced to enable the model to better focus on the lesion area during the segmentation process, reducing missed detection and false detection; loss function: Dice Loss is used to optimize lesion segmentation. Dice Loss is a loss function commonly used in image segmentation tasks: ; in, is the segmentation map generated by the student model, is the segmentation label in the real labeled data.

[0064] Thus, the lesion segmentation decoder can complete the lesion segmentation task and output the lesion edge contour image E1 for lesion area detection and diagnosis.

[0065] Multi-task joint loss function: In order to effectively train the model while performing depth estimation and lesion segmentation at the same time, a multi-task joint loss function is designed: ; in, and They are the loss weights for depth estimation and lesion segmentation, and the initial recommended values ​​are , which can be adjusted later according to the experimental results.

[0066] It should be further noted that the present application may add Dropout and weight decay (L2 regularization) in model training to prevent model overfitting. Dropout is a regularization technique used to prevent neural network overfitting.

[0067] Use gradient clipping technology to prevent gradient explosion problem, and the maximum norm of gradient can be set to 1.0.

[0068] Learning rate and optimizer: Use the learning rate scheduler CosineAnnealingLR to dynamically adjust the learning rate during training. The initial learning rate can be set to (encoder part), while the decoder part can use a higher learning rate (such as 10 times the encoder learning rate).

[0069] Optimizer selection: Use AdamW as the optimizer, combined with L2 regularization to enhance the generalization ability of the model. AdamW is an optimizer.

[0070] Model Validation and Deployment: (1) Model Validation: Cross-validation is performed on multiple datasets to evaluate the performance of the student model in depth estimation and lesion segmentation. MAE, RMSE, 10U, and Dice Coefficient are commonly used evaluation indicators that are suitable for different types of models and tasks.

[0071] Real-time inference and deployment: Deploy the trained EfficientNet-B2 in a computing environment that supports multi-task parallel processing to ensure efficient operation in multi-task scenarios (such as processing multiple images simultaneously).

[0072] Use tools such as TensorFlow Serving or PyTorch Serve to deploy model inference services to ensure responsiveness and accuracy in clinical applications. TensorFlow Serving and PyTorch Serve are tools for deploying machine learning models.

[0073] Model testing and verification: Multiple performance indicators are used for verification on real clinical images: precision, recall, F1-score, etc. F1-score is an indicator used to evaluate the performance of a binary classification model, which comprehensively considers the model's precision and recall.

[0074] Real-time test: Perform inference speed test to ensure that the response time of the model in actual application can meet the real-time requirements while ensuring the accuracy of the inference results.

[0075] The teacher model can perform knowledge distillation based on the shared encoder. Specifically, it can transfer knowledge between the teacher model and the student model through pseudo labels according to the knowledge distillation path, thereby completing the self-training of the student model.

[0076] Figure 3 is a schematic diagram of the pseudo-label generation and self-training process in the lesion size assessment method provided in the embodiment of the present application, such as Figure 3 As shown, the present application can generate pseudo labels as initial pseudo labels by predicting the teacher model after inputting unlabeled medical images, and obtain manually labeled data at the same time. Then, the pseudo label data can be mixed with the labeled data, and the combined data set can be used for student model training. Specifically, self-training iteration can be performed to achieve self-training output. If the optimized student model does not meet the requirements, continue to optimize; when the optimized student model meets the requirements, the optimized student model can be deployed, and a self-training cycle can be performed during the self-training process, and the model is optimized each time.

[0077] On this basis, the present application can use the image acquired by the capsule endoscope device as the image to be evaluated, and input it into the lesion recognition model, perform depth evaluation and lesion segmentation on the image to be evaluated through the lesion recognition model, and obtain the lesion depth image and lesion edge contour image output by the lesion recognition model after completing the depth evaluation and lesion segmentation.

[0078] Figure 4 is a schematic diagram of the input and output structure of lesion identification in the lesion size assessment method provided in the embodiment of the present application, such as Figure 4 As shown, Figure 4 The workflow of depth estimation and edge contour detection by the lesion recognition model in the embodiment of the present application is shown: the digestive tract image is input into the lesion recognition model, and the depth estimation and edge contour detection are performed through the model to obtain the output depth map and edge contour map. Among them, the depth map represents the depth information with different grayscale values, such as 0.2, 0.4, 0.6, 0.8, etc.; the unit of depth can be set according to actual needs, such as centimeters (cm) or millimeters (mm).

[0079] Furthermore, the present application can determine a size conversion coefficient between the under-mirror size and the actual size.

[0080] At the same time, the disparity value of each pixel in the lesion depth image is determined.

[0081] Then, based on each disparity value and the size conversion coefficient, the actual physical coordinates of each pixel point of the lesion edge contour in the lesion edge contour image are determined.

[0082] The lesion size is further evaluated based on the actual physical coordinates of each pixel point in the lesion edge contour.

[0083] After obtaining the lesion size, a lesion treatment plan can be further formulated based on the lesion size, which can be achieved through manual evaluation or deep learning, and is not specifically limited in this application. In a feasible embodiment, the lesion size can be input into a solution prediction model obtained by model training based on the sample lesion size and the corresponding sample treatment plan, and a lesion treatment plan output by the solution prediction model is obtained, so as to implement it after further feasibility evaluation based on the lesion treatment plan.

[0084] According to the lesion size assessment method of the embodiment of the present application, a teacher model is obtained by pre-training based on a synthetic image and a lesion depth annotated image and a lesion edge contour annotated image corresponding to the synthetic image, and after the teacher model generates pseudo labels based on unlabeled sample data, the teacher model performs knowledge distillation based on the unlabeled sample data and the annotated sample data with pseudo labels to obtain a lesion recognition model. Therefore, when lesion recognition is required, the image to be evaluated is input into the lesion recognition model, and the lesion depth image and lesion edge contour image output by the lesion recognition model can be obtained. Thus, the lesion size can be accurately assessed through the lesion depth image and the lesion edge contour image.

[0085] Based on the above embodiment, the lesion size is evaluated based on the lesion depth image and the lesion edge contour image, including: Determine the size conversion factor between the under-the-mirror size and the actual size; Determine the disparity value of each pixel in the lesion depth image; Based on each disparity value and the size conversion coefficient, determining the actual physical coordinates of each pixel point of the lesion edge contour in the lesion edge contour image; The lesion size is evaluated based on the actual physical coordinates of each pixel in the lesion edge contour.

[0086] Specifically, the present application can pre-construct standardized physical models (such as esophagus and stomach models) to establish an accurate size reference system.

[0087] Furthermore, the capsule endoscope is used to simulate the real environment (underwater) to shoot objects of known size, measure the size of the target in the image, and convert the size under the endoscope into the actual size through the size conversion coefficient. This calibration process ensures that when shooting in vivo and underwater, the actual size of the lesion can be accurately estimated while avoiding the influence of the difference in refractive index between air and underwater and the irregular cavity.

[0088] More specifically, the following steps may be included: Data collection and model making: Use transparent materials to make standardized physical models of the esophagus and stomach. The model size should be based on standard anatomical data and ensure millimeter-level accuracy.

[0089] Simulate the shooting environment: add an appropriate amount of liquid (such as water) into the physical model, and use a fixed-resolution capsule endoscope (for example, 480×480 pixels) to shoot the target object (of known size) from different angles and distances.

[0090] Record the shooting distance d, angle θ, liquid environmental conditions and other parameters to ensure the repeatability of the data.

[0091] Furthermore, let the projection size of the target object in the image be , the actual size is , the distance between the lens and the target is d, then the size conversion coefficient C is calculated as: ; Furthermore, the size conversion coefficient C is stored in a database for use in depth estimation and size calculation.

[0092] Therefore, when evaluating the lesion size, we convert the lesion size in the two-dimensional image into the actual physical size based on the disparity information of the lesion depth image and the lesion edge contour image, combined with the size conversion coefficient. The specific steps are as follows: Disparity information and lesion edge contour extraction: The lesion depth image provides a disparity value for each pixel, that is, the relative distance from each pixel to the camera of the capsule endoscope. The lesion edge contour image is edge detected by an edge detection algorithm (such as Canny edge detection) to extract the edge contour of the lesion. Each pixel of the edge contour represents the boundary of the lesion. Canny edge detection is a commonly used edge detection algorithm.

[0093] Disparity conversion based on depth map: In the data calibration stage, the size conversion coefficient C is determined by a simulated object of known size, so that each unit disparity value corresponds to a physical distance (unit: mm). Therefore, for each pixel point in the area formed by the edge contour of the lesion, the disparity value in the lesion depth image is converted to the actual physical depth value. The specific formula is as follows: d actual =C·d disparity ; Among them, d actual is the actual distance (mm or cm), d disparity is the disparity value of the corresponding pixel, and C is the size conversion coefficient.

[0094] Calculation of lesion size: Obtain edge pixel points of the lesion from the lesion edge contour, convert the disparity values ​​of these pixel points into actual physical coordinates, and further calculate the actual width and height of the lesion through these coordinates.

[0095] For example, assuming that the edge contour point set of the lesion is P = {(x1, y1, z1), (x2, y2, z2), ..., (xn, yn, zn)}, then: Width w: Calculate the maximum and minimum coordinate difference of the lesion edge contour in the x direction; Height h: Calculate the maximum and minimum coordinate difference of the lesion edge contour in the y direction; The lesion area can be calculated by the distribution and spacing of each pixel point. The specific formula is as follows: A lesion =N pixels ·a pixel ; Among them, N pixels is the number of pixels within the contour, a pixel The physical area of ​​each pixel (calculated using the size conversion factor).

[0096] And, three-dimensional size and volume estimation: In the absence of specific thickness information, the thickness of the lesion can be assumed based on experience, assuming that the thickness of the lesion is t.

[0097] If the lesion can be regarded as a regular shape (such as an ellipsoid), its volume V can be estimated using the following formula: ; Where w and h are the actual width and height of the lesion, and t is the assumed thickness of the lesion.

[0098] Thus, the lesion size can be obtained.

[0099] In some cases, the edge of the lesion may not be clear due to obstruction by surrounding tissues. The present application uses lesion depth images and edge contour images for comprehensive evaluation. Since the depth image provides more comprehensive three-dimensional information, it can more accurately capture the actual size and shape of the lesion and reduce possible errors in the contour image, so the lesion size can be accurately evaluated.

[0100] The lesion size assessment device provided in the present application is described below. The lesion size assessment device described below and the lesion size assessment method described above can be referenced to each other.

[0101] Furthermore, the present application also provides a lesion size assessment device.

[0102] The lesion size assessment device comprises: A recognition module, used for inputting the image to be evaluated into a lesion recognition model to obtain a lesion depth image and a lesion edge contour image output by the lesion recognition model; A determination module, configured to evaluate the size of a lesion based on the lesion depth image and the lesion edge contour image; Among them, the lesion recognition model is obtained by performing knowledge distillation on the teacher model based on unlabeled sample data and labeled sample data with pseudo labels; the teacher model is trained based on synthetic images and labels corresponding to the synthetic images; the labels corresponding to the synthetic images include lesion depth labeled images and lesion edge contour labeled images; the pseudo labels are generated by the teacher model based on the unlabeled sample data.

[0103] The lesion size assessment device of the present application obtains a teacher model by pre-training based on a synthetic image and a lesion depth annotated image and a lesion edge contour annotated image corresponding to the synthetic image, and after the teacher model generates pseudo labels based on unlabeled sample data, the teacher model performs knowledge distillation based on the unlabeled sample data and the annotated sample data with pseudo labels to obtain a lesion recognition model. Therefore, when lesion recognition is required, the image to be evaluated is input into the lesion recognition model to obtain the lesion depth image and the lesion edge contour image output by the lesion recognition model. Thus, the lesion size can be accurately assessed through the lesion depth image and the lesion edge contour image.

[0104] In one embodiment, the determining module is specifically used to: Determine the size conversion factor between the under-the-mirror size and the actual size; Determine the disparity value of each pixel in the lesion depth image; Determining the actual physical coordinates of each pixel point of the lesion edge contour in the lesion edge contour image based on each of the disparity values ​​and the size conversion coefficient; The lesion size is evaluated according to the actual physical coordinates of each pixel point in the lesion edge contour.

[0105] Figure 5 Schematic diagram of a lesion size assessment device provided in an embodiment of the present application. Figure 5 As shown, in some embodiments, the recognition module in the lesion size assessment device may include an input layer and a parallel processing layer. The determination module may include a result integration layer and an output layer, the result integration layer may include a size estimation module, and the output layer may include a report generation module. Among them, the input layer may include an image processing module, which can be used to input the two-dimensional image into the image processing module after acquiring image data through the capsule endoscope device, and perform basic processing such as noise removal and edge enhancement on the input image through the image processing module, and then input the pre-processed image into the depth estimation module and the lesion segmentation module in the lesion recognition model deployed in the parallel processing layer, respectively, thereby inferring the depth information of the lesion from the image through the depth estimation module and segmenting the lesion area from the pre-processed image through the lesion segmentation module.

[0106] Furthermore, the depth information and the edge contour of the lesion can be input into the size estimation module in the result integration layer, and the size estimation module can calculate the size of the lesion according to the segmentation contour and depth information of the lesion. Furthermore, the physical size of the lesion output by the size estimation module is input into the report generation module of the output layer, and the report generation module generates a diagnosis report according to the physical size of the lesion, in which the segmentation information and size of the lesion are summarized.

[0107] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method: input the image to be evaluated into the lesion recognition model to obtain the lesion depth image and the lesion edge contour image output by the lesion recognition model; Based on the lesion depth image and the lesion edge contour image, evaluating the lesion size; Among them, the lesion recognition model is obtained by performing knowledge distillation on the teacher model based on unlabeled sample data and labeled sample data with pseudo labels; the teacher model is trained based on synthetic images and labels corresponding to the synthetic images; the labels corresponding to the synthetic images include lesion depth labeled images and lesion edge contour labeled images; the pseudo labels are generated by the teacher model based on the unlabeled sample data.

[0108] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0109] In another aspect, the embodiments of the present application further provide a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method provided by the above embodiments is implemented, for example, including: inputting the image to be evaluated into a lesion recognition model, and obtaining a lesion depth image and a lesion edge contour image output by the lesion recognition model; Based on the lesion depth image and the lesion edge contour image, evaluating the lesion size; Among them, the lesion recognition model is obtained by performing knowledge distillation on the teacher model based on unlabeled sample data and labeled sample data with pseudo labels; the teacher model is trained based on synthetic images and labels corresponding to the synthetic images; the labels corresponding to the synthetic images include lesion depth labeled images and lesion edge contour labeled images; the pseudo labels are generated by the teacher model based on the unlabeled sample data.

[0110] In another aspect, the embodiments of the present application further provide a computer program product having a computer program stored thereon, and when the computer program is executed by a processor, the method provided by the above embodiments is implemented, for example, including: inputting the image to be evaluated into a lesion recognition model, and obtaining a lesion depth image and a lesion edge contour image output by the lesion recognition model; Based on the lesion depth image and the lesion edge contour image, evaluating the lesion size; Among them, the lesion recognition model is obtained by performing knowledge distillation on the teacher model based on unlabeled sample data and labeled sample data with pseudo labels; the teacher model is trained based on synthetic images and labels corresponding to the synthetic images; the labels corresponding to the synthetic images include lesion depth labeled images and lesion edge contour labeled images; the pseudo labels are generated by the teacher model based on the unlabeled sample data.

[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0113] Finally, it should be noted that the above implementation modes are only used to illustrate the present application, rather than to limit the present application. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that various combinations, modifications or equivalent substitutions of the technical solutions of the present application do not depart from the spirit and scope of the technical solutions of the present application.

Claims

1. A method for assessing lesion size, characterized in that: include: Inputting the image to be evaluated into a lesion recognition model to obtain a lesion depth image and a lesion edge contour image output by the lesion recognition model; Based on the lesion depth image and the lesion edge contour image, evaluating the lesion size; Among them, the lesion recognition model is obtained by performing knowledge distillation on the teacher model based on unlabeled sample data and labeled sample data with pseudo labels; the teacher model is trained based on synthetic images and labels corresponding to the synthetic images; the labels corresponding to the synthetic images include lesion depth labeled images and lesion edge contour labeled images; the pseudo labels are generated by the teacher model based on the unlabeled sample data.

2. The method for evaluating lesion size according to claim 1, characterized in that: The step of evaluating the lesion size based on the lesion depth image and the lesion edge contour image comprises: Determine the size conversion factor between the under-the-mirror size and the actual size; Determine the disparity value of each pixel in the lesion depth image; Determining the actual physical coordinates of each pixel point of the lesion edge contour in the lesion edge contour image based on each of the disparity values ​​and the size conversion coefficient; The lesion size is evaluated according to the actual physical coordinates of each pixel point in the lesion edge contour.

3. The method for evaluating lesion size according to claim 1, characterized in that: Before performing knowledge distillation based on unlabeled sample data and labeled sample data with pseudo labels, it also includes: Perform data augmentation on unlabeled sample data with pseudo labels.

4. The method for evaluating lesion size according to claim 1, characterized in that: The synthetic image is generated based on a preset three-dimensional physical model; the preset three-dimensional physical model is constructed based on a virtual image engine.

5. The method for evaluating lesion size according to claim 1, characterized in that: The loss function of the teacher model includes depth estimation loss, image segmentation loss and affine invariant loss.

6. The method for evaluating lesion size according to claim 1, characterized in that: The loss function of the lesion recognition model includes depth estimation loss and image segmentation loss.

7. A lesion size assessment device, characterized in that: include: A recognition module, used for inputting the image to be evaluated into a lesion recognition model to obtain a lesion depth image and a lesion edge contour image output by the lesion recognition model; A determination module, configured to evaluate the size of a lesion based on the lesion depth image and the lesion edge contour image; Among them, the lesion recognition model is obtained by performing knowledge distillation on the teacher model based on unlabeled sample data and labeled sample data with pseudo labels; the teacher model is trained based on synthetic images and labels corresponding to the synthetic images; the labels corresponding to the synthetic images include lesion depth labeled images and lesion edge contour labeled images; the pseudo labels are generated by the teacher model based on the unlabeled sample data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for assessing lesion size as described in any one of claims 1 to 6 is implemented.

9. A storage medium, the storage medium being a non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that: When the computer program is executed by a processor, the method for evaluating the size of a lesion as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the lesion size assessment method according to any one of claims 1 to 6 is implemented.

Images