An image cleanliness assessment method and related device based on multi-task learning

Abstract

The present application discloses a method for image cleanliness assessment based on multi-task learning and a related device. The method includes inputting an image to be assessed into a feature extraction module in the assessment network model, determining a number of feature maps of the image to be assessed through the feature extraction module, inputting the number of feature maps into a prediction module in the assessment network model, and determining a mask map and a cleanliness category corresponding to the image to be assessed through the prediction module. The present application extracts a number of feature maps through the feature extraction module, and then determines a mask map and a cleanliness category based on the number of feature maps through the prediction module. The prediction module utilizes instance features for determining the mask map to enhance the cleanliness feature representation of the image, focusing on the division of organs to which the image belongs and the cleanliness assessment of the image, thereby quickly determining a video image whose cleanliness meets predetermined requirements, thereby protecting the subsequent lesion segmentation network from interference from noisy scenes and reducing the computational complexity of lesion detection.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to an image cleanliness assessment method and related devices based on multi-task learning. Background Art

[0002] Digestive tract diseases are among the most common human illnesses, including ulcers, bleeding, polyps, intestinal inflammation, and intestinal tumors. Because the small intestine is located in the middle of the digestive tract and is much longer than other digestive organs (over 6 meters), comprehensive intestinal examination using pathological examinations and invasive laparoscopy is difficult. Conventionally, capsule endoscopy, as a non-invasive medical imaging technique, has become the mainstream method for digestive tract examinations. Before capsule endoscopy, patients must maintain a relatively clean intestine for easy imaging, requiring them to fast or take appropriate medications. However, intestinal cleansing is often ineffective, as food debris, intestinal fluid, and bile can obscure the intestinal mucosa. Furthermore, because capsule endoscopy lacks power and only captures images following intestinal peristalsis, loss of focus, reflections, motion blur, and other factors can prevent the intestinal lumen and mucosa from being captured, resulting in many invalid images. These invalid images lead to a significant amount of wasted effort during lesion segmentation, slowing down segmentation.

[0003] Therefore existing technology still needs to be improved and improved. Summary of the Invention

[0004] The technical problem to be solved by this application is to provide an image cleanliness assessment method and related devices based on multi-task learning in response to the shortcomings of the existing technology.

[0005] In order to solve the above technical problems, the first aspect of the embodiments of the present application provides an image cleanliness assessment method based on multi-task learning, wherein the method applies an assessment network model, and the method includes:

[0006] Inputting the image to be evaluated into a feature extraction module in the evaluation network model, and determining a plurality of feature maps of the image to be evaluated by the feature extraction module, wherein the image sizes of the plurality of feature maps are different from each other;

[0007] A plurality of feature maps are input into a prediction module in the evaluation network membrane, and a mask map and a cleanliness category corresponding to the image to be evaluated are determined by the prediction module, wherein the prediction module includes an interactive attention layer for performing cross-task interactive attention.

[0008] In one implementation, the feature extraction module includes several feature extraction units, and the several feature extraction units and the several feature extraction units are connected. Figure 1In a one-to-one correspondence, each feature map is the output item of its corresponding feature unit and the input item of the next feature unit after its corresponding feature unit.

[0009] In one implementation, the prediction module includes several cascaded prediction units, the prediction units and the feature Figure 1 One-to-one, determining the mask image and cleanliness category corresponding to the image to be evaluated by the prediction module specifically includes:

[0010] Determine segmentation features and classification features based on the feature map at the lowest layer and the prediction unit at the front of the plurality of feature maps;

[0011] Determine a fused feature map based on the segmentation feature and the feature map corresponding to the prediction unit located at the second position, and determine a segmentation feature and a classification feature based on the fused feature map, the classification feature and the prediction unit located at the second position;

[0012] This process is repeated until the last prediction unit is reached to obtain target segmentation features and target classification features.

[0013] A mask image corresponding to the image to be evaluated is determined based on the target segmentation features and the feature map at the top layer, and a cleanliness category corresponding to the image to be evaluated is determined based on the target classification features.

[0014] In one implementation, the prediction unit includes a cross-attention layer and an upsampling layer for performing cross-task attention, and the input of the upsampling layer is the segmentation feature output by the cross-attention layer; the input of the cross-attention layer in the front prediction unit is the feature map in the bottom layer, and the initial classification feature determined based on the feature map in the bottom layer; the input of the cross-attention layer in the remaining prediction units is the output of the upsampling layer in the previous prediction unit and the fused feature map of the feature map corresponding to the prediction unit, as well as the classification feature output by the cross-attention layer in the previous prediction unit.

[0015] In one implementation, the prediction module also includes a global average pooling layer, the input of the global average pooling layer is the feature map located at the bottom layer among several feature maps, and the output of the global average pooling layer is the initial classification feature determined based on the feature map located at the bottom layer.

[0016] In one implementation, the cross-attention layer includes a segmentation attention branch and a classification attention branch, wherein the segmentation attention branch and the classification attention branch both include a multi-head sub-attention module, a first fusion module, a feedforward network module, and a second fusion block, wherein the input item of the first fusion module in the segmentation attention branch is the output item of the multi-head sub-attention module in the segmentation attention branch and the V value and K value in the classification attention branch; the input item of the first fusion module in the classification attention branch is the output item of the multi-head sub-attention module in the classification attention branch and the V value and K value in the segmentation attention branch.

[0017] In one implementation, the K value, V value, and Q value of the multi-head sub-attention module in the segmentation attention branch are all determined based on the fusion feature map of the output item of the upsampling layer in the previous prediction unit and the feature map corresponding to the prediction unit, and the K value, V value, and Q value of the multi-head sub-attention module in the classification attention branch are all determined based on the classification features output by the cross-attention layer in the previous prediction unit.

[0018] A second aspect of an embodiment of the present application provides an image cleanliness assessment system based on multi-task learning, the system comprising:

[0019] a feature extraction module, configured to determine a plurality of feature maps of the image to be evaluated, wherein the image sizes of the plurality of feature maps are different from each other;

[0020] The prediction module is used to determine the mask image and cleanliness category corresponding to the image to be evaluated.

[0021] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any of the above-described methods for image cleanliness assessment based on multi-task learning.

[0022] A fourth aspect of an embodiment of the present application provides a terminal device, comprising: a processor, a memory, and a communication bus; the memory stores a computer-readable program executable by the processor;

[0023] The communication bus realizes the connection and communication between the processor and the memory;

[0024] When the processor executes the computer-readable program, the steps of any of the above-described methods for evaluating image cleanliness based on multi-task learning are implemented.

[0025] Beneficial effects: Compared with the prior art, the present application provides an image cleanliness assessment method and related devices based on multi-task learning, the method comprising inputting the image to be assessed into the feature extraction module in the assessment network model, determining a number of feature maps of the image to be assessed through the feature extraction module, inputting the several feature maps into the prediction module in the assessment network model, and determining the mask map and cleanliness category corresponding to the image to be assessed through the prediction module. The present application extracts a number of feature maps through the feature extraction module, and then determines the mask map and cleanliness category based on the several feature maps through the prediction module. The prediction module utilizes the instance features used to determine the mask map to enhance the cleanliness feature representation of the image, focusing on the division of the organs to which the image belongs and the cleanliness assessment of the image, thereby being able to quickly determine the video image whose cleanliness meets the predetermined requirements, thereby protecting the subsequent lesion segmentation network from interference from noisy scenes, and reducing the computational complexity of lesion detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without inventive work.

[0027] Figure 1 Flowchart of the image cleanliness assessment method based on multi-task learning provided in this application.

[0028] Figure 2 Principle flow chart of the image cleanliness assessment method based on multi-task learning provided in this application.

[0029] Figure 3 Schematic diagram of the cross-attention module layer.

[0030] Figure 4 This is a schematic diagram of the structure of the image cleanliness assessment system based on multi-task learning provided in this application.

[0031] Figure 5 This is a schematic diagram of the structure of the terminal device provided in this application. DETAILED DESCRIPTION

[0032] This application provides a method and related apparatus for image cleanliness assessment based on multi-task learning. To clarify the purpose, technical solutions, and effects of this application, the application is further described below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are intended only to illustrate this application and are not intended to limit it.

[0033] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0034] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0035] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not imply the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0036] After research, the inventors found that digestive tract diseases are one of the most common human diseases, with common ones including ulcers, bleeding, polyps, intestinal inflammation, and intestinal tumors. Among them, because the small intestine is located in the middle of the digestive tract and is much longer than other digestive tract organs (longer than 6 meters), it is difficult to use pathological examination and invasive laparoscopes to conduct a comprehensive examination of the intestine. Under normal circumstances, capsule endoscopy, as a non-invasive medical imaging examination, has become the mainstream digestive tract detection method. Before a patient undergoes a capsule endoscopy, the intestine needs to be kept relatively clean and easy to image, which requires the patient to fast or take appropriate medications. However, the effect of intestinal cleaning is often not ideal. Food residues, intestinal fluid, and bile will obscure the intestinal mucosa. At the same time, because the capsule endoscope itself has no power and only captures images with intestinal peristalsis, there may be defocus, reflections, motion blur, etc., which may result in the inability to capture the intestinal tissue and mucosa, resulting in many invalid images in the acquired images. These invalid images cause a lot of invalid work to be stored during the lesion segmentation process, affecting the speed of lesion segmentation.

[0037] When screening for invalid images from capsule endoscopy examinations, two factors must be considered: First, due to individual patient differences and varying degrees of preoperative bowel preparation, bile, air bubbles, food debris, and turbid intestinal fluid can reduce the visualization of endoscopic images and affect the diagnostic capabilities of capsule endoscopy. Second, due to the inherent nature of endoscopy, temporal factors can degrade image quality, such as loss of focus and motion blur.

[0038] Based on this, in an embodiment of the present application, the image to be evaluated is input into the feature extraction module in the evaluation network model, and several feature maps of the image to be evaluated are determined by the feature extraction module. The several feature maps are input into the prediction module in the evaluation network model, and the mask map and cleanliness category corresponding to the image to be evaluated are determined by the prediction module. The evaluation network model adopted in this application is an image cleanliness evaluation network based on multi-task learning. Several feature maps are extracted by the feature extraction module, and then the mask map and cleanliness category are determined based on the several feature maps by the prediction module. The prediction module uses the instance features used to determine the mask map to enhance the cleanliness feature performance of the image, focusing on the division of the organs to which the image belongs and the cleanliness evaluation of the image, so that the video image whose cleanliness meets the predetermined requirements can be quickly determined, thereby protecting the subsequent lesion segmentation network from interference from noisy scenes, and reducing the computational complexity of lesion detection. In addition, the predicted pixel-level mask map is more similar to the doctor's process of observing and examining the image, and provides the doctor with a more interpretable reference.

[0039] The application content will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0040] This embodiment provides an image cleanliness assessment method based on multi-task learning. Figure 1 and Figure 2 As shown, the method includes:

[0041] S10: Input the image to be evaluated into a feature extraction module in the evaluation network model, and determine several feature maps of the image to be evaluated through the feature extraction module.

[0042] Specifically, the evaluation network model is pre-trained. The input of the evaluation network model is the image to be evaluated, and the output is the mask image corresponding to the image to be evaluated and the cleanliness category. The evaluation network model includes a feature extraction module and a prediction module. The feature extraction module is used to determine a number of feature images. The prediction module is a multi-task prediction module that can determine the mask image and cleanliness category. The image to be evaluated is an endoscopic image acquired through capsule endoscopy. The endoscopic image can be an endoscopic image of the small intestine or an image of the inner diameter of another part. In a typical implementation, the image to be evaluated is an endoscopic image of the small intestine.

[0043] Since the evaluation network model includes classification tasks and segmentation tasks, the overall loss function of the evaluation network model consists of two parts: the pixel-level high-level semantic segmentation loss function of the tissue area and the loss function of the segmentation loss function. and image cleanliness assessment classification loss function Therefore, the calculation formula of the loss function is as follows:

[0044]

[0045] Among them, α is a hyperparameter used to adjust the loss ratio of segmentation task and classification task, and the pixel-level high-level semantic segmentation loss function is The calculation formula is as follows:

[0046]

[0047] Among them, M represents the number of categories, N represents the total number of pixels, and y (i,j) There are two values 0 and 1. When the predicted category of the j-th pixel is the same as the labeled value, it takes 1, otherwise it takes 0. (i,j) It represents the probability that the predicted sample belongs to category i.

[0048] Image cleanliness assessment classification loss function The calculation formula is as follows:

[0049]

[0050] Among them, M represents the total number of categories, y i and p i denote the category label and the predicted probability of belonging to category i, respectively.

[0051] In one implementation, the feature extraction module includes several feature extraction units, and the several feature extraction units and the several feature extraction units are connected. Figure 1 One-to-one correspondence, each feature map is the output item of its corresponding feature unit and the input item of the next feature unit of its corresponding feature unit. Among them, several feature maps are multi-scale feature maps determined by feature extraction modules, that is, the image sizes of each feature map are different. Among them, the feature extraction module can be ResNet50, ConvNeXt or SwinTransformer, etc. The process of determining several feature maps can be: given an image to be evaluated Through the feature extraction module F, multi-scale feature extraction is performed to obtain In one implementation, each feature extraction unit performs downsampling through a dilated convolutional layer.

[0052] S20 , inputting a plurality of feature maps into a prediction module in the evaluation network membrane, and determining a mask map and a cleanliness category corresponding to the image to be evaluated through the prediction module.

[0053] Specifically, the mask image is a pixel-level mask image of the tissue and other intestinal contents in the image to be evaluated, and the cleanliness category is the cleanliness category of the image to be evaluated, wherein the cleanliness category may include invisible, blurry, and clear, etc. The prediction module is a multi-task prediction module, and the multi-task refers to the segmentation task and the classification task, wherein the segmentation task is used to determine the mask image, and the classification task is used to determine the cleanliness category. Therefore, the prediction module may include two prediction branches, one prediction branch is a segmentation branch, and the segmentation branch is used to organize semantic segmentation to predict the pixel-level mask image of the tissue and other intestinal contents; the other is a classification branch, which is used to measure the image clarity to obtain the cleanliness category. In addition, the two prediction branches in the prediction module are parallel. After a number of feature maps are input into the prediction module, the two prediction branches of the prediction module are parallel to synchronously obtain the mask image and the cleanliness category.

[0054] In one implementation, the prediction module includes several cascaded prediction units, the prediction units and the feature Figure 1 One-to-one, determining the mask image and cleanliness category corresponding to the image to be evaluated by the prediction module specifically includes:

[0055] S21, determining segmentation features and classification features based on the feature map at the bottom layer and the prediction unit at the front of the plurality of feature maps;

[0056] S22, determining a fused feature map based on the segmentation feature and the feature map corresponding to the prediction unit located at the second position, and determining a segmentation feature and a classification feature based on the fused feature map, the classification feature, and the prediction unit located at the second position;

[0057] S23, and so on until the last prediction unit is located to obtain the target segmentation feature and the target classification feature;

[0058] S24 , determining a mask image corresponding to the image to be evaluated based on the target segmentation features and the feature map at the top layer, and determining a cleanliness category corresponding to the image to be evaluated based on the target classification features.

[0059] Specifically, in step S21, the prediction unit includes a cross-attention layer and an upsampling layer for performing cross-task attention, wherein the segmentation feature is the output item of the upsampling layer, the classification feature is the output item of the cross-attention layer, the cross-attention layer is connected to the upsampling layer, and the input item of the upsampling layer is the segmentation feature output by the cross-attention layer. It can be understood that the cross-attention layer is the classification branch in the prediction unit, and the cross-attention layer and the upsampling layer are the segmentation branch in the prediction unit, that is, the cross-attention layer in each prediction unit is connected in sequence to form a classification branch, and the cross-attention layer and the upsampling layer in each prediction unit are connected in sequence to form a segmentation branch.

[0060] like Figure 2 As shown, the input items of the prediction unit include input classification features for classification tasks and input segmentation features for segmentation tasks, and the output items include classification features and segmentation features; wherein, the input segmentation features of the prediction unit located at the front are the feature maps of the bottom layer, the input classification features are feature vectors determined based on the feature maps of the bottom layer, the classification features output by the prediction unit located at the front are the classification features output by the cross-attention layer, and the segmentation features output by the prediction unit located at the front are the segmentation features output by the upsampling layer. In one implementation, the prediction module also includes a global average pooling layer, which is used to determine the classification features of the prediction unit located at the front, wherein the input item of the global average pooling layer is the feature map located at the bottom layer among several feature maps, and the output item of the global average pooling layer is the initial classification feature determined based on the feature map located at the bottom layer, and the initial classification feature is the input classification feature of the prediction unit located at the front.

[0061] In step S22, as Figure 2 As shown, the input items of the cross-attention layer in each prediction unit except the prediction unit located at the front is the output item of the upsampling layer in the previous prediction unit and the fused feature map of the feature map corresponding to the prediction unit, as well as the classification feature output by the cross-attention layer in the previous prediction unit. It can be understood that for each prediction unit except the prediction unit located at the front, the input classification feature of the prediction unit is the classification feature output by the cross-attention layer in the previous prediction unit, and the input segmentation feature of the prediction unit is the fused feature map of the output item of the upsampling layer in the previous prediction unit and the feature map corresponding to the prediction unit. In addition, since the cross-attention layer is used to fuse the classification feature and the segmentation feature, for each cross-attention layer, the number of channels of the classification feature input to the cross-attention layer is the same as the number of channels of the segmentation feature, wherein the segmentation feature will be expanded into a one-dimensional representation before being input into the cross-attention layer, so that both the segmentation feature and the classification feature are one-dimensional representations.

[0062] For the mask image, it is a U-Net structure composed of a feature extraction module and a segmentation branch in a prediction module. The feature extraction module is an encoder of the U-Net structure, and the segmentation branch formed by the cross attention layer and the upsampling layer is a decoder of the U-Net structure. The output item of each decoding layer in the decoder is obtained by adding the residual connection of the corresponding feature extraction unit in the corresponding feature extraction module and the high-level features after the cross attention layer and the upsampling layer. The output item based on each decoding layer can be expressed as:

[0063] p i+1 =f i+1 +UpSample(p' i )

[0064] Among them, p i+1 Represents the output item of the i+1th layer decoding layer. As i increases, p i+1 The resolution gradually increases and the number of channels gradually decreases; p' i Represents the segmentation features output by the i-th cross attention layer, and UpSample() represents the upsampling operation.

[0065] For the cleanliness category, the evaluation detection model is determined by a classification branch connected by multiple cross-attention layers. This allows the cross-attention layer to fuse the classification features and segmentation features, that is, to fuse the classification features of the classification branch with the high-level semantic information used as segmentation features in the segmentation branch. In this way, the instance features of lesion detection can be used to enhance the cleanliness feature representation of the image, thereby improving the accuracy of the cleanliness category. The classification features output by the cross-attention layer can be expressed as:

[0066] c' i =ICA(c i )

[0067] Among them, ICA represents the cross attention layer, c i represents the classification features of the input i-th cross attention layer, c' i It represents the classification feature output after the classification feature and the segmentation feature are fused through the i-th cross attention layer.

[0068] In step S24, the segmentation feature p output by the last decoding layer is N is the fusion feature map of the target segmentation feature and the feature map at the top, and N is the number of decoding layers. The segmentation feature p output by the last decoding layer N Finally, a 3×3 convolution is performed to transform the number of channels into the number of categories, and the Softmax operation is performed to obtain the final pixel-level mask map, where the pixel-level mask map is expressed as:

[0069] m=Softmax(Conv(p4))

[0070] Among them, m represents the predicted mask map, M represents the number of categories.

[0071] After the last cross-attention layer outputs the target classification features, the target classification features are converted into a cleanliness category probability sequence through the activation function layer, and the cleanliness category corresponding to the image to be evaluated is determined based on the cleanliness category probability sequence.

[0072] In one implementation, in order to better integrate the segmentation features and classification features of each layer and alleviate the impact of the differences between the two-task learning tasks on the model performance, the cross-attention layer adopts a cross-task cross-attention structure to perform cross-task attention learning. Accordingly, the calculation process of the cross-attention layer can be expressed as:

[0073] p' i ,c' i =ICA(p i ,c i )

[0074] Among them, p i ,c i Represents the segmentation features and classification features of the input i-th layer cross attention layer; p' i ,c' i It means that the i-th cross attention layer fuses the input segmentation features and classification features to obtain segmentation features and classification features.

[0075] like Figure 3 As shown, the cross-attention layer includes a parallel segmentation attention branch and a classification attention branch, wherein the segmentation attention branch is used to determine the segmentation features, the classification attention branch is used to determine the classification task, and the segmentation attention branch interactively learns the input segmentation features and the input classification features and outputs the segmentation features, and the classification attention branch interactively learns the input classification features and the input segmentation features and outputs the classification features.

[0076] Furthermore, both the segmentation attention branch and the classification attention branch include a multi-head sub-attention module, a first fusion module, a feedforward network module and a second fusion block, wherein the first fusion block and the second fusion block both include an adder and a regularization layer. The input items of the first fusion module in the segmentation attention branch are the output items of the multi-head sub-attention module in the segmentation attention branch and the V value and K value in the classification attention branch; the input items of the first fusion module in the classification attention branch are the output items of the multi-head sub-attention module in the classification attention branch and the V value and K value in the segmentation attention branch. In addition, after the candidate segmentation features are determined by the segmentation attention branch, the candidate segmentation features are enhanced by a feedforward neural network, and then the enhanced candidate segmentation features are fused with the input segmentation features to obtain the output segmentation features. Similarly, after the candidate classification features are determined by the classification attention branch, the candidate classification features are enhanced by a feedforward neural network, and then the enhanced candidate classification features are fused with the input classification features to obtain the output classification features. Among them,

[0077] To further illustrate the cross attention layer, here we take the classification feature F c The calculation is as follows:

[0078] First, F c After the linear transformation of the fully connected layer, we get and

[0079]

[0080] Among them, W q ,W k ,W v There are three different weight matrices.

[0081] Secondly, the cross attention map of category and pixel level is calculated by the classification attention branch and its intersection with V Fc The normalized product of c .

[0082] Again, the F' obtained by the feedforward network module is c Perform spatial transformation to obtain the feature F' that enhances the representation ability of classification features c ', where F' c ' is expressed as:

[0083] F” c =LayerNorm(FFN(F c ))

[0084] Among them, FFN represents the feedforward network module.

[0085] Finally, F”c As the residual with the original feature F c Get the sum result and use it as the output classification feature F c .

[0086] To sum up, the process of expressing the attention features output by the cross-attention layer can be expressed as:

[0087]

[0088] Among them, F c and F s They represent the classification features and segmentation features of the cross attention layer respectively, and γ and β represent the fusion coefficients.

[0089] In summary, this embodiment provides an image cleanliness assessment method based on multi-task learning, the method comprising inputting the image to be evaluated into the feature extraction module in the evaluation network model, determining a number of feature maps of the image to be evaluated through the feature extraction module, inputting the several feature maps into the prediction module in the evaluation network model, and determining the mask map and cleanliness category corresponding to the image to be evaluated through the prediction module. The present application extracts a number of feature maps through the feature extraction module, and then determines the mask map and cleanliness category based on the several feature maps through the prediction module. The prediction module utilizes the instance features used to determine the mask map to enhance the cleanliness feature representation of the image, focusing on the division of the organs to which the image belongs and the cleanliness assessment of the image, so that the video image whose cleanliness meets the predetermined requirements can be quickly determined, thereby protecting the subsequent lesion segmentation network from interference from noisy scenes, and reducing the computational complexity of lesion detection.

[0090] Based on the above-mentioned image cleanliness assessment method based on multi-task learning, this embodiment provides an image cleanliness assessment system based on multi-task learning, such as Figure 4 As shown, the system includes:

[0091] A feature extraction module 100 is configured to determine a plurality of feature maps of the image to be evaluated, wherein the image sizes of the plurality of feature maps are different from each other;

[0092] The prediction module 200 is used to determine the mask image and cleanliness category corresponding to the image to be evaluated.

[0093] Based on the above-mentioned image cleanliness assessment method based on multi-task learning, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the image cleanliness assessment method based on multi-task learning as described in the above-mentioned embodiment.

[0094] Based on the above-mentioned image cleanliness assessment method based on multi-task learning, this application also provides a terminal device, such as Figure 5 As shown, it includes at least one processor 20; a display screen 21; and a memory 22. It may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via bus 24. The display screen 21 is configured to display a preset user guidance interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0095] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0096] The memory 22, as a computer-readable storage medium, can be configured to store software programs or computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the software programs, instructions, or modules stored in the memory 22 to perform functional applications and data processing, thereby implementing the methods in the above embodiments.

[0097] The memory 22 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal device. In addition, the memory 22 may include high-speed random access memory and non-volatile memory. For example, various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.

[0098] In addition, the specific process of loading and executing the multiple instructions in the storage medium and the processor in the terminal device has been described in detail in the above method and will not be described here one by one.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for image cleanliness assessment based on multi-task learning, characterized in that: The method uses an evaluation network model, and the method includes: Inputting the image to be evaluated into a feature extraction module in the evaluation network model, and determining a plurality of feature maps of the image to be evaluated by the feature extraction module, wherein the image sizes of the plurality of feature maps are different from each other; Inputting a plurality of feature maps into a prediction module in the evaluation network model, and determining a mask map and a cleanliness category corresponding to the image to be evaluated by the prediction module; Among them, the prediction module includes several cascaded prediction units, and the several prediction units correspond one to one with several feature maps. The prediction units include a cross-attention layer and an upsampling layer for performing cross-task attention. The input of the upsampling layer is the segmentation feature output by the cross-attention layer; the input of the cross-attention layer in the front prediction unit is the feature map in the bottom layer, and the initial classification feature determined based on the feature map in the bottom layer; the input of the cross-attention layer in the remaining prediction units is the fusion feature map of the output of the upsampling layer in the previous prediction unit and the feature map corresponding to the prediction unit, as well as the classification feature output by the cross-attention layer in the previous prediction unit; the cross-attention layer The cross attention layer includes a segmentation attention branch and a classification attention branch, wherein the segmentation attention branch and the classification attention branch both include a multi-head sub-attention module, a first fusion module, a feedforward network module, and a second fusion block, wherein the input item of the first fusion module in the segmentation attention branch is the output item of the multi-head sub-attention module in the segmentation attention branch and the V value and K value in the classification attention branch; the input item of the first fusion module in the classification attention branch is the output item of the multi-head sub-attention module in the classification attention branch and the V value and K value in the segmentation attention branch; the determining of the mask image and the cleanliness category corresponding to the image to be evaluated by the prediction module specifically includes: Determine segmentation features and classification features based on the feature map at the lowest layer and the prediction unit at the front of the plurality of feature maps; Determine a fused feature map based on the segmentation feature and the feature map corresponding to the prediction unit located at the second position, and determine a segmentation feature and a classification feature based on the fused feature map, the classification feature and the prediction unit located at the second position; This process is repeated until the last prediction unit is reached to obtain target segmentation features and target classification features. A mask image corresponding to the image to be evaluated is determined based on the target segmentation features and the feature map at the top layer, and a cleanliness category corresponding to the image to be evaluated is determined based on the target classification features.

2. The image cleanliness assessment method based on multi-task learning according to claim 1, characterized in that: The feature extraction module includes several feature extraction units, and the several feature extraction units correspond one-to-one to several feature maps. Each feature map is the output item of the feature unit corresponding to it, and is the input item of the feature unit following the feature unit corresponding to it.

3. The image cleanliness assessment method based on multi-task learning according to claim 1, characterized in that: The prediction module further includes a global average pooling layer, the input item of the global average pooling layer is the feature map located at the lowest layer among the several feature maps, and the output item of the global average pooling layer is the initial classification feature determined based on the feature map located at the lowest layer.

4. The image cleanliness assessment method based on multi-task learning according to claim 1, characterized in that: The K value, V value and Q value of the multi-head sub-attention module in the segmentation attention branch are all determined based on the fusion feature map of the output item of the upsampling layer in the previous prediction unit and the feature map corresponding to the prediction unit. The K value, V value and Q value of the multi-head sub-attention module in the classification attention branch are all determined based on the classification features output by the cross-attention layer in the previous prediction unit.

5. An image cleanliness assessment system based on multi-task learning, characterized in that: The system is used to perform the image cleanliness assessment method based on multi-task learning as claimed in claim 1, and the system includes a feature extraction module and a prediction module.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the image cleanliness assessment method based on multi-task learning according to any one of claims 1 to 4.

7. A terminal device, characterized in that: include: processor, memory, and communication bus; The memory stores a computer-readable program executable by the processor; The communication bus realizes the connection and communication between the processor and the memory; When the processor executes the computer-readable program, the steps of the image cleanliness assessment method based on multi-task learning are implemented.

Images