Lesion area positioning method for image quality evaluation combined with ultrasonic endoscope

Abstract

The invention relates to the technical field of image processing, in particular to a focus area positioning method for image quality evaluation combined with an ultrasonic endoscope, and the method comprises the steps: obtaining a to-be-positioned ultrasonic image, carrying out the preprocessing of the to-be-positioned ultrasonic image, and obtaining a normalized ultrasonic image; inputting the normalized ultrasonic image into a backbone network encoder to obtain an ultrasonic image feature vector; respectively inputting the ultrasonic image feature vector into an image quality evaluation branch and a decoder to respectively obtain a quality evaluation result and focus area information of the ultrasonic image to be positioned; and outputting the focus area information based on the quality evaluation result. The method is based on deep learning, ultrasonic endoscope image quality evaluation and suspected cancerization area positioning are achieved at the same time, doctors can be assisted in ultrasonic endoscope examination, and efficiency and accuracy are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for locating a lesion region combined with image quality assessment of an ultrasonic endoscope. Background Art

[0002] Endoscopic Ultrasonography (EUS) is a digestive tract examination technology that combines endoscopy and ultrasound. Endoscopic ultrasound is to install an ultrasound probe at the front end of the endoscope. While performing endoscopic examination, the ultrasound probe can be used to perform ultrasound scanning of the digestive tract wall and adjacent organs. In this way, the histological characteristics of each layer of the digestive tract wall and the ultrasound images of the surrounding organs can be obtained, providing richer information for the diagnosis of the disease.

[0003] However, ultrasound endoscopy also has some disadvantages. The first is that the technical requirements are high, which is mainly reflected in two aspects: Difficult operation: The operation of ultrasound endoscopy requires the mastery of both endoscopic technology and ultrasound technology, which requires very high professional skills of doctors. Doctors need to undergo special training and long-term practice to master the operation skills of ultrasound endoscopy, which to a certain extent limits its wide application. Complex image interpretation: The images obtained by ultrasound endoscopy are relatively complex, requiring doctors to have rich clinical experience and professional knowledge to accurately interpret them. Different lesions may appear similar on ultrasound images, which can easily lead to misdiagnosis or missed diagnosis. Summary of the invention

[0004] The main purpose of this application is to provide a method for locating lesion areas combined with image quality evaluation of ultrasound endoscopy, which prompts doctors whether their operations are reasonable through image quality evaluation, and assists doctors in interpreting areas suspected of cancer, effectively improving the efficiency and accuracy of ultrasound endoscopy inspections.

[0005] To achieve the above objectives, this application provides the following technical solutions: According to a first aspect of the present invention, the present invention claims a method for locating a lesion region combined with an ultrasound endoscope image quality assessment, comprising: Acquiring an ultrasonic image to be located, and preprocessing the ultrasonic image to be located to obtain a normalized ultrasonic image; Inputting the normalized ultrasound image into a backbone network encoder to obtain an ultrasound image feature vector; Inputting the ultrasonic image feature vector into the image quality assessment branch and the decoder respectively, and obtaining the quality assessment result and the lesion area information of the ultrasonic image to be located respectively; The lesion region information is output based on the quality assessment result.

[0006] Furthermore, the step of acquiring the ultrasonic image to be located, preprocessing the ultrasonic image to be located, and obtaining a normalized ultrasonic image further includes: Using an image processing library to read the ultrasound image to be located, and representing the ultrasound image to be located as an ultrasound image matrix or an ultrasound image tensor; Performing image scaling and normalization processing on the ultrasound image matrix or ultrasound image tensor; During the normalization process, the maximum and minimum pixel values ​​of the ultrasonic image matrix or the ultrasonic image tensor are obtained; The ultrasound image matrix or ultrasound image tensor is processed using a normalization formula to obtain a normalized ultrasound image.

[0007] Furthermore, the step of inputting the normalized ultrasound image into a backbone network encoder to obtain an ultrasound image feature vector further includes: The backbone network encoder includes an initial module, a first stage layer, a first image block embedding layer, a second stage layer, a second image block embedding layer, a third stage layer, a third image block embedding layer, and a fourth stage layer in sequence; The initial module includes an initial convolution layer and an initial pooling layer; The first stage layer performs a first training and a first inference to obtain a first feature vector; Inputting the first feature vector into a first image block embedding layer to obtain a first embedded feature vector; Inputting the first embedded feature vector into the second stage layer to perform second training and second inference to obtain a second feature vector; Inputting the second feature vector into a second image block embedding layer to obtain a second embedded feature vector; Inputting the second embedded feature vector into the third stage layer to perform third training and third inference to obtain a third feature vector; Inputting the third feature vector into a third image block embedding layer to obtain a third embedded feature vector; The third embedded feature vector is input into the fourth stage layer to perform fourth training and fourth reasoning to obtain a fourth feature vector as an ultrasound image feature vector.

[0008] Furthermore, the method further comprises: When the first stage layer performs the first training, the feature vector is input to the first BN layer and the first depth-separable convolutional layer, and then input to the first convolutional feedforward network to obtain the first training feature vector; When the first stage layer performs the first inference, the input feature vector passes through the second depth-separable convolutional layer and the second convolutional feed-forward network in sequence to obtain the first inference feature vector; Inputting the first feature vector into a first image block embedding layer to obtain a first embedded feature vector further includes: The first training feature vector is input into the first image block embedding training layer, and is superimposed with the result of the fourth depthwise separable convolutional layer with a preset step size through the third depthwise separable convolutional layer and the second BN layer, and then activated. The obtained results are respectively sent to two branches, including the first convolutional layer, the third BN layer branch, and the fourth BN layer, respectively. The results of the two branches are superimposed and activated, and the first embedding training feature vector is output; The first inference feature vector is input into the first image block embedding inference layer, sequentially passes through the fifth depth-separable convolutional layer and is activated, and then input into the second convolutional layer and is activated to obtain a second embedding inference feature vector.

[0009] Furthermore, the method further comprises: When the second stage layer performs the second training, the first embedding training feature vector is input to the fifth BN layer and the sixth depth-separable convolution layer, and then input to the third convolution feed-forward network to obtain the second training feature vector; When the second stage layer performs the second reasoning, the second embedding reasoning feature vector is input and sequentially passed through the seventh depth-separable convolutional layer and the fourth convolutional feed-forward network to obtain the second reasoning feature vector; Inputting the second feature vector into a second image block embedding layer to obtain a second embedded feature vector, further comprising: The second training feature vector is input into the second image block embedding training layer, passed through the eighth depth-separable convolution layer and the sixth BN layer, and superimposed with the result of the ninth depth-separable convolution layer with a preset step size, and then activated. The obtained results are respectively sent to two branches, including the third convolution layer, the seventh BN layer branch and the eighth BN layer, respectively. The results of the two branches are superimposed and activated, and the second embedding training feature vector is output; The second inference feature vector is input into the second image block embedding inference layer, sequentially passes through the tenth depth-separable convolutional layer and is activated, and then input into the fourth convolutional layer and is activated to obtain the second embedding inference feature vector.

[0010] Furthermore, the method further comprises: When the third stage layer performs the third training, the second embedding training feature vector is input to the ninth BN layer and the eleventh depth-separable convolution layer, and then input to the fifth convolution feed-forward network to obtain a third training feature vector; When the third stage layer performs the third reasoning, the third embedding reasoning feature vector is input and sequentially passed through the twelfth depth-separable convolutional layer and the sixth convolutional feed-forward network to obtain a third reasoning feature vector; Inputting the third feature vector into a third image block embedding layer to obtain a third embedded feature vector, further comprising: The third training feature vector is input into the third image block embedding training layer, passed through the thirteenth depthwise separable convolution layer and the tenth BN layer, and superimposed with the result of the fourteenth depthwise separable convolution layer with a preset step size, and then activated, and the obtained results are respectively sent to two branches, including the fifth convolution layer, the eleventh BN layer branch and the twelfth BN layer, respectively, and the results of the two branches are superimposed and activated, and the third embedding training feature vector is output; The third inference feature vector is input into the third image block embedding inference layer, sequentially passes through the fifteenth depth-separable convolutional layer and is activated, and then input into the sixth convolutional layer and is activated to obtain the third embedding inference feature vector.

[0011] Further, the third embedded feature vector is input into the fourth stage layer to perform fourth training and fourth reasoning to obtain a fourth feature vector as the ultrasound image feature vector, and further includes: Input the third embedded training feature vector into the first CPE module, pass through the thirteenth BN layer in sequence and then activate it, and then send it into the seventh convolution feedforward network to output a fourth training feature vector; The third embedding reasoning feature vector is input into the second CPE module, is activated after passing through the fourteenth BN layer in sequence, and is then sent into the eighth convolutional feedforward network to output a fourth reasoning feature vector; The fourth training feature vector and the fourth inference feature vector are fused to obtain a fourth feature vector as an ultrasound image feature vector.

[0012] Furthermore, the ultrasonic image feature vectors are respectively input into the image quality assessment branch to obtain the quality assessment result of the ultrasonic image to be located, and further includes: Sending the ultrasonic image feature vector to an image quality assessment branch network to obtain a sixteenth depth-separable convolutional layer and a global average pooling vector; Performing label classification on the ultrasound image to be located according to the global average pooling vector to obtain a classification result; Performing sigmoid and binarization operations on the classification results to obtain the result of the final image quality assessment branch; The label classification at least includes probe placement rationality, structure clarity, and background clarity.

[0013] Furthermore, the ultrasonic image feature vectors are respectively input into a decoder to obtain the lesion area information of the ultrasonic image to be located, and further comprising: The decoder comprises a first decoding layer, a second decoding layer, a third decoding layer, a fourth decoding layer and a fifth decoding layer; The first decoding layer inputs the ultrasonic image feature vector, which passes through the first decoding convolution layer, the first decoding BN layer, the first ReLU layer and the first decoding deconvolution layer in sequence to obtain a first decoding feature vector; Inputting the first decoding feature vector into the second decoding layer, and sequentially passing through a second decoding convolutional layer, a second decoding BN layer, a second ReLU layer, and a second decoding deconvolutional layer to obtain a second decoding feature vector; Inputting the second decoding feature vector into the third decoding layer, and sequentially passing through a third decoding convolutional layer, a third decoding BN layer, a third ReLU layer, and a third decoding deconvolutional layer to obtain a third decoding feature vector; Inputting the third decoding feature vector into the fourth decoding layer, and sequentially passing through a fourth decoding convolutional layer, a fourth decoding BN layer, a fourth ReLU layer, and a fourth decoding deconvolutional layer to obtain a fourth decoding feature vector; The fourth decoding feature vector is input into the fifth decoding layer, and sequentially passes through a fifth decoding convolutional layer, a fifth decoding BN layer, a fifth ReLU layer and a fifth decoding deconvolutional layer to obtain a fifth decoding feature vector.

[0014] Furthermore, the outputting of the lesion area information based on the quality assessment result further includes: When the sum of the scores of the three label categories of the quality assessment result is less than a preset score, the lesion area information is not displayed.

[0015] The present application relates to the field of image processing technology, and in particular to a method for locating a lesion region in combination with an ultrasonic endoscope image quality assessment, wherein an ultrasonic image to be located is acquired, the ultrasonic image to be located is preprocessed to obtain a normalized ultrasonic image; the normalized ultrasonic image is input into a backbone network encoder to obtain an ultrasonic image feature vector; the ultrasonic image feature vector is input into an image quality assessment branch and a decoder respectively to obtain a quality assessment result and lesion region information of the ultrasonic image to be located; and the lesion region information is output based on the quality assessment result. The present invention is based on deep learning, and simultaneously realizes ultrasonic endoscope image quality assessment and suspected cancer region location, which can assist doctors in ultrasonic endoscopy examinations and improve efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A workflow diagram of a method for locating a lesion area combined with image quality assessment of an ultrasonic endoscope as claimed in an embodiment of the present application; Figure 2 A flowchart of a backbone network encoder for a method for localizing a lesion region in combination with an ultrasound endoscope for image quality assessment as claimed in an embodiment of the present application; Figure 3 This is an example flow chart of a method for locating a lesion area combined with image quality assessment using ultrasound endoscopy as claimed in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] The terms "first", "second" and "third" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first", "second" and "third" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. All directional indications (such as up, down, left, right, front, back...) in the embodiments of this application are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0019] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0020] According to the first embodiment of the present invention, the present invention claims a method for locating a lesion area combined with an ultrasound endoscope image quality assessment, referring to Figure 1 ,include: Acquiring an ultrasonic image to be located, and preprocessing the ultrasonic image to be located to obtain a normalized ultrasonic image; Inputting the normalized ultrasound image into a backbone network encoder to obtain an ultrasound image feature vector; Inputting the ultrasonic image feature vector into the image quality assessment branch and the decoder respectively, and obtaining the quality assessment result and the lesion area information of the ultrasonic image to be located respectively; The lesion region information is output based on the quality assessment result.

[0021] Furthermore, the step of acquiring the ultrasonic image to be located, preprocessing the ultrasonic image to be located, and obtaining a normalized ultrasonic image further includes: Using an image processing library to read the ultrasound image to be located, and representing the ultrasound image to be located as an ultrasound image matrix or an ultrasound image tensor; Performing image scaling and normalization processing on the ultrasound image matrix or ultrasound image tensor; During the normalization process, the maximum and minimum pixel values ​​of the ultrasonic image matrix or the ultrasonic image tensor are obtained; The ultrasound image matrix or ultrasound image tensor is processed using a normalization formula to obtain a normalized ultrasound image.

[0022] In this embodiment, the image is scaled to a size of 512x512 and then normalized to a range of 0-1.

[0023] For each pixel value x, the following formula is used to normalize it to between 0 and 1:

[0024] Where x is the original pixel value; xmin is the minimum value of a pixel, usually 0, representing the darkest part of the image; xmax is the maximum value of a pixel, usually 255, representing the brightest part of the image, for 8-bit images.

[0025] When reading images, use an image processing library (such as OpenCV, PIL) to read the image and represent it as a matrix or tensor. Pixel values ​​are usually integers from 0 to 255 (for 8-bit images).

[0026] When calculating the maximum and minimum values, usually the minimum value of an 8-bit grayscale image is 0 and the maximum value is 255. If it is a color image, each channel (R, G, B) is normalized separately.

[0027] According to the above formula, each pixel value is mapped to the range of 0 to 1.

[0028] The normalized image will have floating point numbers between 0 and 1 as pixel values.

[0029] Furthermore, the step of inputting the normalized ultrasound image into a backbone network encoder to obtain an ultrasound image feature vector further includes: Reference Figure 2 , the backbone network encoder includes an initial module, a first stage layer, a first image block embedding layer, a second stage layer, a second image block embedding layer, a third stage layer, a third image block embedding layer, and a fourth stage layer in sequence; The initial module includes an initial convolution layer and an initial pooling layer; The first stage layer performs a first training and a first inference to obtain a first feature vector; Inputting the first feature vector into a first image block embedding layer to obtain a first embedded feature vector; Inputting the first embedded feature vector into the second stage layer to perform second training and second inference to obtain a second feature vector; Inputting the second feature vector into a second image block embedding layer to obtain a second embedded feature vector; Inputting the second embedded feature vector into the third stage layer to perform third training and third inference to obtain a third feature vector; Inputting the third feature vector into a third image block embedding layer to obtain a third embedded feature vector; The third embedded feature vector is input into the fourth stage layer to perform fourth training and fourth reasoning to obtain a fourth feature vector as an ultrasound image feature vector.

[0030] Among them, in the embodiments, FastViT is taken as an example for description, and it is noted that it can be replaced as a whole.

[0031] The initial module is the first few layers of the network, which is responsible for the initial processing and feature extraction of input data, and usually includes convolutional layers and pooling layers; During inference, the initial module consists of a 3x3 convolutional layer with a stride of 2, an activation layer, a 3x3 depthwise separable convolutional layer with a stride of 2, an activation layer, and a 1x1 convolutional layer and an activation layer; the training process is similar.

[0032] The image patch embedding layer divides the input image into small patches and converts each patch into a vector for processing in subsequent networks; In convolution and pooling operations, the stride is the length of the convolution kernel sliding on the input feature map. The size of the stride directly affects the spatial size of the output feature map.

[0033] Furthermore, the method further comprises: When the first stage layer performs the first training, the feature vector is input to the first BN layer and the first depth-separable convolutional layer, and then input to the first convolutional feedforward network to obtain the first training feature vector; When the first stage layer performs the first inference, the input feature vector passes through the second depth-separable convolutional layer and the second convolutional feed-forward network in sequence to obtain the first inference feature vector; Inputting the first feature vector into a first image block embedding layer to obtain a first embedded feature vector further includes: The first training feature vector is input into the first image block embedding training layer, and is superimposed with the result of the fourth depthwise separable convolutional layer with a preset step size through the third depthwise separable convolutional layer and the second BN layer, and then activated. The obtained results are respectively sent to two branches, including the first convolutional layer, the third BN layer branch, and the fourth BN layer, respectively. The results of the two branches are superimposed and activated, and the first embedding training feature vector is output; The first inference feature vector is input into the first image block embedding inference layer, sequentially passes through the fifth depth-separable convolutional layer and is activated, and then input into the second convolutional layer and is activated to obtain a second embedding inference feature vector.

[0034] Furthermore, the method further comprises: When the second stage layer performs the second training, the first embedding training feature vector is input to the fifth BN layer and the sixth depth-separable convolution layer, and then input to the third convolution feed-forward network to obtain the second training feature vector; When the second stage layer performs the second reasoning, the second embedding reasoning feature vector is input and sequentially passed through the seventh depth-separable convolutional layer and the fourth convolutional feed-forward network to obtain the second reasoning feature vector; Inputting the second feature vector into a second image block embedding layer to obtain a second embedded feature vector, further comprising: The second training feature vector is input into the second image block embedding training layer, passed through the eighth depth-separable convolution layer and the sixth BN layer, and superimposed with the result of the ninth depth-separable convolution layer with a preset step size, and then activated. The obtained results are respectively sent to two branches, including the third convolution layer, the seventh BN layer branch and the eighth BN layer, respectively. The results of the two branches are superimposed and activated, and the second embedding training feature vector is output; The second inference feature vector is input into the second image block embedding inference layer, sequentially passes through the tenth depth-separable convolutional layer and is activated, and then input into the fourth convolutional layer and is activated to obtain the second embedding inference feature vector.

[0035] Furthermore, the method further comprises: When the third stage layer performs the third training, the second embedding training feature vector is input to the ninth BN layer and the eleventh depth-separable convolution layer, and then input to the fifth convolution feed-forward network to obtain a third training feature vector; When the third stage layer performs the third reasoning, the third embedding reasoning feature vector is input and sequentially passed through the twelfth depth-separable convolutional layer and the sixth convolutional feed-forward network to obtain a third reasoning feature vector; Inputting the third feature vector into a third image block embedding layer to obtain a third embedded feature vector, further comprising: The third training feature vector is input into the third image block embedding training layer, passed through the thirteenth depthwise separable convolution layer and the tenth BN layer, and superimposed with the result of the fourteenth depthwise separable convolution layer with a preset step size, and then activated, and the obtained results are respectively sent to two branches, including the fifth convolution layer, the eleventh BN layer branch and the twelfth BN layer, respectively, and the results of the two branches are superimposed and activated, and the third embedding training feature vector is output; The third inference feature vector is input into the third image block embedding inference layer, sequentially passes through the fifteenth depth-separable convolutional layer and is activated, and then input into the sixth convolutional layer and is activated to obtain the third embedding inference feature vector.

[0036] Further, the third embedded feature vector is input into the fourth stage layer to perform fourth training and fourth reasoning to obtain a fourth feature vector as the ultrasound image feature vector, and further includes: Input the third embedded training feature vector into the first CPE module, pass through the thirteenth BN layer in sequence and then activate it, and then send it into the seventh convolution feedforward network to output a fourth training feature vector; The third embedding reasoning feature vector is input into the second CPE module, is activated after passing through the fourteenth BN layer in sequence, and is then sent into the eighth convolutional feedforward network to output a fourth reasoning feature vector; The fourth training feature vector and the fourth inference feature vector are fused to obtain a fourth feature vector as an ultrasound image feature vector.

[0037] In this embodiment, the first stage layer, the second stage layer and the third stage layer have the same structure, including: During training, it is composed of a skip connection structure consisting of a BatchNorm layer + a 3x3 depth-separable convolution layer and a convolutional feedforward network ConvFFN structure. During inference, it is composed of a 3x3 depth-separable convolution layer and a ConvFFN structure. ConvFFN is composed of a skip connection structure consisting of a 7x7 depth-separable convolution, BN, 1x1 convolution, activation layer, and 1x1 convolution. Each image block embedding inference layer has the same structure, including: The training process consists of a 7x7 depthwise separable convolution with a stride of 2 + BN, which is superimposed with a 2x2 depthwise separable convolution with a stride of 2, and then activated. The results are sent to two branches respectively, one of which is a 1x1 convolution + BN, and the other branch has only BN, and the results of the two branches are superimposed.

[0038] The reasoning process consists of: 7x7 depthwise separable convolution with a stride of 2, activation layer, 1x1 convolution, activation layer; The fourth stage layer is composed of a CPE module, a skip connection structure consisting of a BN+ activation layer, and a ConvFFN structure; during the training process, the CPE module will perform a skip connection structure alone.

[0039] Furthermore, the ultrasonic image feature vectors are respectively input into the image quality assessment branch to obtain the quality assessment result of the ultrasonic image to be located, and further includes: Sending the ultrasonic image feature vector to an image quality assessment branch network to obtain a sixteenth depth-separable convolutional layer and a global average pooling vector; Performing label classification on the ultrasound image to be located according to the global average pooling vector to obtain a classification result; Performing sigmoid and binarization operations on the classification results to obtain the result of the final image quality assessment branch; The label classification at least includes probe placement rationality, structure clarity, and background clarity.

[0040] Among them, in this embodiment, the image quality assessment result is the output of the classification head. After the sigmoid operation, the result obtained ranges from 0 to 1. There are three categories in total, namely "reasonableness of probe placement", "structural clarity" and "background clarity". The image quality assessment result of each category indicates the rationality of probe placement: rationality of probe position and angle; structural clarity: the boundary between high and low echoes; background clarity: the amount of noise and artifacts.

[0041] The Sigmoid function is a common activation function in machine learning and deep learning, which maps the input value to between 0 and 1.

[0042] It outputs 0.5 when the input is 0, approaches 0 when the input approaches negative infinity, and approaches 1 when the input approaches positive infinity.

[0043] At this time, if we take three independent confidence levels T1, T2, and T3 of "reasonableness of probe placement", "structural clarity", and "background clarity", we can binarize the three image quality assessment results and convert them into 0 and 1.

[0044] Reference Figure 3 In the example process of this embodiment, assuming that the original output is 0.6, 0.9, 0.2, and T1=T2=T3=0.5, the result after binarization is [1, 1, 0]. Binarization: Calculate according to the threshold value of each dimension 0.6>T1: the result is 1; 0.9>T2: the result is 1; 0.2<T3: the result is 0.

[0045] Furthermore, the ultrasonic image feature vectors are respectively input into a decoder to obtain the lesion area information of the ultrasonic image to be located, and further comprising: The decoder comprises a first decoding layer, a second decoding layer, a third decoding layer, a fourth decoding layer and a fifth decoding layer; The first decoding layer inputs the ultrasonic image feature vector, which passes through the first decoding convolution layer, the first decoding BN layer, the first ReLU layer and the first decoding deconvolution layer in sequence to obtain a first decoding feature vector; Inputting the first decoding feature vector into the second decoding layer, and sequentially passing through a second decoding convolutional layer, a second decoding BN layer, a second ReLU layer, and a second decoding deconvolutional layer to obtain a second decoding feature vector; Inputting the second decoding feature vector into the third decoding layer, and sequentially passing through a third decoding convolutional layer, a third decoding BN layer, a third ReLU layer, and a third decoding deconvolutional layer to obtain a third decoding feature vector; Inputting the third decoding feature vector into the fourth decoding layer, and sequentially passing through a fourth decoding convolutional layer, a fourth decoding BN layer, a fourth ReLU layer, and a fourth decoding deconvolutional layer to obtain a fourth decoding feature vector; The fourth decoding feature vector is input into the fifth decoding layer, and sequentially passes through a fifth decoding convolutional layer, a fifth decoding BN layer, a fifth ReLU layer and a fifth decoding deconvolutional layer to obtain a fifth decoding feature vector.

[0046] In this embodiment, the structures of the various layers of the decoder are as follows: The first decoding layer includes a Conv2D (3x3) - BN - ReLU layer and a DeConv (2x2) layer. The input is the result of encoder stage 4, and the output feature map has 4*Z channels and a size of one sixteenth of the input height H and width W (i.e. H / 16 and W / 16); The second decoding layer contains two Conv2D (3x3) - BN - ReLU layers and one DeConv (2x2) layer. The input is the concatenation of the result of DEC.1 and the encoder stage3. The output feature map has 2*Z channels and the size is one eighth of the input height H and width W (i.e. H / 8 and W / 8). The third decoding layer contains two Conv2D (3x3) - BN - ReLU layers and one DeConv (2x2) layer. The input is the concatenation of the DEC.2 result and the encoder stage2. The output feature map has Z channels and a size of one quarter of the input height H and width W (i.e. H / 4 and W / 4). The fourth decoding layer contains a Conv2D (3x3) - BN - ReLU layer and a DeConv (2x2) layer. The input is the concatenation of the DEC.3 result and the encoder stage 1. The output feature map has Z channels and its size is half of the input height H and width W (i.e. H / 2 and W / 2). The fifth decoding layer contains a Conv2D (3x3) - BN - ReLU layer and a DeConv (2x2) layer. The input is the concatenation of the result of DEC.4 and the original image. The number of channels of the output feature map is Z, and the size is the height H and width W of the input. The Z in the above description is 384; The lesion location is accurately segmented, including a Conv2D (3x3) - BN - ReLU layer and a Conv (1x1) layer. The input is the output of the decoder, and its output is a probability map with a dimension of H x W. Furthermore, the outputting of the lesion area information based on the quality assessment result further includes: When the sum of the scores of the three label categories of the quality assessment result is less than a preset score, the lesion area information is not displayed.

[0047] In this embodiment, the output probability map of the segmentation head is binarized by taking the threshold T4, and the mask map of the lesion and its corresponding contour result can be obtained; When the sum of the scores of the three dimensions of ultrasound image quality is less than 2 points, the lesion area will not be displayed because the quality of the ultrasound image collected by the doctor in this case is not ideal.

[0048] For example: Example score: (1, 1, 0) 1+1+0 ≥ 2; Therefore, it is considered that the image quality meets the requirements and can display the lesion area.

[0049] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0050] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

[0051] The specific implementation methods of the invention are described in detail above, but they are only examples, and the present application is not limited to the specific implementation methods described above. For those skilled in the art, any equivalent modification or substitution of the invention is also within the scope of the present application, and therefore, the equalization, modification, and improvement made without departing from the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. A method for locating a lesion region combined with image quality assessment of endoscopic ultrasound, characterized in that: include: Acquiring an ultrasonic image to be located, and preprocessing the ultrasonic image to be located to obtain a normalized ultrasonic image; Inputting the normalized ultrasound image into a backbone network encoder to obtain an ultrasound image feature vector; Inputting the ultrasonic image feature vector into the image quality assessment branch and the decoder respectively, and obtaining the quality assessment result and the lesion area information of the ultrasonic image to be located respectively; The lesion region information is output based on the quality assessment result.

2. The method for locating lesion areas combined with image quality assessment of ultrasound endoscopy as claimed in claim 1, characterized in that: The step of acquiring the ultrasonic image to be located, and preprocessing the ultrasonic image to be located to obtain a normalized ultrasonic image, further includes: Using an image processing library to read the ultrasound image to be located, and representing the ultrasound image to be located as an ultrasound image matrix or an ultrasound image tensor; Performing image scaling and normalization processing on the ultrasound image matrix or ultrasound image tensor; During the normalization process, the maximum and minimum pixel values ​​of the ultrasonic image matrix or the ultrasonic image tensor are obtained; The ultrasound image matrix or ultrasound image tensor is processed using a normalization formula to obtain a normalized ultrasound image.

3. The method for locating lesion areas combined with image quality assessment of ultrasound endoscopy as claimed in claim 1, characterized in that: The step of inputting the normalized ultrasound image into a backbone network encoder to obtain an ultrasound image feature vector further includes: The backbone network encoder includes an initial module, a first stage layer, a first image block embedding layer, a second stage layer, a second image block embedding layer, a third stage layer, a third image block embedding layer, and a fourth stage layer in sequence; The initial module includes an initial convolution layer and an initial pooling layer; The first stage layer performs a first training and a first inference to obtain a first feature vector; Inputting the first feature vector into a first image block embedding layer to obtain a first embedded feature vector; Inputting the first embedded feature vector into the second stage layer to perform second training and second inference to obtain a second feature vector; Inputting the second feature vector into a second image block embedding layer to obtain a second embedded feature vector; Inputting the second embedded feature vector into the third stage layer to perform third training and third inference to obtain a third feature vector; Inputting the third feature vector into a third image block embedding layer to obtain a third embedded feature vector; The third embedded feature vector is input into the fourth stage layer to perform fourth training and fourth reasoning to obtain a fourth feature vector as an ultrasound image feature vector.

4. The method for locating lesion areas combined with image quality assessment of ultrasound endoscopy as claimed in claim 3, characterized in that: Also includes: When the first stage layer performs the first training, the feature vector is input to the first BN layer and the first depth-separable convolutional layer, and then input to the first convolutional feedforward network to obtain the first training feature vector; When the first stage layer performs the first inference, the input feature vector passes through the second depth-separable convolutional layer and the second convolutional feed-forward network in sequence to obtain the first inference feature vector; Inputting the first feature vector into a first image block embedding layer to obtain a first embedded feature vector further includes: The first training feature vector is input into the first image block embedding training layer, and is superimposed with the result of the fourth depthwise separable convolutional layer with a preset step size through the third depthwise separable convolutional layer and the second BN layer, and then activated. The obtained results are respectively sent to two branches, including the first convolutional layer, the third BN layer branch, and the fourth BN layer, respectively. The results of the two branches are superimposed and activated, and the first embedding training feature vector is output; The first inference feature vector is input into the first image block embedding inference layer, sequentially passes through the fifth depth-separable convolutional layer and is activated, and then input into the second convolutional layer and is activated to obtain a second embedding inference feature vector.

5. The method for locating lesion areas combined with image quality assessment of ultrasound endoscopy as claimed in claim 4, characterized in that: Also includes: When the second stage layer performs the second training, the first embedding training feature vector is input to the fifth BN layer and the sixth depth-separable convolution layer, and then input to the third convolution feed-forward network to obtain the second training feature vector; When the second stage layer performs the second reasoning, the second embedding reasoning feature vector is input and sequentially passed through the seventh depth-separable convolutional layer and the fourth convolutional feed-forward network to obtain the second reasoning feature vector; Inputting the second feature vector into a second image block embedding layer to obtain a second embedded feature vector, further comprising: The second training feature vector is input into the second image block embedding training layer, passed through the eighth depth-separable convolution layer and the sixth BN layer, and superimposed with the result of the ninth depth-separable convolution layer with a preset step size, and then activated. The obtained results are respectively sent to two branches, including the third convolution layer, the seventh BN layer branch and the eighth BN layer, respectively. The results of the two branches are superimposed and activated, and the second embedding training feature vector is output; The second inference feature vector is input into the second image block embedding inference layer, sequentially passes through the tenth depth-separable convolutional layer and is activated, and then input into the fourth convolutional layer and is activated to obtain the second embedding inference feature vector.

6. The method for locating lesion areas combined with image quality assessment of ultrasound endoscopy as claimed in claim 5, characterized in that: Also includes: When the third stage layer performs the third training, the second embedding training feature vector is input to the ninth BN layer and the eleventh depth-separable convolution layer, and then input to the fifth convolution feed-forward network to obtain a third training feature vector; When the third stage layer performs the third reasoning, the third embedding reasoning feature vector is input and sequentially passed through the twelfth depth-separable convolutional layer and the sixth convolutional feed-forward network to obtain a third reasoning feature vector; Inputting the third feature vector into a third image block embedding layer to obtain a third embedded feature vector, further comprising: The third training feature vector is input into the third image block embedding training layer, passed through the thirteenth depthwise separable convolution layer and the tenth BN layer, and superimposed with the result of the fourteenth depthwise separable convolution layer with a preset step size, and then activated, and the obtained results are respectively sent to two branches, including the fifth convolution layer, the eleventh BN layer branch and the twelfth BN layer, respectively, and the results of the two branches are superimposed and activated, and the third embedding training feature vector is output; The third inference feature vector is input into the third image block embedding inference layer, sequentially passes through the fifteenth depth-separable convolutional layer and is activated, and then input into the sixth convolutional layer and is activated to obtain the third embedding inference feature vector.

7. The method for locating lesion areas combined with image quality assessment of ultrasound endoscopy as claimed in claim 6, characterized in that: Inputting the third embedded feature vector into the fourth stage layer to perform fourth training and fourth reasoning to obtain a fourth feature vector as an ultrasound image feature vector, further comprising: Input the third embedded training feature vector into the first CPE module, pass through the thirteenth BN layer in sequence and then activate it, and then send it into the seventh convolution feedforward network to output a fourth training feature vector; The third embedding reasoning feature vector is input into the second CPE module, is activated after passing through the fourteenth BN layer in sequence, and is then sent into the eighth convolutional feedforward network to output a fourth reasoning feature vector; The fourth training feature vector and the fourth inference feature vector are fused to obtain a fourth feature vector as an ultrasound image feature vector.

8. The method for locating lesion areas combined with image quality assessment of ultrasound endoscopy as claimed in claim 1, characterized in that: Inputting the ultrasonic image feature vectors into the image quality assessment branch respectively to obtain the quality assessment result of the ultrasonic image to be located, further comprising: Sending the ultrasonic image feature vector to an image quality assessment branch network to obtain a sixteenth depth-separable convolutional layer and a global average pooling vector; Performing label classification on the ultrasound image to be located according to the global average pooling vector to obtain a classification result; Performing sigmoid and binarization operations on the classification results to obtain the result of the final image quality assessment branch; The label classification at least includes probe placement rationality, structure clarity, and background clarity.

9. The method for locating lesion areas combined with image quality assessment of ultrasound endoscopy as claimed in claim 1, characterized in that: Inputting the ultrasonic image feature vectors into a decoder respectively to obtain the lesion area information of the ultrasonic image to be located, further comprising: The decoder comprises a first decoding layer, a second decoding layer, a third decoding layer, a fourth decoding layer and a fifth decoding layer; The first decoding layer inputs the ultrasonic image feature vector, which passes through a first decoding convolutional layer, a first decoding BN layer, a first ReLU layer and a first decoding deconvolutional layer in sequence to obtain a first decoding feature vector; Inputting the first decoding feature vector into the second decoding layer, and sequentially passing through a second decoding convolutional layer, a second decoding BN layer, a second ReLU layer, and a second decoding deconvolutional layer to obtain a second decoding feature vector; Inputting the second decoding feature vector into the third decoding layer, sequentially passing through a third decoding convolutional layer, a third decoding BN layer, a third ReLU layer and a third decoding deconvolutional layer, to obtain a third decoding feature vector; Inputting the third decoding feature vector into the fourth decoding layer, and sequentially passing through a fourth decoding convolutional layer, a fourth decoding BN layer, a fourth ReLU layer, and a fourth decoding deconvolutional layer to obtain a fourth decoding feature vector; The fourth decoding feature vector is input into the fifth decoding layer, and sequentially passes through a fifth decoding convolutional layer, a fifth decoding BN layer, a fifth ReLU layer and a fifth decoding deconvolutional layer to obtain a fifth decoding feature vector.

10. The method for locating lesion areas combined with image quality assessment of ultrasound endoscopy as claimed in claim 1, characterized in that: The outputting of the lesion area information based on the quality assessment result further includes: When the sum of the scores of the three label categories of the quality assessment result is less than a preset score, the lesion area information is not displayed.

Images