Ultrasonic image lesion recognition method and related device

Abstract

The embodiments of the present application provide a method for identifying lesions in ultrasound images and related devices, which obtain the extraction results of multiple image features of ultrasound images through multiple separable convolution layers. Each time a convolution is performed, the size of the ultrasound image is reduced by N times and the number of RGB channels is increased by N times, so that the extraction results of the lower convolution layers contain richer global features. Furthermore, the lesion area features are determined from each extraction result based on the attention mechanism, and an image feature map corresponding to the extraction result is generated based on the lesion area features. Finally, the feature map to be displayed is determined based on the size of the lesion area in each image feature map. The above embodiments can realize home diagnosis of lesions in ultrasound images, avoid the risk of misdiagnosis that is inevitable in manual diagnosis, and save the cost of ultrasound examination for patients.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method for identifying lesions in ultrasonic images and related devices. Background Art

[0002] Traditional ultrasound lesion examinations rely on the radiologist's diagnosis to determine the lesion's location and whether the tumor is benign or malignant. This manual diagnostic method relies on the radiologist's own experience and knowledge.

[0003] Inexperienced doctors often struggle to diagnose complex lesions, while experienced doctors can experience visual fatigue after analyzing numerous ultrasound images. The burden of extensive image review also creates the potential risk of misdiagnosis. Consequently, for patients undergoing ultrasound examinations, these diagnostic methods require the time and expense of a hospital visit, leading to high medical costs. Summary of the Invention

[0004] The embodiments of the present application provide a method and related device for lesion recognition in ultrasound images, which reduces the amount of model calculation through a separable convolutional layer and adds an attention mechanism to suppress the output of invalid features, thereby improving the accuracy of lesion recognition and providing recognition accuracy.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying lesions in an ultrasound image, the method comprising:

[0006] Obtaining a currently captured ultrasound image, and sequentially extracting multiple image features representing different levels from the ultrasound image through multiple separable convolutional layers (DSCs); wherein each time the DSCs are performed, the size of the ultrasound image is reduced by a factor of N, and the number of RGB channels is increased by a factor of N, where N is a positive integer;

[0007] Determine the lesion region features in each extraction result based on the attention mechanism, and generate an image feature map corresponding to the extraction result based on the lesion region features;

[0008] According to the size of the lesion area in each image feature map, a feature map to be displayed is determined from multiple image feature maps, and the feature map to be displayed is output.

[0009] The embodiment of the present application obtains the extraction results of multiple image features of an ultrasound image through multiple separable convolution layers. Each time a convolution is performed, the size of the ultrasound image is reduced by N times and the number of RGB channels is increased by N times, so that the extraction results of the lower convolution layers contain richer global features. Furthermore, the lesion area features are determined from each extraction result based on the attention mechanism, and an image feature map corresponding to the extraction result is generated based on the lesion area features. Finally, the feature map to be displayed is determined according to the size of the lesion area in each image feature map. The embodiment of the present application uses separable convolution layers to reduce the amount of model calculation, and adds an attention mechanism to suppress the output of invalid features, thereby improving the accuracy of lesion identification and providing recognition accuracy. The above algorithm can realize home diagnosis of lesions in ultrasound images, avoid the risk of misdiagnosis that is inevitable in manual diagnosis, and save the cost of ultrasound examination for patients.

[0010] In some possible embodiments, before sequentially extracting the extraction results of multiple image features representing different levels from the ultrasound image through multiple separable convolutional layers (DSC), the method further includes:

[0011] The ultrasound image is input into the convolution layer so that the number of RGB channels of the ultrasound image is increased to a preset threshold without changing its size.

[0012] In the embodiment of the present application, before performing multiple separable convolutions, the ultrasound image is pre-convolved with ordinary convolution to increase the image features in the ultrasound image.

[0013] In some possible embodiments, sequentially extracting the extraction results of multiple image features representing different levels from the ultrasound image through multiple separable convolutional layers DSC includes:

[0014] determining an image feature of the ultrasound image according to an output result of the DSC;

[0015] The image features are input into a deformable convolution DC, and the output result of the DC is used as the extraction result.

[0016] After the separable convolution layer of the embodiment of the present application obtains the image features of the ultrasound image, the ability of the neural network model to learn geometric deformation is improved through deformable convolution, so as to enhance the feature learning ability of the model and thereby improve the edge feature extraction of the lesion by the shallow network.

[0017] In some possible embodiments, before determining and outputting a feature map to be displayed from a plurality of image feature maps based on the size of the lesion area in the image feature map, the method further includes:

[0018] The extraction result corresponding to the bottom DSC is input into the spatial pyramid pooling layer, and the spatial pyramid pooling layer performs a pooling operation based on different preset pooling kernels to determine the local features and global features of the extraction result;

[0019] Performing feature fusion on the local features and the global features to determine a fused feature result, and determining a lesion area feature in the fused feature result based on the attention machine;

[0020] An image feature map corresponding to the fusion feature result is generated according to the lesion area features.

[0021] In this embodiment, the extraction results of the bottom-level DSC are input into a spatial pyramid pooling layer, which is equipped with different pooling kernels to determine the global and local features of the extraction results. The global and local features are fused and the fusion result is converted into an image feature map, thereby enriching the lesion area features in the image feature map and improving the recognition accuracy of the neural network model.

[0022] In some possible embodiments, each image feature map is preset with at least one prior frame, and determining and outputting a feature map to be displayed from multiple image feature maps based on the size of the lesion area in each image feature map includes:

[0023] In the order of the size of each image feature map from small to large, the size of the prior box of each image feature map is compared with the size of the lesion area in the image feature map;

[0024] The image feature map whose ratio between the size of the prior frame and the size of the lesion area meets the first ratio threshold is used as the feature map to be processed;

[0025] Based on non-maximum suppression (NSM), an image whose priori box size best matches the lesion area size is selected from each feature map to be processed, and the image is output as the feature map to be displayed.

[0026] In this embodiment, a priori boxes are pre-set for image feature maps of various sizes. The sizes of the priori boxes are compared with the sizes of the lesion areas to select the image feature maps that are suitable for the user to view. Furthermore, NSM is used for secondary screening to select the image feature map that is most suitable for the user to view as the feature map to be displayed.

[0027] In some possible embodiments, determining and outputting a feature map to be displayed from a plurality of image feature maps based on the size of the lesion area in each image feature map includes:

[0028] In the order of the size of each image feature map from small to large, the size of each image feature map is compared with the size of the corresponding lesion area;

[0029] If the ratio of the size of the image feature map to the size of the corresponding lesion area meets a second ratio threshold, the image feature map is output as the feature map to be displayed.

[0030] In the embodiment of the present application, image feature maps of different sizes are compared with the sizes of their corresponding lesion areas. When the size ratio meets the second ratio threshold, it indicates that the lesion area features in the image feature map can be well displayed to the user, so that the user can view the lesion area.

[0031] In some possible embodiments, before outputting the feature graph to be displayed, the method further includes:

[0032] The size of the image feature map is determined to be the minimum size among the image feature maps.

[0033] Before outputting the feature map to be displayed, the embodiment of the present application needs to determine that the feature map to be displayed is an image feature map of minimum size, so that the image feature map contains rich lesion area features and improves recognition accuracy.

[0034] In some possible embodiments, the method further includes:

[0035] If the size of the image feature map is not the minimum size among the image feature maps, deconvolution is performed on each image to be processed so that the size and the number of RGB channels of the image to be processed are the same as those of the image feature map; wherein the image to be processed is the image feature map whose size is smaller than that of the image feature map;

[0036] The deconvolved image to be processed is subjected to feature fusion with the image feature map, and the fusion result is used as the feature map to be displayed.

[0037] In the embodiment of the present application, if the image feature map is not the image feature map of the minimum size, it is necessary to perform deconvolution on other image feature maps with a size smaller than the image feature map to ensure that the size and number of RGB channels of each image feature map are consistent with the image feature map. Then, the image feature maps are fused by feature fusion to increase the characteristics of the lesion area in the image and improve the recognition accuracy.

[0038] In some possible embodiments, before outputting the image feature map, the method further includes:

[0039] After determining the feature map to be displayed, marking the lesion area in the feature map to be displayed;

[0040] Add description information of the recognition result of the lesion area at a preset position of the feature image to be displayed.

[0041] The feature map to be displayed output by the embodiment of the present application is marked with the lesion area in the image, and a description of the recognition result of the lesion area is attached at a preset position to allow the user to intuitively understand the diagnosis result of the lesion.

[0042] In a second aspect, an embodiment of the present application provides a device for identifying lesions using ultrasound images, the device comprising:

[0043] a feature extraction module configured to acquire a currently captured ultrasound image and sequentially extract multiple image features representing different levels from the ultrasound image through multiple separable convolutional layers (DSCs); wherein each time the DSCs are performed, the size of the ultrasound image is reduced by a factor of N and the number of RGB channels is increased by a factor of N, where N is a positive integer;

[0044] a feature map confirmation module, configured to determine the lesion region features in each extraction result based on the attention mechanism, and generate an image feature map corresponding to the extraction result based on the lesion region features;

[0045] The feature map display module is configured to determine the feature map to be displayed from multiple image feature maps based on the size of the lesion area in each image feature map, and output the feature map to be displayed.

[0046] In some possible embodiments, before executing the extraction results of sequentially extracting multiple image features representing different levels from the ultrasound image through multiple separable convolutional layers (DSC), the feature extraction module is further configured to:

[0047] The ultrasound image is input into the convolution layer so that the number of RGB channels of the ultrasound image is increased to a preset threshold without changing its size.

[0048] In some possible embodiments, the extraction results of sequentially extracting multiple image features representing different levels from the ultrasound image through multiple separable convolutional layers (DSC) are performed, and the feature extraction module is configured to:

[0049] determining an image feature of the ultrasound image according to an output result of the DSC;

[0050] The image features are input into a deformable convolution DC, and the output result of the DC is used as the extraction result.

[0051] In some possible embodiments, before determining and outputting the feature map to be displayed from multiple image feature maps based on the size of the lesion area in the image feature map, the feature map confirmation module is further configured to:

[0052] The extraction result corresponding to the bottom DSC is input into the spatial pyramid pooling layer, and the spatial pyramid pooling layer performs a pooling operation based on different preset pooling kernels to determine the local features and global features of the extraction result;

[0053] Performing feature fusion on the local features and the global features to determine a fused feature result, and determining a lesion area feature in the fused feature result based on the attention machine;

[0054] An image feature map corresponding to the fusion feature result is generated according to the lesion area features.

[0055] In some possible embodiments, each image feature map is preset with at least one prior frame, and the feature map confirmation module is configured to determine and output the feature map to be displayed from multiple image feature maps based on the size of the lesion area in each image feature map:

[0056] In the order of the size of each image feature map from small to large, the size of the prior box of each image feature map is compared with the size of the lesion area in the image feature map;

[0057] The image feature map whose ratio between the size of the prior frame and the size of the lesion area meets the first ratio threshold is used as the feature map to be processed;

[0058] Based on non-maximum suppression (NSM), an image whose priori box size best matches the lesion area size is selected from each feature map to be processed, and the image is output as the feature map to be displayed.

[0059] In some possible embodiments, the step of determining and outputting a feature map to be displayed from a plurality of image feature maps based on the size of the lesion area in each image feature map may be performed, and the feature map confirmation module may be configured to:

[0060] In the order of the size of each image feature map from small to large, the size of each image feature map is compared with the size of the corresponding lesion area;

[0061] If the ratio of the size of the image feature map to the size of the corresponding lesion area meets a second ratio threshold, the image feature map is output as the feature map to be displayed.

[0062] In some possible embodiments, before outputting the feature graph to be displayed, the feature graph confirmation module is further configured to:

[0063] The size of the image feature map is determined to be the minimum size among the image feature maps.

[0064] In some possible embodiments, the feature map confirmation module is further configured to:

[0065] If the size of the image feature map is not the minimum size among the image feature maps, deconvolution is performed on each image to be processed so that the size and the number of RGB channels of the image to be processed are the same as those of the image feature map; wherein the image to be processed is the image feature map whose size is smaller than that of the image feature map;

[0066] The deconvolved image to be processed is subjected to feature fusion with the image feature map, and the fusion result is used as the feature map to be displayed.

[0067] In some possible embodiments, before outputting the image feature map, the feature map display module is further configured to:

[0068] After determining the feature map to be displayed, marking the lesion area in the feature map to be displayed;

[0069] Add description information of the recognition result of the lesion area at a preset position of the feature image to be displayed.

[0070] In a third aspect, an embodiment of the present application also provides an ultrasound device, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided in the first aspect of the embodiment of the present application.

[0071] In a fourth aspect, an embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method provided in the first aspect of the embodiment of the present application.

[0072] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0074] Figure 1 A schematic diagram of an application scenario shown in an embodiment of the present application;

[0075] Figure 2A schematic diagram of a generative adversarial network shown in an embodiment of the present application;

[0076] Figure 3a This is an overall flow chart of the method for identifying lesions using ultrasound images according to an embodiment of the present application;

[0077] Figure 3b This is a schematic diagram of the network structure of the recognition model shown in the embodiment of the present application;

[0078] Figure 3c This is a schematic diagram showing characteristics of the lesion area according to an embodiment of the present application;

[0079] Figure 3d This is a diagram showing the principle of the attention mechanism according to an embodiment of the present application;

[0080] Figure 4 This is a structural diagram of an ultrasound image lesion identification device 400 according to an embodiment of the present application;

[0081] Figure 5 This is a schematic diagram of an ultrasound device 130 according to an embodiment of the present application. DETAILED DESCRIPTION

[0082] The technical solutions in the embodiments of the present application will be described clearly and in detail below with reference to the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, "face" will mean or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0083] In the description of the embodiments of the present application, unless otherwise specified, the term "multiple" refers to two or more, and other quantifiers should be understood similarly. The preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0084] To further illustrate the technical solutions provided by the embodiments of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of the present application provide the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present application. During the actual processing process or when the control device is executed, the method can be executed in the order of the methods shown in the embodiments or drawings or in parallel.

[0085] Traditional medical diagnosis requires radiologists to manually interpret and analyze ultrasound images for lesion information. This visual fatigue caused by analyzing numerous images can lead to potential misdiagnoses. Furthermore, subjectivity is a major drawback of traditional manual methods, with diagnostic results limited by the radiologist's own experience and knowledge. Consequently, for patients undergoing ultrasound examinations, these diagnostic methods require time and expense from hospital visits, resulting in high costs.

[0086] In order to solve the above problems, the inventive concept of the present application is as follows: the embodiment of the present application obtains the extraction results of multiple image features of the ultrasound image through multiple separable convolution layers. Each time the convolution is performed, the ultrasound image is proportionally reduced and the number of RGB channels is increased by a preset number. Therefore, the extraction results of the lower convolution layers contain richer global features. Furthermore, the lesion area features are determined from each extraction result based on the attention mechanism to generate a corresponding image feature map. According to the size of the lesion area in each image feature map, the feature map to be displayed is determined for display. In the above manner, a separable convolution layer is used to reduce the amount of model calculation, and an attention mechanism is added to suppress the output of invalid features, thereby improving the accuracy of lesion identification and providing recognition accuracy. In this way, home diagnosis of lesions in ultrasound images is achieved, the risk of misdiagnosis in manual diagnosis is avoided, and the cost of ultrasound examination for patients is saved.

[0087] The following describes in detail a method for identifying lesions in ultrasound images provided in an embodiment of the present application with reference to the accompanying drawings.

[0088] See also Figure 1 , Figure 1 FIG1 is a schematic diagram of an application environment according to an embodiment of the present application, including a network 100 , at least one terminal device 101 , an ultrasound device 102 , and a server 103 .

[0089] At least one terminal device 101 communicates with the ultrasound device 102 via a network 100 , and the network 100 may be a wired network 100 or a wireless network 100 .

[0090] The ultrasound device 102 is embedded with a recognition algorithm for intelligently recognizing ultrasound images, and users can use the ultrasound device to perform ultrasound examinations on human organs such as the thyroid gland and breast.

[0091] In some possible embodiments, the above-mentioned recognition algorithm may be embedded in the server 103 , and the ultrasound device 102 sends the captured ultrasound image to the server 103 for lesion diagnosis.

[0092] The server 103 uses a pre-trained recognition model to identify lesions in the input ultrasound image and transmits the recognition results to the terminal device 101 through the network 10 for the user to view. Figure 1 The terminal devices shown include a smart computer 101_1, a smart phone 101_2, and a smart TV 101_3, which have display functions.

[0093] Deep learning algorithms are widely used in technical fields such as image classification, object detection, and image segmentation. However, the limited sample size of medical data has hindered the further integration of deep learning with medical diagnosis and treatment. On the one hand, certain major diseases have low incidence rates and scattered patient distribution. This makes it difficult to collect sufficient samples for network construction due to the imbalance and isolation of medical resources. On the other hand, the acquisition, transmission, and recording of medical images are limited by factors such as imaging equipment, radiation dose, and acquisition time, which inevitably introduces noise. Many medical images are contaminated, making it difficult to form datasets that support network training.

[0094] For these reasons, the current data augmentation methods commonly used in medical testing are still primarily traditional methods, such as rotation, translation, and random cropping. Traditional data augmentation contributes to neural network performance by mimicking the state of real-world objects. However, in medical imaging research, translation and occlusion often have little impact on network performance. For categories with scarce data, this augmentation method can also lead to severe overfitting, hindering network analysis.

[0095] To solve the above problems, the embodiment of the present application pre-expands the training samples of medical images based on the generative adversarial network to obtain more medical images suitable for training. Figure 2 As shown in the figure, the generative adversarial network model mainly consists of two parts: the generation model (Generator) and the discriminator model (Discriminator).

[0096] The Generator (G) is an image-generating network that receives random noise (noise) based on prior knowledge and uses it to generate fake images (fake). The prior knowledge consists of image characteristics of lesions, such as tumor size, growth direction, aspect ratio, boundary features, echogenicity, cysticity, and calcifications. By expressing this as a sum vector and linking a random noise vector with a conditional vector generated by the prior knowledge, the Generator (G) can be conditioned to produce higher-quality images. The Discriminator (D) is a discriminator network that determines whether the input image is "real." Its input parameter is an image, and its output is the recognition probability of the image. An output of 1 indicates that the image is 100% real (prediction result: Real), while an output of 0 indicates that the image is 100% not real (prediction result: Fake). During training, the goal of the Generator (G) is to generate realistic images as much as possible to deceive the Discriminator (D). The goal of the Discriminator (D) is to distinguish the images generated by the Generator (G) from real images as much as possible. Thus, the Discriminator (D) and the Generator (G) form a dynamic "game." In this way, the two models compete with each other to reach an equilibrium. Ultimately, the output of the generated model can reflect the characteristics of the original sample to a certain extent, thereby achieving the goal of expanding the medical image dataset.

[0097] By expanding a small number of medical images through the above-mentioned generative adversarial network, a large number of high-quality medical images can be obtained to serve as training samples for training a recognition model for identifying lesion information in ultrasound images. Specifically, the lesion area in each medical image can be annotated by manual annotation, and the recognition model can be trained using the annotated medical image set so that the trained recognition model can recognize lesion information in medical images. Among them, the lesion information is the location of the lesion area in the ultrasound image and the characteristics of the lesion area. The lesion area characteristics are the characteristic information such as tumor size, growth direction, aspect ratio, boundary characteristics, echo, cysticity and calcification in the ultrasound image mentioned above.

[0098] Below through Figure 3a A method for identifying lesions in ultrasound images provided in an embodiment of the present application is described in detail, comprising the following steps:

[0099] Step 301: Obtain a currently captured ultrasound image, and sequentially extract multiple image features representing different levels from the ultrasound image through multiple separable convolutional layers (DSCs); wherein each time the DSCs are performed, the size of the ultrasound image is reduced by a factor of N, and the number of RGB channels is increased by a factor of N, where N is a positive integer;

[0100] The currently captured ultrasound image is pre-entered into a convolution layer, which increases the number of RGB channels to a preset threshold while preserving the image's size. This enriches the image's features. Next, multiple layers of deep separable convolution (DSC) are applied to the feature enhancement region, generating multiple feature outputs of varying sizes.

[0101] Specific as Figure 3b As shown, the size of the ultrasound image set in the embodiment of the present application is 416×416, including 3 RGB channels. After Convolution, the size of the ultrasound image remains unchanged, and the number of RGB channels is changed to 32. Furthermore, the image is input into two layers of DSC in sequence. After each layer of DSC is convolution processed, the size of the image is halved and the number of RGB channels is doubled, thereby obtaining image features of images of different sizes. That is, after the first layer of DSC, the image size becomes 208×208, and the number of RGB channels is 64; after the second layer of DSC, the image size becomes 104×104, and the number of RGB channels is 128. In this way, by setting multiple DSCs instead of ordinary convolutions, it is used to extract richer image features, and using multiple DSCs instead of ordinary convolutions can reduce the amount of computation of the network model.

[0102] In some possible embodiments, the ability of the neural network model to learn geometric deformation is enhanced by deformable convolution DC, so as to enhance the feature learning ability of the model and thereby improve the edge feature extraction of the lesion by the shallow network. Figure 3b As shown in the figure, the image after two layers of DSC is further input into multiple DSCs for convolution processing, and the image features output by DSC are convolved with deformable convolution DC. The output of DC is used as the convolution result of this layer, that is, the extraction result of the image features corresponding to this layer. Accordingly, the size of the image after each DSC and deformable convolution processing will be reduced to half of the original size, and the number of RGB channels will be doubled. That is, Figure 3b After passing through the second layer of DSC, the image is input into the third layer of DSC, and after DC convolution, the image size becomes 52×52, and the number of RGB channels is 256; correspondingly, after entering the bottom layer of DSC, the image size becomes 26×26, and the number of channels becomes 512 after DC convolution.

[0103] It should be understood that the shallow network area in the network structure is mainly responsible for extracting smaller lesion features, and has more detailed features that are conducive to lesion positioning, while the deep network area is responsible for extracting larger lesion features, and has more semantic features that are conducive to lesion classification.

[0104] In order to achieve the fusion of semantic information and detail features, a feature pyramid structure is used to adjust the number of channels of the deep DSC layer through 1×1 convolution, and then the scale is adapted to the upper feature layer through 2x upsampling, and finally the feature fusion of the two layers is achieved through the addition operation. However, the deepest features of the feature pyramid structure are not fused with other features after the 1×1 channel dimensionality reduction, which will cause the loss of strong semantic information, which is not conducive to the category output prediction of the deepest feature layer. To solve the above problems, the embodiment of the present application inputs the extraction result corresponding to the bottom layer DSC into the spatial pyramid pooling layer, and performs a pooling operation based on different preset pooling kernels through the spatial pyramid pooling layer to determine the local features and global features of the extraction results. The local features and the global features are feature fused to determine the fused feature result, and the lesion area features in the fused feature result are determined based on the attention machine, and the image feature map corresponding to the fused feature result is generated according to the lesion area features.

[0105] Specific as Figure 3b As shown in FIG, the embodiment of the present application introduces spatial pyramid pooling (SPP) to enhance the deepest features. In the spatial pyramid pooling structure, by adopting a pooling operation with a fixed block size, the fusion of local features and global features is achieved to solve the problem of feature information loss in the deepest layer. The spatial pyramid pooling structure in the embodiment of the present application is shown in FIG. Figure 3b As shown in the figure, there are three pooling kernels: 13, 6, and 1. After obtaining the global and local features of the bottom-level DSC extraction results using different pooling kernels, the global and local features are fused using the Concat function to obtain multi-dimensional image feature information, namely, image features with a size of 13×13 and 1024 RGB channels. This improves the recognition model's accuracy in distinguishing benign and malignant tumors from lesions.

[0106] Step 302: determining the lesion region features in each extraction result based on the attention mechanism, and generating an image feature map corresponding to the extraction result based on the lesion region features;

[0107] The embodiment of the present application adds an attention model (AM) to the network structure to simulate the doctor's diagnosis process. Figure 3bThe extraction results obtained by each "DSC+DC" are subjected to ordinary convolution and then the attention mechanism AM is used to enhance the weight of important features in the extraction results and suppress the output of invalid features, thereby improving the accuracy of diagnosis of benign and malignant lesions.

[0108] The introduction of the attention mechanism is inspired by the diagnostic process of radiologists. For example, when radiologists screen for thyroid nodules based on ultrasound images, they often observe the image sections from different perspectives. Once a suspicious lesion is found during the observation process, the lesion will be magnified and carefully observed. The key features of the thyroid nodule in the ultrasound image are comprehensively scored, and the severity of the suspected lesion is finally graded. The attention mechanism is used to better learn the lesion features, specifically Figure 3c As shown, Figure 3c The thick line area on the left side of the figure is the lesion area. After AM increases the weight of the lesion area features and suppresses the output of invalid features, as shown in Figure 3c In the figure on the right, the information within the lesion area is clearly displayed.

[0109] The execution principle of the above attention mechanism can be as follows Figure 3d As shown, F1 is the extraction result corresponding to a certain DSC layer, that is, a feature map F. Among them, F1∈R W×H×C ; W represents the width of the feature map, H represents the height of the feature map, and C represents the number of RGB channels. The Pool function is used to compress the input F1 in the spatial dimension to obtain the feature M∈R W×H×C , F2 obtained by M contains richer lesion area features.

[0110] Step 303: Determine a feature map to be displayed from multiple image feature maps according to the size of the lesion area in each image feature map, and output the image feature map.

[0111] In the embodiment of the present application, at least one priori frame is provided for each size of the image feature map (i.e., the image feature map corresponding to each DSC). The priori frame is an anchor frame of artificially set size, which is used to compare the size with the region of interest in the image to determine whether the region of interest in the image can be well displayed. Specifically, a feature map to be displayed suitable for the user to watch is selected from each image feature map based on the priori frame. During implementation, the size of the priori frame in each image feature map is compared with the size of the corresponding lesion area in order from small to large in size of each image feature map. The image feature map whose ratio of the priori frame size to the lesion area size meets the first ratio threshold is used as the feature map to be processed.

[0112] In some possible embodiments, before outputting the feature map to be displayed, the embodiment of the present application needs to mark the lesion area in the feature map to be displayed, which can be specifically as follows: Figure 3bAs shown in the figure, non-maximum suppression (NMS) is used to filter the prior frame to achieve accurate prediction of the lesion. NMS can adapt the preset optimal prior frame to the lesion area. That is, it selects the image with the prior frame size that best matches the lesion area size from each feature map to be processed and outputs this image as the feature map to be displayed. NSM is then used for secondary screening to select the image feature map that is most suitable for the user to view as the feature map to be displayed.

[0113] After determining the feature map to be displayed in the above manner, the lesion area can be marked with the prior frame to allow the user to focus more directly on the lesion area in the image. In addition, this application will add a description of the recognition result of the lesion area at a preset position of the feature map to be displayed, so that the user can intuitively understand the diagnosis result of the lesion.

[0114] In some possible embodiments, after obtaining the image feature map corresponding to each DSC, the size of each image feature map is sequentially compared with the size of the corresponding lesion region in ascending order. If the ratio of the size of the image feature map to the size of the corresponding lesion region satisfies a second ratio threshold, the image feature map is output as the feature map to be displayed.

[0115] In addition, considering that the image feature map size and number of RGB channels corresponding to each convolutional layer are different, that is, the image features contained are different, based on this, if the size of the image feature map selected as the feature map to be displayed is not the minimum size among all image feature maps, each image to be processed needs to be deconvolved so that the size and number of RGB channels of the image to be processed are the same as those of the image feature map. The image to be processed is the image feature map with a size smaller than that of the image feature map. Furthermore, the deconvolved image to be processed is fused with the image feature map to enrich the features of the lesion area in the feature map to be displayed, and finally the fusion result is output as the feature map to be displayed.

[0116] In the above process, the embodiment of the present application uses separable convolution layers for convolution to reduce the model's computational complexity. An attention mechanism is also added to suppress the output of invalid features, improving the accuracy of lesion recognition and providing recognition accuracy. This allows the recognition model to have advantages such as simplicity, low computational complexity, and low memory usage. This enables home diagnosis of lesions in ultrasound images, avoiding the inevitable risk of misdiagnosis associated with manual diagnosis while saving patients the cost of ultrasound examinations.

[0117] Based on the same inventive concept, the embodiment of the present application provides an ultrasound image lesion recognition device 400, specifically as follows Figure 4 Shown, including:

[0118] The feature extraction module 401 is configured to acquire a currently captured ultrasound image and sequentially extract multiple image features representing different levels from the ultrasound image through multiple separable convolutional layers (DSCs); wherein each time the DSCs are performed, the size of the ultrasound image is reduced by a factor of N and the number of RGB channels is increased by a factor of N, where N is a positive integer;

[0119] The feature map confirmation module 402 is configured to determine the lesion region features in each extraction result based on the attention mechanism, and generate an image feature map corresponding to the extraction result based on the lesion region features;

[0120] The feature map display module 403 is configured to determine a feature map to be displayed from multiple image feature maps according to the size of the lesion area in each image feature map, and output the feature map to be displayed.

[0121] In some possible embodiments, before executing the extraction results of sequentially extracting multiple image features representing different levels from the ultrasound image through multiple separable convolutional layers DSC, the feature extraction module 401 is further configured to:

[0122] The ultrasound image is input into the convolution layer so that the number of RGB channels of the ultrasound image is increased to a preset threshold without changing its size.

[0123] In some possible embodiments, the extraction results of sequentially extracting multiple image features representing different levels from the ultrasound image through multiple separable convolutional layers (DSC) are performed, and the feature extraction module 401 is configured to:

[0124] determining an image feature of the ultrasound image according to an output result of the DSC;

[0125] The image features are input into a deformable convolution DC, and the output result of the DC is used as the extraction result.

[0126] In some possible embodiments, before determining and outputting the feature map to be displayed from multiple image feature maps based on the size of the lesion area in the image feature map, the feature map confirmation module 402 is further configured to:

[0127] The extraction result corresponding to the bottom DSC is input into the spatial pyramid pooling layer, and the spatial pyramid pooling layer performs a pooling operation based on different preset pooling kernels to determine the local features and global features of the extraction result;

[0128] Performing feature fusion on the local features and the global features to determine a fused feature result, and determining a lesion area feature in the fused feature result based on the attention machine;

[0129] An image feature map corresponding to the fusion feature result is generated according to the lesion area features.

[0130] In some possible embodiments, each image feature map is preset with at least one prior frame. To determine and output a feature map to be displayed from multiple image feature maps based on the size of the lesion area in each image feature map, the feature map confirmation module 402 is configured to:

[0131] In the order of the size of each image feature map from small to large, the size of the prior box of each image feature map is compared with the size of the lesion area in the image feature map;

[0132] The image feature map whose ratio between the size of the prior frame and the size of the lesion area meets the first ratio threshold is used as the feature map to be processed;

[0133] Based on non-maximum suppression (NSM), an image whose priori box size best matches the lesion area size is selected from each feature map to be processed, and the image is output as the feature map to be displayed.

[0134] In some possible embodiments, the feature map confirmation module 402 is configured to:

[0135] In the order of the size of each image feature map from small to large, the size of each image feature map is compared with the size of the corresponding lesion area;

[0136] If the ratio of the size of the image feature map to the size of the corresponding lesion area meets a second ratio threshold, the image feature map is output as the feature map to be displayed.

[0137] In some possible embodiments, before outputting the feature graph to be displayed, the feature graph confirmation module 402 is further configured to:

[0138] The size of the image feature map is determined to be the minimum size among the image feature maps.

[0139] In some possible embodiments, the feature map confirmation module 402 is further configured to:

[0140] If the size of the image feature map is not the minimum size among the image feature maps, deconvolution is performed on each image to be processed so that the size and the number of RGB channels of the image to be processed are the same as those of the image feature map; wherein the image to be processed is the image feature map whose size is smaller than that of the image feature map;

[0141] The deconvolved image to be processed is subjected to feature fusion with the image feature map, and the fusion result is used as the feature map to be displayed.

[0142] In some possible embodiments, before outputting the image feature map, the feature map display module 403 is further configured to:

[0143] After determining the feature map to be displayed, marking the lesion area in the feature map to be displayed;

[0144] Add description information of the recognition result of the lesion area at a preset position of the feature image to be displayed.

[0145] Refer to the following Figure 5 The ultrasound device 130 according to this embodiment of the present application will be described. Figure 5 The ultrasound device 130 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0146] like Figure 5 As shown, the ultrasound device 130 is implemented as a general electronic device. Components of the ultrasound device 130 may include, but are not limited to, the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0147] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a processor or local bus using any of a variety of bus architectures.

[0148] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0149] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0150] The ultrasound device 130 can also communicate with one or more external devices 134 (e.g., a keyboard, pointing device, etc.), one or more devices that enable a user to interact with the ultrasound device 130, and / or any device that enables the ultrasound device 130 to communicate with one or more other electronic devices (e.g., a router, modem, etc.). Such communication can occur via an input / output (I / O) interface 135. Furthermore, the ultrasound device 130 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules of the ultrasound device 130 via a bus 133. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with the ultrasound device 130, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0151] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 132 including instructions, and the instructions can be executed by the processor 131 of the apparatus 400 to perform the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0152] In an exemplary embodiment, a computer program product is further provided, including a computer program / instruction, which, when executed by the processor 131 , implements any of the methods for identifying lesions in ultrasound images provided in the present application.

[0153] In an exemplary embodiment, various aspects of the method for identifying lesions in ultrasound images provided by the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the method for identifying lesions in ultrasound images according to various exemplary embodiments of the present application described above in this specification.

[0154] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0155] The program product for lesion identification in ultrasound images of an embodiment of the present application may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on an electronic device. However, the program product of the present application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0156] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0157] Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0158] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on the remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect via the Internet).

[0159] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0160] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0161] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0162] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable image scaling device to produce a machine, so that the instructions executed by the processor of the computer or other programmable image scaling device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable image scaling device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, the instruction device being implemented in the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable image scaling device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process described in the flow. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0165] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0166] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for identifying lesions in ultrasound images, characterized in that: Applied to ultrasound equipment, the method comprises: Obtaining a currently captured ultrasound image, and sequentially extracting multiple image features representing different levels from the ultrasound image through multiple separable convolutional layers (DSCs); wherein each time the DSCs are performed, the size of the ultrasound image is reduced by a factor of N, and the number of RGB channels is increased by a factor of N, where N is a positive integer; The following method is used to obtain corresponding fusion feature results from the extraction results of the multiple DSCs: For every two adjacent DSC layers except the bottom DSC, execute separately; Performing N-fold upsampling on the extraction result of the deep DSC of the two adjacent DSCs, so that the extraction result of the deep DSC has the same scale as the extraction result of the other DSC; Fusing the up-sampled extraction result of the deep DSC with the extraction result of the other DSC layer to obtain fused feature results corresponding to the two adjacent DSCs; For the bottom-level DSC, execute: Inputting the extraction result corresponding to the bottom layer DSC into the spatial pyramid pooling layer, and performing a pooling operation based on preset different pooling kernels through the spatial pyramid pooling layer to determine the local features and global features of the extraction result; The local features and the global features are fused using the concatenation function Concat to obtain the fused feature result corresponding to the bottom-level DSC; Determining the lesion region features in each of the fused feature results based on the attention mechanism, and generating an image feature map corresponding to each of the fused feature results according to the lesion region features; According to the size of the lesion area in each image feature map, a feature map to be displayed is determined from multiple image feature maps, and the feature map to be displayed is output.

2. The method according to claim 1, characterized in that Before sequentially extracting the extraction results of multiple image features representing different levels from the ultrasound image through multiple separable convolutional layers (DSC), the method further includes: The ultrasound image is input into the convolution layer so that the number of RGB channels of the ultrasound image is increased to a preset threshold without changing its size.

3. The method according to claim 1, characterized in that The extraction results of sequentially extracting multiple image features representing different levels from the ultrasound image through multiple separable convolutional layers DSC include: determining an image feature of the ultrasound image according to an output result of the DSC; The image features are input into a deformable convolution DC, and the output result of the DC is used as the extraction result.

4. The method according to claim 1, wherein Each image feature map is preset with at least one prior frame. The method of determining and outputting a feature map to be displayed from multiple image feature maps based on the size of the lesion area in each image feature map includes: In the order of the size of each image feature map from small to large, the size of the prior box of each image feature map is compared with the size of the lesion area in the image feature map; The image feature map whose ratio between the size of the prior frame and the size of the lesion area meets the first ratio threshold is used as the feature map to be processed; Based on non-maximum suppression (NSM), an image whose priori box size best matches the lesion area size is selected from each feature map to be processed, and the image is output as the feature map to be displayed.

5. The method according to claim 1, wherein The step of determining and outputting a feature map to be displayed from a plurality of image feature maps according to the size of the lesion area in each image feature map comprises: In the order of the size of each image feature map from small to large, the size of each image feature map is compared with the size of the corresponding lesion area; If the ratio of the size of the image feature map to the size of the corresponding lesion area meets a second ratio threshold, the image feature map is output as the feature map to be displayed.

6. The method according to any one of claims 4 or 5, characterized in that Before outputting the feature graph to be displayed, the method further includes: The size of the image feature map is determined to be the minimum size among the image feature maps.

7. The method according to claim 6, characterized in that The method further comprises: If the size of the image feature map is not the minimum size among the image feature maps, deconvolution is performed on each image to be processed so that the size and the number of RGB channels of the image to be processed are the same as those of the image feature map; wherein the image to be processed is the image feature map whose size is smaller than that of the image feature map; The deconvolved image to be processed is subjected to feature fusion with the image feature map, and the fusion result is used as the feature map to be displayed.

8. The method according to any one of claims 1 to 5 or 7, characterized in that: Before outputting the image feature map, the method further includes: After determining the feature map to be displayed, marking the lesion area in the feature map to be displayed; Add description information of the recognition result of the lesion area at a preset position of the feature image to be displayed.

9. An ultrasonic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for identifying lesions in ultrasound images as described in any one of claims 1 to 7.

Images