Ultrasonic breast lesion classification method, device and storage medium

Abstract

The present application relates to a method, device and storage medium for classifying ultrasonic breast lesions, and belongs to the field of deep learning technology. The method comprises: dividing a target ultrasonic breast image or video (a video is a frame-by-frame image) into n1*n2 data blocks; converting each data block into p1*p2*c dimensional vector data; merging the vector data corresponding to the n1*n2 data blocks to obtain a two-dimensional data matrix of n1n2×p1p2c; generating a position coding vector corresponding to the position according to the position of each data block in the target ultrasonic breast image, and adding the position coding vector to the two-dimensional data matrix to obtain a data matrix to be processed; inputting the data matrix to be processed into an image classification network to obtain a classification of the lesion properties of the target ultrasonic breast lesions; the problem of low classification efficiency when manually classifying ultrasonic breast images can be solved; and the accuracy and efficiency of classifying ultrasonic breast images can be improved.

Description

Technical Field

[0001] The present application relates to a method, device and storage medium for classifying ultrasonic breast lesions, and belongs to the field of deep learning technology. Background Art

[0002] Breast cancer is one of the leading causes of mortality among women, and early screening is an important factor in preventing breast disease and prolonging survival.

[0003] Existing screening methods include: after collecting an ultrasonic breast image of a human body, medical personnel analyze the ultrasonic breast image to obtain the type of breast disease.

[0004] However, manual analysis of ultrasound breast images is slow and inefficient. Summary of the invention

[0005] The present application provides a method, device and storage medium for classifying ultrasound breast lesions, which can solve the problem of low efficiency of manual analysis of ultrasound breast images. The present application provides the following technical solutions:

[0006] In a first aspect, a method for classifying ultrasonic breast lesions is provided, the method comprising:

[0007] Acquire target ultrasonic breast information to be classified, wherein the target ultrasonic breast information is a target ultrasonic breast image or a target ultrasonic breast video, and the target ultrasonic breast video includes at least two frames of target ultrasonic breast images;

[0008] For each frame of the target ultrasound breast image,

[0009] Divide the target ultrasound breast image into n1*n2 data blocks; wherein n1 is the number of data blocks divided in the image height direction, n2 is the number of data blocks divided in the image width direction, and n1 and n2 are positive integers;

[0010] Convert each data block into p1*p2*c dimensional vector data; where n1=H / p1, n2=W / p2; H is the height of the input image, W is the width of the input image, p1 is the height of the segmented data block, and p2 is the width of the segmented data block;

[0011] Merge the vector data corresponding to the n1*n2 data blocks to obtain a two-dimensional data matrix of n1n2×p1p2c;

[0012] According to the position of each data block in the target ultrasound breast image, a position coding vector corresponding to the position is generated, and the position coding vector is added to the two-dimensional data matrix to obtain a data matrix to be processed; segmentation

[0013] The data matrix to be processed is input into a pre-trained image classification network to obtain the lesion property classification corresponding to the target ultrasonic breast image.

[0014] Optionally, the image classification network includes a multi-head attention module, a feedforward neural module and a multi-layer fully connected classification module;

[0015] The multi-head attention module includes three fully connected networks, an activation function layer and a multi-dimensional logistic regression layer, the input of each fully connected network is the data matrix to be processed, and the output is feature data of a preset dimension; wherein, after the feature data output by the preset two fully connected networks are multiplied and divided by a preset scale factor, a logistic regression result is obtained by calculating through a multi-dimensional logistic regression layer; after the logistic regression result is multiplied by the feature data of another fully connected network, the output result of the multi-head attention module is obtained; the other fully connected network is a fully connected network among the three networks that is different from the preset two fully connected networks;

[0016] The feedforward neural module includes a fully connected network, a linear rectification activation function connected to the fully connected network, and layer normalization; the output result of the multi-head attention module is passed through the fully connected network, the linear rectification activation function connected to the fully connected network, and layer normalization to obtain the output result of the feedforward neural module;

[0017] After receiving the output result of the feedforward neural module, the multi-layer fully connected classification module processes the data through a fully connected layer; the processed data is layer normalized to obtain the lesion property classification.

[0018] Optionally, if the target ultrasound breast image is an image in the target ultrasound breast video, the method further includes:

[0019] After obtaining the lesion property classification according to each frame of the target ultrasound breast image, the lesion property classification corresponding to the target ultrasound breast video is determined according to the lesion property classification obtained from each frame of the target ultrasound breast image in the target ultrasound breast video.

[0020] Optionally, the step of determining the lesion property classification corresponding to the target ultrasound breast video according to the lesion property classification obtained from each frame of the target ultrasound breast image in the target ultrasound breast video includes:

[0021] If there is no image classified as malignant in the target ultrasound breast images of each frame of the target ultrasound breast video, then counting the lesion property classification with the largest number among the lesion property classifications corresponding to the target ultrasound breast images of each frame, and determining the lesion property classification obtained by counting as the lesion property classification corresponding to the target ultrasound breast video;

[0022] If there is an image whose lesion property is classified as malignant in each frame of the target ultrasound breast image of the target ultrasound breast video, the lesion property classification corresponding to the target ultrasound breast video is determined to be malignant.

[0023] Optionally, the lesion property classification includes: benign type and malignant type; or, includes at least one of benign type, malignant type, inflammatory type, adenosis type, hyperplasia type, ductal dilatation type, early invasive cancer, invasive cancer, non-invasive cancer, lobular adenocarcinoma, ductal adenocarcinoma, medullary carcinoma, scirrhous cancer, simple cancer, carcinoma in situ, early cancer, invasive cancer, undifferentiated cancer, poorly differentiated cancer, moderately differentiated cancer, and well-differentiated cancer.

[0024] Optionally, there are multiple multi-head attention modules and multiple feedforward neural modules.

[0025] Optionally, the target ultrasonic breast image is a whole ultrasonic breast image or a breast lesion area image.

[0026] Optionally, the image classification network is trained based on random activation of weights.

[0027] In a second aspect, a device for classifying ultrasonic breast lesions is provided, the device comprising:

[0028] An information acquisition unit, used for acquiring target ultrasonic breast information to be classified, wherein the target ultrasonic breast information is a target ultrasonic breast image or a target ultrasonic breast video, wherein the target ultrasonic breast video includes at least two frames of target ultrasonic breast images;

[0029] An image segmentation unit, for each frame of the target ultrasound breast image, segments the target ultrasound breast image into n1*n2 data blocks; wherein n1 is the number of data blocks segmented in the image height direction, n2 is the number of data blocks segmented in the image width direction, and n1 and n2 are positive integers;

[0030] A data conversion unit, used for converting each data block into vector data of p1*p2*c dimensions; wherein n1=H / p1, n2=W / p2; H is the height of the input image, W is the width of the input image, p1 is the height of the data block after segmentation, and p2 is the width of the data block after segmentation;

[0031] A vector merging unit is used to merge the vector data corresponding to the n1*n2 data blocks to obtain a two-dimensional data matrix of n1n2×p1p2c;

[0032] A matrix generating unit, used for generating a position coding vector corresponding to each data block according to the position of the data block in the target ultrasonic breast image, and adding the position coding vector to the two-dimensional data matrix to obtain a data matrix to be processed;

[0033] The lesion classification unit is used to input the data matrix to be processed into a pre-trained image classification network to obtain the lesion property classification and segmentation corresponding to the target ultrasonic breast image.

[0034] Optionally, the image classification network includes a multi-head attention module, a feedforward neural module and a multi-layer fully connected classification module;

[0035] The multi-head attention module includes three fully connected networks, an activation function layer and a multi-dimensional logistic regression layer, the input of each fully connected network is the data matrix to be processed, and the output is feature data of a preset dimension; wherein, after the feature data output by the preset two fully connected networks are multiplied and divided by a preset scale factor, a logistic regression result is obtained through multi-layer logistic regression calculation; after the logistic regression result is multiplied by the feature data of another fully connected network, the output result of the multi-head attention module is obtained; the other fully connected network is a fully connected network among the three networks that is different from the preset two fully connected networks;

[0036] The output of the multi-head attention module is as follows:

[0037]

[0038] Among them, Q, K, and V are the results of the input data blocks after full connection, and d is the scale factor.

[0039] The feedforward neural module includes a fully connected network, a linear rectification activation function connected to the fully connected network, and layer normalization; the output result of the multi-head attention module is passed through the fully connected network, the linear rectification activation function connected to the fully connected network, and layer normalization to obtain the output result of the feedforward neural module;

[0040] The output of the feedforward neural network is as follows:

[0041] z out =LN(RELU(MLP(RELU(MLP(z in )))))), where z in is the input of the feedforward neural network, z out is the output of the feedforward neural network, LN is the layer normalization operation, RELU is the linear rectification activation function, and MLP is the fully connected layer.

[0042] After receiving the output result of the feedforward neural module, the multi-layer fully connected classification module processes the data through a fully connected layer; the processed data is layer normalized to obtain the lesion property classification.

[0043] In a third aspect, a device for classifying ultrasonic breast lesions is provided, the device comprising a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement the method for classifying ultrasonic breast lesions provided in the first aspect.

[0044] In a fourth aspect, a computer-readable storage medium is provided, wherein a program is stored in the storage medium, and when the program is executed by a processor, the method for classifying ultrasonic breast lesions provided in the first aspect is used to implement the method.

[0045] The beneficial effects of the present application include at least: for each frame of the target ultrasound breast image, the target ultrasound breast image is divided into n1*n2 data blocks; each data block is converted into p1*p2*c dimensional vector data; the vector data corresponding to the n1*n2 data blocks are merged to obtain a two-dimensional data matrix of n1n2×p1p2c; according to the position of each data block in the target ultrasound breast image, a position coding vector corresponding to the position is generated, and the position coding vector is added to the two-dimensional data matrix to obtain a data matrix to be processed; the data matrix to be processed is input into a pre-trained image classification network to obtain a lesion property classification corresponding to the target ultrasound breast image; the problem of low classification efficiency when manually classifying ultrasound breast images can be solved; since automatic classification can be achieved through an image classification model, the accuracy and efficiency of classifying ultrasound breast images can be improved.

[0046] In addition, by setting the image classification network to not include convolution operations, image classification is achieved through a fully connected network combined with other computing methods. Therefore, the amount and difficulty of model calculation can be reduced and the model calculation efficiency can be improved.

[0047] In addition, by dividing the image into multiple data blocks, converting each data block into vector data and merging them to input the image classification network, the amount of calculation of the input image classification network can be reduced, further improving the calculation efficiency of the model.

[0048] In addition, by setting the number of multi-head attention modules and feed-forward neural modules to multiple, the generalization ability of the image classification network can be improved.

[0049] In addition, by obtaining an image classification network based on weight random activation training, the generalization ability of the image classification network can also be improved.

[0050] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1is a flow chart of a method for classifying ultrasonic breast lesions provided by an embodiment of the present application;

[0052] Figure 2 is a schematic diagram of an image classification model provided by an embodiment of the present application;

[0053] Figure 3 is a block diagram of a device for classifying ultrasonic breast lesions provided by an embodiment of the present application;

[0054] Figure 4 This is a block diagram of an ultrasonic breast lesion classification device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0055] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0056] Optionally, the present application uses the execution subject of each embodiment as an electronic device with computing capability as an example for illustration. The electronic device may be a terminal or a server. The terminal may be an ultrasound imaging device, a computer, a mobile phone, a tablet computer, etc. This embodiment does not limit the type of electronic device.

[0057] The following is an introduction to the classification method of ultrasonic breast lesions provided in this application.

[0058] Figure 1 This is a flow chart of a method for classifying ultrasonic breast lesions provided by an embodiment of the present application. The method includes at least the following steps:

[0059] Step 101, obtaining target ultrasonic breast information to be classified, where the target ultrasonic breast information is a target ultrasonic breast image or a target ultrasonic breast video, and the target ultrasonic breast video includes at least two frames of target ultrasonic breast images.

[0060] Optionally, the target ultrasound breast image is a whole ultrasound breast image or a breast lesion area image.

[0061] Step 102, for each frame of the target ultrasound breast image, divide the target ultrasound breast image into n1*n2 data blocks; wherein n1 is the number of data blocks divided in the image height direction, n2 is the number of data blocks divided in the image width direction, and n1 and n2 are positive integers.

[0062] In one example, the target ultrasound breast image is segmented according to a preset size p1×p2. p1 and p2 corresponding to different data blocks are the same or different. p1 and p2 can be set by the user or by default in the electronic device. This embodiment does not limit the values ​​of p1 and p2.

[0063] Step 103, convert each data block into vector data of p1*p2*c dimensions, where p1 and p2 are the sizes of each data block.

[0064] Among them, n1=H / p1, n2=W / p2; H is the height of the input image, W is the width of the input image, p1 is the height of the data block after segmentation, p2 is the width of the data block after segmentation; n1 is the number of data blocks segmented in the direction of image height, and n2 is the number of data blocks segmented in the direction of image width.

[0065] In this embodiment, a target ultrasound breast image is divided into multiple data blocks, and each data block is converted into vector data. In this way, the amount of data input to the image classification model can be compressed, thereby improving the calculation efficiency.

[0066] Optionally, the method of converting each data block into vector data of p1*p2*c dimensions includes but is not limited to: using a neural network to convert the data block into a corresponding feature vector to obtain vector data; or, taking each pixel value in the data block as vector data. This embodiment does not limit the method of obtaining vector data.

[0067] Step 104, merge the vector data corresponding to the n1*n2 data blocks to obtain a two-dimensional data matrix of n1n2×p1p2c.

[0068] Step 105 , generating a position coding vector corresponding to the position of each data block in the target ultrasonic breast image, and adding the position coding vector to the two-dimensional data matrix to obtain a data matrix to be processed.

[0069] The position encoding vector is used to indicate the position of the data block in the target ultrasound breast image.

[0070] Step 106, inputting the data matrix to be processed into a pre-trained image classification network to obtain a classification of lesion properties corresponding to the target ultrasound breast image.

[0071] Among them, the lesion property classification is used to indicate the pathological property corresponding to the target ultrasound breast image.

[0072] refer to Figure 2 , the image classification network includes a multi-head attention module 21, a feedforward neural module 22 and a multi-layer fully connected classification module 23.

[0073] The multi-head attention module 21 includes three fully connected networks, an activation function layer and a multi-dimensional logistic regression layer. Of course, the multi-head attention module 21 can also include other network structures, which are not listed one by one in this embodiment. The input of each fully connected network is the data matrix to be processed, and the output is the feature data of a preset dimension; wherein, the feature data output by the preset two fully connected networks are multiplied and divided by the preset proportional factor, and then the logistic regression result is obtained through multi-dimensional logistic regression calculation; the logistic regression result is multiplied by the feature data of another fully connected network to obtain the output result of the multi-head attention module; the other fully connected network is a fully connected network among the three networks that is different from the preset two fully connected networks;

[0074] In one example, the output of the multi-head attention module is:

[0075]

[0076] Among them, Q, K, and V are the results of the input data blocks after full connection, and d is the scale factor.

[0077] The feedforward neural module 22 includes a fully connected network, a linear rectification activation function connected to the fully connected network, and layer normalization; the output result of the multi-head attention module passes through the fully connected network, the linear rectification activation function connected to the fully connected network, and layer normalization to obtain the output result of the feedforward neural module.

[0078] In one example, the output of the feedforward neural module 22 is:

[0079] z out =LN(RELU(MLP(RELU(MLP(z in )))))), where z in is the input of the feedforward neural network, z out is the output of the feedforward neural network, LN is the layer normalization operation, RELU is the linear rectification activation function, and MLP is the fully connected layer.

[0080] After receiving the output result of the feedforward neural module, the multi-layer fully connected classification module 23 processes the data through the fully connected layer; the processed data is layer normalized to obtain the classification of lesion properties.

[0081] Among them, the logistic regression calculation can be implemented through the softmax in the multi-head attention module 21.

[0082] In the feedforward neural module 22, the number of network units formed by the fully connected network and the linear rectifier activation function is one or more, Figure 2 In the description, an example is given in which the number of the network units is two.

[0083] Optionally, the lesion property classification includes: benign type and malignant type; or includes at least one of benign type, malignant type, inflammatory type, adenosis type, hyperplasia type, ductal dilatation type, early invasive cancer, invasive cancer, non-invasive cancer, lobular adenocarcinoma, ductal adenocarcinoma, medullary carcinoma, scirrhous cancer, simple cancer, carcinoma in situ, early cancer, invasive cancer, undifferentiated cancer, poorly differentiated cancer, moderately differentiated cancer, and well-differentiated cancer. In other embodiments, the lesion property classification can also be divided into other types, and this embodiment does not limit the division method of the lesion property classification.

[0084] Optionally, there are multiple multi-head attention modules and feedforward neural modules, which can improve the generalization of the model.

[0085] In this embodiment, the image classification network is obtained by training the initial neural network model using training data. The training data includes a sample data matrix corresponding to the sample ultrasound breast image and a classification label corresponding to the sample data matrix. During the training process, the sample data matrix is ​​input into the initial neural network model to obtain a model result; the difference between the model result and the classification label is calculated using a preset loss function, and the initial neural network model is iteratively trained according to the calculation result to finally obtain an image classification network. Schematically, the image classification network is obtained based on random activation training of weights, so that the generalization of the model can be further improved.

[0086] The type of classification label corresponds to the output type of the image classification network, and the network structure of the image classification network is the same as the network structure of the initial neural network model.

[0087] Optionally, if the target ultrasound breast image is an image in a target ultrasound breast video, the method further includes: after obtaining a lesion property classification based on each frame of the target ultrasound breast image, determining a property classification corresponding to the target ultrasound breast video based on the lesion property classification obtained from each frame of the target ultrasound breast image in the target ultrasound breast video.

[0088] Among them, the lesion property classification corresponding to the target ultrasonic breast video is determined according to the lesion property classification obtained from each frame of the target ultrasonic breast image in the target ultrasonic breast video, including: if there is no image with lesion property classified as malignant in each frame of the target ultrasonic breast image of the target ultrasonic breast video, then the lesion property classification with the largest number in the lesion property classification corresponding to each frame of the target ultrasonic breast image is counted, and the lesion property classification obtained by counting is determined as the lesion property classification corresponding to the target ultrasonic breast video; if there is an image with lesion property classified as malignant in each frame of the target ultrasonic breast image of the target ultrasonic breast video, then the lesion property classification corresponding to the target ultrasonic breast video is determined as malignant.

[0089] In summary, the ultrasonic breast lesion classification method provided in the present embodiment divides the target ultrasonic breast image into n1*n2 data blocks for each frame of the target ultrasonic breast image; converts each data block into p1*p2*c dimensional vector data; merges the vector data corresponding to the n1*n2 data blocks to obtain a two-dimensional data matrix of n1n2×p1p2c; generates a position coding vector corresponding to the position according to the position of each data block in the target ultrasonic breast image, and adds the position coding vector to the two-dimensional data matrix to obtain a data matrix to be processed; inputs the data matrix to be processed into a pre-trained image classification network to obtain a lesion property classification corresponding to the target ultrasonic breast image; can solve the problem of low classification efficiency when manually classifying ultrasonic breast images; because automatic classification can be achieved through an image classification model, the accuracy and efficiency of classifying ultrasonic breast images can be improved.

[0090] In addition, by setting the image classification network to not include convolution operations, image classification is achieved through a fully connected network combined with other computing methods. Therefore, the amount and difficulty of model calculation can be reduced and the model calculation efficiency can be improved.

[0091] In addition, by dividing the image into multiple data blocks, converting each data block into vector data and merging them to input the image classification network, the amount of calculation of the input image classification network can be reduced, further improving the calculation efficiency of the model.

[0092] In addition, by setting the number of multi-head attention modules and feed-forward neural modules to multiple, the generalization ability of the image classification network can be improved.

[0093] In addition, by obtaining an image classification network based on weight random activation training, the generalization ability of the image classification network can also be improved.

[0094] Figure 3 This is a block diagram of an ultrasonic breast lesion classification device provided by an embodiment of the present application. The device includes at least the following modules: an information acquisition unit 310, an image segmentation unit 320, a data conversion unit 330, a vector merging unit 340, a matrix generation unit 350 and a lesion classification unit 360.

[0095] The information acquisition unit 310 is used to acquire target ultrasonic breast information to be classified, wherein the target ultrasonic breast information is a target ultrasonic breast image or a target ultrasonic breast video, wherein the target ultrasonic breast video includes at least two frames of target ultrasonic breast images;

[0096] The image segmentation unit 320 is used to segment each frame of the target ultrasound breast image into n1*n2 data blocks; wherein n1 is the number of data blocks segmented in the image height direction, n2 is the number of data blocks segmented in the image width direction, and n1 and n2 are positive integers;

[0097] The data conversion unit 330 is used to convert each data block into vector data of p1*p2*c dimensions, where p1 and p2 are the sizes of each data block; wherein n1=H / p1, n2=W / p2; H is the height of the input image, W is the width of the input image, p1 is the height of the data block after segmentation, and p2 is the width of the data block after segmentation; n1 is the number of data blocks segmented in the image height direction, and n2 is the number of data blocks segmented in the image width direction;

[0098] A vector merging unit 340 is used to merge the vector data corresponding to the n1*n2 data blocks to obtain a two-dimensional data matrix of n1n2×p1p2c;

[0099] A matrix generating unit 350 is used to generate a position coding vector corresponding to each data block according to the position of the data block in the target ultrasonic breast image, and add the position coding vector to the two-dimensional data matrix to obtain a data matrix to be processed;

[0100] The lesion classification unit 360 is used to input the data matrix to be processed into a pre-trained image classification network to obtain the lesion property classification corresponding to the target ultrasound breast image.

[0101] For relevant details, refer to the above method embodiment.

[0102] It should be noted that: the ultrasonic breast lesion classification device provided in the above embodiment only uses the division of the above functional modules as an example when classifying ultrasonic breast lesions. In actual applications, the above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the ultrasonic breast lesion classification device is divided into different functional modules to complete all or part of the functions described above. In addition, the ultrasonic breast lesion classification device provided in the above embodiment and the ultrasonic breast lesion classification method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0103] Figure 4 4 is a block diagram of a device for classifying ultrasonic breast lesions provided in one embodiment of the present application. The device at least includes a processor 401 and a memory 402.

[0104] The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0105] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, which is used to be executed by the processor 401 to implement the ultrasonic breast lesion classification method provided in the method embodiment of the present application.

[0106] In some embodiments, the ultrasonic breast lesion classification device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 401, the memory 402 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0107] Of course, the ultrasonic breast lesion classification device may also include fewer or more components, which is not limited in this embodiment.

[0108] Optionally, the present application also provides a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the ultrasonic breast lesion classification method of the above method embodiment.

[0109] Optionally, the present application also provides a computer product, which includes a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the ultrasonic breast lesion classification method of the above method embodiment.

[0110] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for classifying breast lesions using ultrasound, characterized in that: The method comprises: Acquire target ultrasonic breast information to be classified, wherein the target ultrasonic breast information is a target ultrasonic breast image or a target ultrasonic breast video, and the target ultrasonic breast video includes at least two frames of target ultrasonic breast images; For each frame of target ultrasound breast image, the target ultrasound breast image is divided into n1*n2 data blocks; wherein n1 is the number of data blocks divided in the image height direction, n2 is the number of data blocks divided in the image width direction, and n1 and n2 are positive integers; Convert each data block into p1*p2*c dimensional vector data; where n1= H / p1, n2= W / p2; H is the height of the input image, W is the width of the input image, p1 is the height of the segmented data block, and p2 is the width of the segmented data block; Merge the vector data corresponding to n1*n2 data blocks to obtain a two-dimensional data matrix of n1n2× p1p2c; Generate a position coding vector corresponding to each data block according to the position in the target ultrasonic breast image, and add the position coding vector to the two-dimensional data matrix to obtain a data matrix to be processed; The data matrix to be processed is input into a pre-trained image classification network to obtain the lesion property classification corresponding to the target ultrasonic breast image.

2. The method according to claim 1, characterized in that The image classification network includes a multi-head attention module, a feedforward neural module and a multi-layer fully connected classification module; The multi-head attention module includes three fully connected networks, an activation function layer and a multi-dimensional logistic regression layer, the input of each fully connected network is the data matrix to be processed, and the output is feature data of a preset dimension; wherein, after the feature data output by the preset two fully connected networks are multiplied and divided by a preset scale factor, the logistic regression result is calculated by the multi-dimensional logistic regression layer; after the logistic regression result is multiplied by the feature data of another fully connected network, the output result of the multi-head attention module is obtained; the other fully connected network is a fully connected network among the three fully connected networks that is different from the preset two fully connected networks; The feedforward neural module includes a fully connected network, a linear rectification activation function connected to the fully connected network, and layer normalization; the output result of the multi-head attention module is passed through the fully connected network, the linear rectification activation function connected to the fully connected network, and layer normalization to obtain the output result of the feedforward neural module; After receiving the output result of the feedforward neural module, the multi-layer fully connected classification module processes the data through a fully connected layer; the processed data is layer normalized to obtain the lesion property classification.

3. The method according to claim 1, characterized in that If the target ultrasound breast image is an image in the target ultrasound breast video, the method further includes: After obtaining the lesion property classification according to each frame of the target ultrasound breast image, the lesion property classification corresponding to the target ultrasound breast video is determined according to the lesion property classification obtained from each frame of the target ultrasound breast image in the target ultrasound breast video.

4. The method according to claim 3, characterized in that The step of determining the lesion property classification corresponding to the target ultrasound breast video according to the lesion property classification obtained from each frame of the target ultrasound breast image in the target ultrasound breast video comprises: If there is no image classified as malignant in the target ultrasound breast images of each frame of the target ultrasound breast video, then counting the lesion property classification with the largest number among the lesion property classifications corresponding to the target ultrasound breast images of each frame, and determining the lesion property classification obtained by counting as the lesion property classification corresponding to the target ultrasound breast video; If there is an image whose lesion property is classified as malignant in each frame of the target ultrasound breast image of the target ultrasound breast video, the lesion property classification corresponding to the target ultrasound breast video is determined to be malignant.

5. The method according to any one of claims 1 to 4, characterized in that: The lesion property classification includes: benign type and malignant type; or, includes at least one of benign type, malignant type, inflammatory type, adenosis type, hyperplasia type, ductal dilatation type, early invasive cancer, invasive cancer, non-invasive cancer, lobular adenocarcinoma, ductal adenocarcinoma, medullary carcinoma, scirrhous cancer, simple cancer, carcinoma in situ, early cancer, undifferentiated cancer, poorly differentiated cancer, moderately differentiated cancer, and well-differentiated cancer.

6. The method according to claim 2, characterized in that There are multiple multi-head attention modules and multiple feedforward neural modules.

7. The method according to any one of claims 1 to 4, characterized in that: The target ultrasonic breast image is a whole ultrasonic breast image or a breast lesion area image.

8. An ultrasonic breast lesion classification device, characterized in that: The device comprises: An information acquisition unit, used for acquiring target ultrasonic breast information to be classified, wherein the target ultrasonic breast information is a target ultrasonic breast image or a target ultrasonic breast video, wherein the target ultrasonic breast video includes at least two frames of target ultrasonic breast images; An image segmentation unit, for each frame of the target ultrasound breast image, segments the target ultrasound breast image into n1*n2 data blocks; wherein n1 is the number of data blocks segmented in the image height direction, n2 is the number of data blocks segmented in the image width direction, and n1 and n2 are positive integers; A data conversion unit is used to convert each data block into vector data of p1*p2*c dimensions; wherein n1= H / p1, n2= W / p2; H is the height of the input image, W is the width of the input image, p1 is the height of the data block after segmentation, and p2 is the width of the data block after segmentation; A vector merging unit is used to merge the vector data corresponding to n1*n2 data blocks to obtain a two-dimensional data matrix of n1n2× p1p2c; A matrix generating unit, used for generating a position coding vector corresponding to each data block according to the position of the data block in the target ultrasonic breast image, and adding the position coding vector to the two-dimensional data matrix to obtain a data matrix to be processed; The lesion classification unit is used to input the data matrix to be processed into a pre-trained image classification network to obtain the lesion property classification corresponding to the target ultrasonic breast image.

9. An ultrasonic breast lesion classification device, characterized in that: The device comprises a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement the ultrasonic breast lesion classification method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and when the program is executed by the processor, it is used to implement the ultrasonic breast lesion classification method according to any one of claims 1 to 7.

Images