Duty frame extraction method, medical video classification method, device and medium

Abstract

The application provides a responsibility frame extraction method, a medical video classification method, an electronic device and a storage medium, and the method comprises the following steps: acquiring a region of interest image corresponding to each frame of medical image by using a target detection neural network model; acquiring a feature matrix of each frame of region of interest image by using a skeleton network of a static medical image classification neural network model; extracting malignant responsibility frames according to the feature matrix of each frame of region of interest image until a malignant feature entropy corresponding to a malignant responsibility frame set composed of all the malignant responsibility frames reaches a minimum value; and / or extracting benign responsibility frames according to the feature matrix of each frame of region of interest image until a benign feature entropy corresponding to a benign responsibility frame set composed of all the benign responsibility frames reaches a minimum value. The application can extract the malignant responsibility frames and / or the benign responsibility frames which contribute different important features for medical video classification from the medical video.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a responsibility frame extraction method, a medical video classification method, an electronic device and a storage medium. BACKGROUND

[0002] Ultrasound is a common means of medical imaging examination for diseases, which can be used for the diagnosis of diseases of various tissues and organs, and has the characteristics of wide application of diseases and low cost compared with CT, MRI and other large-scale imaging equipment. At the same time, the portability of ultrasound hardware is constantly upgraded, and the product form of palm ultrasound realizes the unity of function and portability, which is suitable for primary disease screening scenes. However, because the ultrasound image has high granularity, there are a large number of speckle noise, artifacts, attenuation and other problems, and the ultrasound diagnosis is difficult to be standardized and standardized, and is highly dependent on the clinical experience of ultrasound physicians. Primary hospitals, community hospitals, township clinics and other primary medical institutions lack experienced ultrasound physicians, and it is difficult to accurately judge the benign and malignant of ultrasound videos.

[0003] In clinical practice, when the ultrasound physician makes a preliminary diagnosis, reviews, and conveys diagnosis suggestions to the attending physician, the responsibility frame (an image picture with obvious benign and malignant indications) extracted from the video by the physician will be used. An ideal artificial intelligence ultrasound system can automatically give the responsibility frame on which the video is judged to be benign or malignant, reducing the workload of the physician and assisting the physician in diagnosis. Therefore, how to extract the responsibility frame from the video is particularly important.

[0004] It should be noted that the information disclosed in the background section of the present application is only intended to deepen the understanding of the general background of the present application, and should not be regarded as acknowledging or implying in any form that the information constitutes prior art known to those skilled in the art. SUMMARY

[0005] The purpose of the present application is to provide a responsibility frame extraction method, a medical video classification method, an electronic device and a storage medium, which can automatically extract responsibility frames that contribute different important features to the classification of medical videos (such as the classification of benign and malignant nodule videos) from medical videos, so as to improve the efficiency and accuracy of medical video classification (such as the classification of benign and malignant nodule videos).

[0006] To achieve the above-mentioned purpose, the present application provides a responsibility frame extraction method, comprising:

[0007] extracting the region of interest of each frame of medical image in the acquired medical video by using a target detection neural network model to obtain the region of interest image corresponding to each frame of medical image;

[0008] extracting features of each frame of region of interest image by using the skeleton network of the static medical image classification neural network model to obtain the feature matrix of each frame of region of interest image;

[0009] Based on the feature matrix of the region of interest image in each frame, malicious responsibility frames are extracted until the malicious feature entropy corresponding to the set of malicious responsibility frames constituted by all the malicious responsibility frames reaches its minimum value; and / or

[0010] Based on the feature matrix of the region of interest image of each frame, benign responsibility frames are extracted until the benign feature entropy corresponding to the benign responsibility frame set composed of all the benign responsibility frames reaches the minimum value.

[0011] Optionally, the step of using a target detection neural network model to extract the region of interest (ROI) for each frame of the acquired medical video to obtain the ROI image corresponding to each frame of the medical video includes:

[0012] An object detection neural network model is used to extract the region of interest (ROI) from each frame of the acquired medical video to obtain the location information of the ROI corresponding to each frame of the medical video.

[0013] Based on the location information of the region of interest corresponding to each frame of medical images, the corresponding region is cropped from each frame of medical images to obtain the region of interest image corresponding to each frame of medical images.

[0014] Optionally, before using the backbone network of a static medical image classification neural network model to extract features from each frame of the region of interest image, the method further includes:

[0015] For each frame of the region of interest image:

[0016] The larger of the width and height dimensions of the region of interest image is used as the target side length.

[0017] The region of interest image is padded to adjust the smaller of the width and height dimensions of the region of interest image to the target side length dimension;

[0018] The region of interest image, adjusted to the target side length, is magnified or reduced to adjust its size to a preset size.

[0019] Optionally, the step of extracting malicious responsibility frames based on the feature matrix of each frame's region of interest image until the malicious feature entropy corresponding to the set of malicious responsibility frames constituted by all the malicious responsibility frames reaches its minimum value includes:

[0020] For each frame of the region of interest image, the malignant feature matrix of the region of interest image is obtained based on the feature matrix of the region of interest image and the difference between the malignant feature weight parameters and the benign feature weight parameters corresponding to the static medical image classification neural network model.

[0021] Based on the malignant feature matrix of each frame's region of interest image, malignant responsibility frames are extracted until the malignant feature entropy corresponding to the set of all such malignant responsibility frames reaches its minimum value; and / or

[0022] The step of extracting benign responsibility frames based on the feature matrix of each frame's region of interest image, until the benign feature entropy corresponding to the set of benign responsibility frames constituted by all the benign responsibility frames reaches its minimum value, includes:

[0023] For each frame of the region of interest image, the benign feature matrix of the region of interest image is obtained based on the feature matrix of the region of interest image and the difference between the benign feature weight parameters and the malignant feature weight parameters corresponding to the static medical image classification neural network model.

[0024] Based on the benign feature matrix of the region of interest image in each frame, benign responsibility frames are extracted until the benign feature entropy corresponding to the set of benign responsibility frames composed of all the benign responsibility frames reaches the minimum value.

[0025] Optionally, obtaining the malignant feature matrix of the region of interest image based on the feature matrix of the region of interest image and the difference between the malignant feature weight parameters and the benign feature weight parameters corresponding to the static medical image classification neural network model includes:

[0026] The malignant feature matrix of the region of interest image is obtained according to the following formula:

[0027]

[0028]

[0029] In the formula, This represents the feature value of the j-th feature dimension in the feature matrix of the i-th frame's region of interest image. This represents the malignancy feature weight of the j-th feature dimension corresponding to the static medical image classification neural network model. This represents the benign feature weight of the j-th feature dimension corresponding to the static medical image classification neural network model. [FM] represents the malignant feature value of the j-th feature dimension in the malignant feature matrix of the i-th frame's region of interest image. iRepresents the malignant feature matrix of the region of interest image in the i-th frame; and / or

[0030] The step of obtaining the benign feature matrix of the region of interest image based on the feature matrix of the region of interest image and the difference between the benign feature weight parameters and the malignant feature weight parameters corresponding to the static medical image classification neural network model includes:

[0031] The benign feature matrix of the region of interest image is obtained according to the following formula:

[0032]

[0033]

[0034] In the formula, This represents the feature value of the j-th feature dimension in the feature matrix of the i-th frame's region of interest image. This represents the benign feature weight of the j-th feature dimension corresponding to the static medical image classification neural network model. This represents the malignancy feature weight of the j-th feature dimension corresponding to the static medical image classification neural network model. Let [FB] represent the benign feature value of the j-th feature dimension in the benign feature matrix of the i-th frame's region of interest image. i Let represent the benign feature matrix of the region of interest image in the i-th frame.

[0035] Optionally, the step of extracting malicious responsibility frames based on the malicious feature matrix of each frame's region of interest image, until the malicious feature entropy corresponding to the set of malicious responsibility frames constituted by all the malicious responsibility frames reaches its minimum value, includes:

[0036] For each frame of the region of interest image, the malignant feature values ​​of all feature dimensions in the malignant feature matrix of the region of interest image are summed to obtain the total malignant feature value of the region of interest image.

[0037] Based on the total malignant feature value of the region of interest image in each frame, malignant responsibility frames are extracted until the malignant feature entropy corresponding to the set of malignant responsibility frames constituted by all the malignant responsibility frames reaches the minimum value; and / or

[0038] The step of extracting benign responsibility frames based on the benign feature matrix of each frame's region of interest image, until the benign feature entropy corresponding to the set of benign responsibility frames constituted by all the benign responsibility frames reaches its minimum value, includes:

[0039] For each frame of the region of interest image, the benign feature values ​​of all feature dimensions in the benign feature matrix of the region of interest image are summed to obtain the total benign feature value of the region of interest image.

[0040] Based on the total benign feature value of the region of interest image in each frame, benign responsibility frames are extracted until the benign feature entropy corresponding to the set of benign responsibility frames composed of all the benign responsibility frames reaches the minimum value.

[0041] Optionally, the step of extracting malicious responsibility frames based on the total malicious feature values ​​of the region of interest images in each frame, until the malicious feature entropy corresponding to the set of malicious responsibility frames composed of all the malicious responsibility frames reaches its minimum value, includes:

[0042] Step A1: Sort the total malignant feature values ​​of the region of interest images in each frame, and determine the region of interest image with the largest total malignant feature value as the malignant responsible frame;

[0043] Step A2: Combine all the malicious responsibility frames and each non-malicious responsibility frame into a first image set, and calculate the total malicious feature value of each first image set. The total malicious feature value of the first image set is equal to the sum of the malicious feature values ​​of all feature dimensions in the malicious feature matrix obtained by max pooling the malicious feature matrix of the region of interest images of all frames in the first image set. The non-malicious responsibility frames are the region of interest images that have not been identified as malicious responsibility frames.

[0044] Step A3: Determine whether the malignant feature entropy corresponding to the first image set with the smallest total malignant feature value is less than the malignant feature entropy corresponding to the set of malignant responsibility frames composed of all malignant responsibility frames;

[0045] If yes, proceed to step A4; otherwise, proceed to step A5.

[0046] Step A4: Identify all frames in the first image set with the smallest total malignant feature value as the frames responsible for the malignant attack, and return to step A2.

[0047] Step A5: End the extraction of malicious responsibility frames; and / or

[0048] The step of extracting benign responsibility frames based on the total benign feature values ​​of the region of interest images in each frame, until the benign feature entropy corresponding to the set of benign responsibility frames constituted by all the benign responsibility frames reaches its minimum value, includes:

[0049] Step B1: Sort the total benign feature values ​​of the region of interest images in each frame, and determine the region of interest image with the largest total benign feature value as the benign responsibility frame;

[0050] Step B2: Combine all the benign responsibility frames and each non-benign responsibility frame into a second image set, and calculate the total benign feature value for each second image set. The total benign feature value of the second image set is equal to the sum of the benign feature values ​​of all feature dimensions in the benign feature matrix obtained by max pooling the benign feature matrix of all frames in the second image set. The non-benign responsibility frames are the region of interest images that have not yet been determined as benign responsibility frames.

[0051] Step B3: Determine whether the benign feature entropy corresponding to the second image set with the smallest total benign feature value is less than the benign feature entropy corresponding to the benign responsibility frame set composed of all benign responsibility frames;

[0052] If yes, proceed to step B4; otherwise, proceed to step B5.

[0053] Step B4: Identify all region-of-interest images in the second image set with the smallest total benign feature value as benign responsibility frames, and return to step B2.

[0054] Step B5: End the extraction of benign responsibility frames.

[0055] Optionally, the malignant feature entropy of the image set can be calculated using the following formula:

[0056] H1(A) = -p1(A) × log2 p1(A)

[0057]

[0058] In the formula, H1(A) represents the malignant feature entropy of image set X, MScoreA represents the total malignant feature value of image set A, BScoreA represents the total benign feature value of image set A; and / or

[0059] Calculate the benign feature entropy of the image set using the following formula:

[0060] H0(A)=-p0(A)×log2 p0(A)

[0061]

[0062] In the formula, H0(A) represents the benign feature entropy of image set A, MScoreA represents the total malignant feature value of image set A, and BScoreA represents the total benign feature value of image set A.

[0063] To achieve the above objectives, the present invention also provides a medical video classification method, comprising:

[0064] Using the responsibility frame extraction method described above, malignant and / or benign responsibility frames are extracted from the acquired medical video to be classified; and

[0065] The medical video to be classified is classified based on the feature matrix of the extracted malicious and / or benign responsibility frames.

[0066] Optionally, classifying the medical video to be classified based on the feature matrix of the extracted malicious and / or benign responsibility frames includes:

[0067] Max pooling is performed on the feature matrices of the extracted malicious and / or benign responsibility frames to obtain the feature matrices of the responsibility frame set.

[0068] The medical videos to be classified are classified based on the feature matrix of the set of responsible frames.

[0069] Optionally, after extracting the malicious responsibility frames and / or benign responsibility frames from the acquired medical videos to be classified, the medical video classification method further includes:

[0070] Display the extracted malicious and / or benign liability frames, and receive user confirmation or deletion operations for the extracted malicious and / or benign liability frames, as well as receive user operations to add liability frames.

[0071] To achieve the above objectives, the present invention also provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements the responsibility frame extraction method or medical video classification method described above.

[0072] To address the aforementioned technical problems, the present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the responsibility frame extraction method or medical video classification method described above.

[0073] Compared with existing technologies, the responsibility frame extraction method, medical video classification method, electronic device, and storage medium provided by this invention have the following advantages:

[0074] (1) The responsibility frame extraction method provided by this invention first uses a target detection neural network model to extract the region of interest (ROI) of each frame of medical image in the acquired medical video to obtain the ROI image corresponding to each frame of medical image; then, it uses the skeleton network of a static medical image classification neural network model to extract features from each ROI image to obtain the feature matrix of each ROI image; finally, based on the feature matrix of each ROI image, it extracts malignant responsibility frames until the malignant feature entropy corresponding to the set of malignant responsibility frames formed by all the malignant responsibility frames reaches the minimum value; and / or based on the feature matrix of each ROI image, it extracts benign responsibility frames until the benign feature entropy corresponding to the set of benign responsibility frames formed by all the benign responsibility frames reaches the minimum value. Therefore, this invention can automatically extract malignant responsibility frames and / or benign responsibility frames from medical videos that contribute different important features to medical video classification (e.g., classification of benign and malignant nodule videos), thereby improving the efficiency and accuracy of medical video classification (e.g., classification of benign and malignant nodule videos) and better assisting doctors in improving the efficiency and accuracy of diagnosis. Furthermore, this invention can automatically determine the required number of malignant and / or benign responsibility frames, further improving the efficiency and accuracy of medical video classification (e.g., classification of benign and malignant nodule videos). Additionally, the responsibility frame extraction method provided by this invention first uses a target detection neural network model to extract the region of interest (ROI) image from each frame of the acquired medical video, and then extracts malignant and / or benign responsibility frames based on the feature matrix of each ROI image. This effectively reduces image noise interference during the extraction process, further improving the efficiency and accuracy of malignant and / or benign responsibility frame extraction.

[0075] (2) The medical video classification method provided by this invention first extracts malignant and / or benign responsibility frames from the acquired medical video to be classified using the responsibility frame extraction method described above, and then classifies the medical video to be classified based on the feature matrix of the extracted malignant and / or benign responsibility frames. Therefore, the medical video classification method provided by this invention, by classifying the acquired medical video to be classified based on the malignant and / or benign responsibility frames extracted using the responsibility frame extraction method described above, can effectively reduce the interference of noise frames in the medical video, effectively improve the accuracy of medical video classification, and thus effectively reduce the misdiagnosis rate of doctors.

[0076] (3) Since the electronic device and readable storage medium provided by the present invention belong to the same inventive concept as the responsibility frame extraction method or the medical video classification method described above, the electronic device and readable storage medium provided by the present invention have all the advantages of the responsibility frame extraction method or the medical video classification method described above. Therefore, the beneficial effects of the electronic device and readable storage medium provided by the present invention will not be described in detail here. Attached Figure Description

[0077] Figure 1 A schematic diagram of a mass-type breast cancer provided as a specific example of the present invention;

[0078] Figure 2 A schematic diagram of non-mass breast cancer provided as a specific example of the present invention;

[0079] Figure 3 A flowchart illustrating a responsibility frame extraction method provided in one embodiment of the present invention;

[0080] Figure 4 A schematic diagram of a medical image provided as a specific example of the present invention;

[0081] Figure 5 From Figure 4 A schematic diagram of the region of interest extracted from the image;

[0082] Figure 6 A schematic diagram illustrating the relationship between the feature entropy of a set of responsible frame images and the number of responsible frames, provided in one embodiment of the present invention;

[0083] Figure 7 This is a schematic diagram illustrating the specific process of extracting malicious responsibility frames according to an embodiment of the present invention;

[0084] Figure 8 A schematic diagram illustrating the specific process of extracting benign responsibility frames according to an embodiment of the present invention;

[0085] Figure 9 This is a flowchart illustrating a medical video classification method according to an embodiment of the present invention;

[0086] Figure 10 This is a schematic diagram illustrating video classification using a random forest classifier, as provided in one embodiment of the present invention.

[0087] Figure 11 A schematic diagram of an adjustment responsibility frame provided according to an embodiment of the present invention;

[0088] Figure 12 This is a block diagram of an electronic device provided according to an embodiment of the present invention.

[0089] The reference numerals in the attached figures are as follows:

[0090] Processor-101; Communication interface-102; Memory-103; Communication bus-104. Detailed Implementation

[0091] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, further illustrates the responsibility frame extraction method, medical video classification method, electronic device, and storage medium proposed in this invention. The advantages and features of this invention will become clearer from the following description. It should be noted that the drawings are in a very simplified form and use non-precise proportions, used only to facilitate and clearly illustrate the embodiments of this invention. Please refer to the drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for illustrative purposes and to enable those skilled in the art to understand and read them, and are not intended to limit the implementation conditions of this invention. Any modifications to the structure, changes in proportions, or adjustments to the size, provided that the effects and objectives achieved by this invention are the same or similar, should still fall within the scope of the technical content disclosed in this invention.

[0092] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0093] Furthermore, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0094] Before introducing the responsibility frame extraction method provided by this invention, we will first describe the existing responsibility frame extraction methods so that those skilled in the art can better understand the advantages of the responsibility frame extraction method provided by this invention.

[0095] Existing methods for extracting responsibility frames include those based on grayscale values, target tracking algorithms, and attention mechanisms.

[0096] The responsibility frame identification method based on grayscale values ​​converts each frame in the ultrasound video into a grayscale image, performs a linear transformation on the grayscale image, and then sets a threshold based on the grayscale value feature map after the linear transformation to select the responsibility frame. However, the accuracy of the responsibility frame identification method based on grayscale values ​​is not good, especially when the grayscale value change amplitude caused by small lesions is small, it cannot effectively identify the responsibility frame.

[0097] The responsibility frame identification method based on target tracking algorithm uses a target detection network to locate lesions (such as nodules in the thyroid or breast) in ultrasound video and identifies whether the current frame is the responsibility frame based on the characteristics of the lesion. However, although the responsibility frame identification method based on target tracking algorithm can identify lesions such as nodules in the thyroid or breast, it is still not a reliable method for identifying responsibility frames. Figure 1 Clear lesions such as masses can be effectively identified, but for... Figure 2 The non-mass lesions shown present difficulties in labeling and identifying the lesion areas, which in turn makes it difficult to identify the responsible frame.

[0098] Attention-based responsibility frame identification methods rely on contextual information to determine the importance of the current frame. However, for a frame sequence consisting of consecutive frames, the attention scores given may be very close (these frames themselves and the images of the preceding and following frames are similar), making it difficult to distinguish and find the most important key frames.

[0099] It is evident that existing methods for identifying responsibility frames have limitations in terms of applicability, the distinguishability of responsibility frames, and the degree of automation. In particular, grayscale-based and target-tracking-based methods perform poorly in some lesion types, including minute lesions and non-mass lesions. Furthermore, existing methods cannot avoid identifying multiple highly similar responsibility frames, and they lack a mechanism to automatically determine the required number of responsibility frames based on the complexity of the ultrasound video content. This leads to issues such as the loss of video information due to either an excessive number of identified responsibility frames or an insufficient number of manually defined responsibility frames.

[0100] The core idea of ​​this invention is to provide a method for extracting responsibility frames, a method for classifying medical videos, an electronic device, and a storage medium, which can automatically extract responsibility frames from medical videos that contribute different important features to the classification of medical videos (e.g., the classification of benign and malignant nodule videos), thereby improving the efficiency and accuracy of medical video classification (e.g., the classification of benign and malignant nodule videos) and thus solving one or more of the above problems.

[0101] It should be noted that the responsibility frame extraction method and medical video classification method of the present invention can be applied to the electronic device of the present invention. The electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a mobile phone, a tablet computer, or other hardware device with various operating systems.

[0102] To achieve the above-mentioned goals, this invention provides a method for extracting responsibility frames. Please refer to [the relevant documentation]. Figure 3 The flowchart illustrating a responsibility frame extraction method provided by an embodiment of the present invention is shown below. Figure 3 As shown, the responsibility frame extraction method includes the following steps:

[0103] Step S110: Use an object detection neural network model to extract the region of interest for each frame of medical image in the acquired medical video, so as to obtain the region of interest image corresponding to each frame of medical image.

[0104] Step S120: Use the skeleton network of the static medical image classification neural network model to extract features from each frame of the region of interest image to obtain the feature matrix of each frame of the region of interest image.

[0105] Step S130: Extract malicious responsibility frames based on the feature matrix of each frame's region of interest image until the malicious feature entropy corresponding to the set of malicious responsibility frames composed of all the malicious responsibility frames reaches the minimum value; and / or extract benign responsibility frames based on the feature matrix of each frame's region of interest image until the benign feature entropy corresponding to the set of benign responsibility frames composed of all the benign responsibility frames reaches the minimum value.

[0106] Therefore, the responsibility frame extraction method provided by this invention can automatically extract malignant and / or benign responsibility frames from medical videos that contribute different important features to medical video classification (e.g., classification of benign and malignant nodule videos), thereby improving the efficiency and accuracy of medical video classification (e.g., classification of benign and malignant nodule videos) and better assisting doctors in improving the efficiency and accuracy of diagnosis. Furthermore, this invention can automatically determine the required number of malignant and / or benign responsibility frames, further improving the efficiency and accuracy of medical video classification (e.g., classification of benign and malignant nodule videos).

[0107] Please refer to Figure 4 and Figure 5 ,in Figure 4 This is a schematic diagram of a medical image provided as a specific example of the present invention. Figure 5 The illustration shows the origin from Figure 4 A schematic diagram of the region of interest extracted from the image. (See diagram below.) Figure 4 and Figure 5 As shown, due to differences in equipment and inspection modes, the style of the information prompt bar outside the window varies. Therefore, the responsibility frame extraction method provided by this invention first uses a target detection neural network model to extract the region of interest image from each frame of the acquired medical video, and then extracts malignant responsibility frames and / or benign responsibility frames based on the feature matrix of each frame of the region of interest image. This can effectively reduce the interference of image noise in the process of extracting malignant responsibility frames and / or benign responsibility frames, and further improve the efficiency and accuracy of extracting malignant responsibility frames and / or benign responsibility frames.

[0108] It should be noted that, as those skilled in the art will understand, the medical video can be ultrasound scan video (e.g., scan data for breast cancer, thyroid nodules, etc.), or it can be medical video acquired by other medical imaging equipment, such as medical video acquired by endoscopy. Furthermore, it should be noted that, as those skilled in the art will understand, the format of the medical video is frame count × width × height × number of channels (RGB channels, value 3), and the imported medical video can be stored in a matrix format on the computer.

[0109] It should be noted that when the malignant feature entropy increases, it means that the uncertainty of the prediction result begins to rise. Therefore, when the malignant feature entropy no longer decreases, it is necessary to stop extracting new malignant responsibility frames. Thus, this invention determines whether to stop extracting malignant responsibility frames based on whether the malignant feature entropy has reached its minimum value, and can automatically extract the required number of malignant responsibility frames that can contribute important features to the classification of the acquired medical video. Please refer to [reference needed]. Figure 6 This diagram illustrates the relationship between the feature entropy of the set of responsible frame images and the number of responsible frames provided in one embodiment of the present invention. Figure 6 As shown, when the number of image frames in the malicious responsibility frame set is less than 5, the malicious feature entropy of the malicious responsibility frame set continues to decrease. When the number of image frames in the malicious responsibility frame set exceeds 5, the malicious feature entropy begins to increase. This means that the uncertainty of the prediction result increases. Therefore, for Figure 6 In the example shown, when the number of malicious responsibility frames in the malicious responsibility frame set reaches 5 frames, the malicious feature entropy of the malicious responsibility frame set reaches its minimum value. Therefore, after the number of malicious responsibility frames reaches 5 frames, it is not necessary to continue extracting new malicious responsibility frames.

[0110] Similarly, when the benign feature entropy increases, it means that the uncertainty of the prediction result begins to rise. Therefore, when the benign feature entropy no longer decreases, it is necessary to stop extracting new benign responsibility frames. Thus, this invention determines whether to stop extracting benign responsibility frames based on whether the benign feature entropy has reached its minimum value, and can automatically extract the required number of benign responsibility frames that can contribute important features to the classification of medical videos based on the content of the acquired medical videos.

[0111] In one exemplary embodiment, the step of using a target detection neural network model to extract the region of interest (ROI) for each frame of the acquired medical video to obtain the ROI image corresponding to each frame of the medical video includes:

[0112] An object detection neural network model is used to extract the region of interest (ROI) from each frame of the acquired medical video to obtain the location information of the ROI corresponding to each frame of the medical video.

[0113] Based on the location information of the region of interest corresponding to each frame of medical images, the corresponding region is cropped from each frame of medical images to obtain the region of interest image corresponding to each frame of medical images.

[0114] Therefore, by employing a target detection neural network model, the location information of the region of interest (i.e., ultrasound window) in each frame of a medical image can be accurately obtained (this information can be represented by the location information of the upper left and lower right corners of the detected region of interest border, or the length, width, and center coordinates of the detected region of interest border). Thus, for each frame of a medical image, the corresponding region of interest image can be cropped from the medical image based on the location information of the region of interest.

[0115] Specifically, the object detection neural network model can be a ResNet50 neural network model. Because ResNet uses skip connections (or shortcuts), it directly passes the activation values ​​of a certain network layer to deeper layers. Furthermore, skip connections only transmit data; through skip connections, the signal can be transmitted without attenuation during backpropagation, without worrying about gradient changes, and can effectively transmit effective gradients to the upper layer. Therefore, skip connections can effectively alleviate the gradient vanishing problem caused by deepening network layers. By stacking residual blocks, very deep network models can be constructed, enabling effective training even at deep network layers. It should be noted that, as those skilled in the art will understand, the object detection neural network model can also use other neural network models besides ResNet50, and this invention does not limit this.

[0116] In one exemplary embodiment, the method further includes, prior to feature extraction of each frame of the region of interest image using the backbone network of a static medical image classification neural network model:

[0117] For each frame of the region of interest image:

[0118] The larger of the width and height dimensions of the region of interest image is used as the target side length.

[0119] The region of interest image is padded to adjust the smaller of the width and height dimensions of the region of interest image to the target side length dimension;

[0120] The region of interest image, adjusted to the target side length, is magnified or reduced to adjust its size to a preset size.

[0121] Since the static medical image classification neural network model requires images of uniform size as input, the size of the region of interest (ROI) image needs to be adjusted to a preset size before the backbone network of the static image classification neural network model extracts features from each frame of the ROI image. The preset size can be set according to specific circumstances, and this invention does not limit it. Preferably, in the preset size, the height and width of the image are the same, that is, the ROI image after adjustment to the preset size is a square image, for example, the preset size is 448*448. Therefore, by setting the height and width to be the same in the preset size, it is easier to adjust the size of the ROI image to the preset size. Specifically, a "zero-pixel" filling method can be used to fill the ROI image to adjust the width and height of the ROI image to be the same. It should be noted that, as those skilled in the art will understand, since the object detection neural network model also requires images of a uniform size as input, before using the object detection neural network model to extract the region of interest from each frame of the acquired medical video, it is also necessary to adjust the size of each frame of medical images to a target size to meet the input requirements of the object detection neural network model. For details on how to adjust the size of each frame of medical images to the target size, please refer to the relevant content on the size adjustment of the region of interest image above, which will not be repeated here.

[0122] In one exemplary embodiment, the static image classification neural network model includes a backbone network for feature extraction and a classification network for classification. The backbone network can be selected from different convolutional neural networks, such as MobileNet, DenseNet121, and Xception. Further details about MobileNet, DenseNet121, and Xception networks can be found in existing technologies and will not be elaborated upon here. The classification network includes at least one fully connected layer, which performs nonlinear mapping regression on the features extracted by the classification network to obtain the classification result. Specifically, a medical video can be viewed as a collection of medical images, with the information of the entire medical video distributed across each frame of the medical image. The contribution of the region of interest image corresponding to each frame of the medical image in each feature dimension is represented by its feature matrix, where the number of feature dimensions is determined by the backbone network, and each feature dimension represents a deep-space image feature (e.g., the feature of a malignant or benign nodule). The feature matrix of the region of interest (ROI) corresponding to each frame of the acquired medical video can be obtained by performing multiple convolution operations and global pooling on the backbone network of the static image classification neural network model. The feature matrix of the ROI corresponding to each frame of the medical video can be represented by a 1×k matrix, where k represents the feature dimension, determined by the structure of the static image classification neural network model. Specifically, the feature matrix X corresponding to the i-th frame of the medical video... i It can be represented as:

[0123]

[0124] In a preferred embodiment, the skeleton network of the static image classification neural network uses parallel processing to simultaneously extract features from the regions of interest corresponding to multiple frames of medical images in the acquired medical video, so as to simultaneously obtain the feature matrix of the regions of interest corresponding to multiple frames of medical images.

[0125] Therefore, by employing parallel processing to simultaneously extract features from multiple frames of region-of-interest (ROI) images, and obtaining feature matrices for multiple ROI images simultaneously (one ROI image corresponds to one feature matrix), the extraction efficiency of the responsible frame extraction method provided by this invention can be further improved. It should be noted that, as those skilled in the art will understand, the total number of ROI frames that can be processed in parallel each time is determined by the computing power of the computer's GPU; the stronger the computing power of the computer's GPU, the more ROI frames can be processed in parallel each time.

[0126] In one exemplary implementation, the step of extracting malicious responsibility frames based on the feature matrix of each frame's region of interest image, until the malicious feature entropy corresponding to the set of malicious responsibility frames constituted by all the malicious responsibility frames reaches its minimum value, includes:

[0127] For each frame of the region of interest image, the malignant feature matrix of the region of interest image is obtained based on the feature matrix of the region of interest image and the difference between the malignant feature weight parameters and the benign feature weight parameters corresponding to the static medical image classification neural network model.

[0128] Based on the malignant feature matrix of each frame's region of interest image, malignant responsibility frames are extracted until the malignant feature entropy corresponding to the set of malignant responsibility frames composed of all the malignant responsibility frames reaches the minimum value.

[0129] Specifically, in the static medical image classification neural network model, the benign or malignant determination of each frame's region of interest image is based on the feature matrix of the region of interest image, and the static medical image classification neural network model predicts the output probability Y. pred It can be expressed as:

[0130]

[0131] In equation (1), Y0 represents the probability that the region of interest image belongs to the benign category, Y1 represents the probability that the region of interest image belongs to the malignant category, W1 represents the malignant feature weight parameter corresponding to the static medical image classification neural network model, W0 represents the benign feature weight parameter corresponding to the static medical image classification neural network model, and B0 and B1 represent the bias parameters corresponding to the static medical image classification neural network model.

[0132] As shown in equation (1) above, the probability Y1 of the region of interest image belonging to the malignant category is determined only by the relative difference between the malignant feature weight parameter and the benign feature weight parameter and the feature matrix X of the region of interest image. Therefore, this invention obtains the malignant feature matrix of the region of interest image based on the feature matrix of the region of interest image and the difference between the malignant feature weight parameter and the benign feature weight parameter corresponding to the static medical image classification neural network model. Then, based on the malignant feature matrix of each frame of the region of interest image, the malignant responsibility frames are extracted until the malignant feature entropy corresponding to the set of malignant responsibility frames composed of all the malignant responsibility frames reaches the minimum value, thereby accurately extracting the malignant responsibility frames with a large amount of malignant contribution information. It should be noted that, as those skilled in the art can understand, the malignant feature weight parameter W1 is a matrix with k malignant feature weights, and the benign feature weight parameter W0 is a matrix with k benign feature weights, that is, each feature dimension corresponds to one malignant feature weight and one benign feature weight.

[0133] In one exemplary embodiment, obtaining the malignant feature matrix of the region of interest image based on the feature matrix of the region of interest image and the difference between the malignant feature weight parameters and the benign feature weight parameters corresponding to the static medical image classification neural network model includes:

[0134] The malignant feature matrix of the region of interest image is obtained according to the following formulas (2) and (3):

[0135]

[0136]

[0137] In equation (2), [FM] i In equation (3), the malignant feature matrix of the region of interest image in the i-th frame is represented. This represents the feature value of the j-th feature dimension in the feature matrix of the i-th frame's region of interest image. This represents the malignancy feature weight of the j-th feature dimension corresponding to the static medical image classification neural network model. This represents the benign feature weight of the j-th feature dimension corresponding to the static medical image classification neural network model. This represents the malignant feature value of the j-th feature dimension in the malignant feature matrix of the i-th frame region of interest image.

[0138] It should be noted that, as those skilled in the art will understand, Indicates from 0 and Take the larger one, that is, if If it is greater than 0, then Pick like If less than 0, then Take 0.

[0139] Furthermore, in an exemplary embodiment, the step of extracting malicious responsibility frames based on the malicious feature matrix of each frame's region of interest image, until the malicious feature entropy corresponding to the set of malicious responsibility frames constituted by all the malicious responsibility frames reaches its minimum value, includes:

[0140] For each frame of the region of interest image, the malignant feature values ​​of all feature dimensions in the malignant feature matrix of the region of interest image are summed to obtain the total malignant feature value of the region of interest image.

[0141] Based on the total malignant feature value of the region of interest image in each frame, malignant responsibility frames are extracted until the malignant feature entropy corresponding to the set of malignant responsibility frames composed of all the malignant responsibility frames reaches the minimum value.

[0142] Specifically, combining formulas (2) and (3) above, the total malignant feature value of the i-th frame region of interest image can be expressed by the following formula:

[0143]

[0144] Therefore, by extracting malicious responsibility frames based on the total malicious feature values ​​of the region of interest images in each frame, not only can the extraction efficiency of malicious responsibility frames be further improved, but also the extraction of malicious responsibility frames with similar features can be effectively prevented.

[0145] Please continue to refer to this. Figure 7 The diagram illustrates a specific process for extracting malicious responsibility frames according to an embodiment of the present invention. Figure 7 As shown, in one exemplary implementation, the step of extracting malicious responsibility frames based on the total malicious feature values ​​of the region of interest images in each frame, until the malicious feature entropy corresponding to the set of malicious responsibility frames constituted by all the malicious responsibility frames reaches its minimum value, includes:

[0146] Step A1: Sort the total malignant feature values ​​of the region of interest images in each frame, and determine the region of interest image with the largest total malignant feature value as the malignant responsible frame;

[0147] Step A2: Combine all the malicious responsibility frames and each non-malicious responsibility frame into a first image set, and calculate the total malicious feature value of each first image set. The total malicious feature value of the first image set is equal to the sum of the malicious feature values ​​of all feature dimensions in the malicious feature matrix obtained by max pooling the malicious feature matrix of the region of interest images of all frames in the first image set. The non-malicious responsibility frames are the region of interest images that have not been identified as malicious responsibility frames.

[0148] Step A3: Determine whether the malignant feature entropy corresponding to the first image set with the smallest total malignant feature value is less than the malignant feature entropy corresponding to the set of malignant responsibility frames composed of all malignant responsibility frames;

[0149] If yes, proceed to step A4; otherwise, proceed to step A5.

[0150] Step A4: Identify all frames in the first image set with the smallest total malignant feature value as the frames responsible for the malignant attack, and return to step A2.

[0151] Step A5: End the extraction of malicious responsibility frames.

[0152] Specifically, the max pooling operation on the malignant feature matrices of all frames in the first image set's region of interest (ROI) means taking the maximum malignant eigenvalue along the column direction (i.e., the direction of the feature dimension) of the malignant feature matrices of all frames in the first image set's ROI, resulting in a 1×k malignant feature matrix where the malignant eigenvalue of each feature dimension is the maximum malignant eigenvalue of the malignant feature matrices of all frames in the first image set in that feature dimension. The malignant feature matrix obtained by the max pooling operation integrates the malignant information contributed by each frame in the first image set's ROI. That is, for an image set A (A = [frame ... ... a ,frame b ,…frame n The total malignant feature value of the image set A is equal to the following relationship:

[0153]

[0154] Therefore, the responsibility frame extraction method provided by the present invention first identifies the region of interest image with the largest total malignant feature value as the first malignant responsibility frame in the set of malignant responsibility frames (i.e., the set of malignant responsibility frames). Then, each of the remaining region of interest images that has not been identified as malignant responsibility frames is combined with the first malignant responsibility frame to form a first image set (each first image set includes the first malignant responsibility frame and a region of interest image that has not been identified as a malignant responsibility frame). The total malignant feature value of each first image set is calculated. Then, the region of interest image in the first image set with the largest total malignant feature value that has not been identified as a malignant responsibility frame is the second malignant responsibility frame in the set of malignant responsibility frames. Next, the first malicious responsibility frame, the second malicious responsibility frame, and each remaining region of interest image that was not identified as a malicious responsibility frame are each combined into a first image set (each first image set includes the first malicious responsibility frame, the second malicious responsibility frame, and a region of interest image that was not identified as a malicious responsibility frame). By calculating the total malicious feature value of each first image set, the first image set with the largest total malicious feature value can be found. If the malicious feature entropy of the first image set with the largest total malicious feature value is greater than or equal to the malicious feature entropy of the malicious responsibility frame set composed of the first and second malicious responsibility frames, then the extraction of malicious responsibility frames ends, and the extracted first and second malicious responsibility frames are taken as the final malicious responsibility frames. If the malicious feature entropy of the first image set with the largest total malicious feature value is less than the malicious feature entropy of the malicious responsibility frame set composed of the first and second malicious responsibility frames, then the region of interest image in the first image set with the largest total malicious feature value that was not identified as a malicious responsibility frame is identified as the third malicious responsibility frame in the malicious responsibility frame set. Repeat the above steps until the malignant feature entropy of the first image set with the largest total malignant feature value is no less than the malignant feature entropy of the malignant responsibility frame set composed of all the malignant responsibility frames. Since visually identical regions of interest images usually share similar malignant feature matrices, adding similar regions of interest images will not significantly affect the total malignant feature value of the image set. Therefore, the malignant responsibility frame extraction method described above will not repeatedly select similar malignant responsibility frames.

[0155] Furthermore, in an exemplary implementation, the malignant feature entropy of the image set is calculated according to the following formulas (6) and (7):

[0156] H1(A)=-p1(A)×log2 p1(A) (6)

[0157]

[0158] In the formula, H1(A) represents the malignant feature entropy of image set A, MScoreA represents the total malignant feature value of image set A, and BScoreA represents the total benign feature value of image set A.

[0159] In one exemplary implementation, the step of extracting benign responsibility frames based on the feature matrix of each frame's region of interest image, until the benign feature entropy corresponding to the set of benign responsibility frames constituted by all the benign responsibility frames reaches its minimum value, includes:

[0160] For each frame of the region of interest image, the benign feature matrix of the region of interest image is obtained based on the feature matrix of the region of interest image and the difference between the benign feature weight parameters and the malignant feature weight parameters corresponding to the static medical image classification neural network model.

[0161] Based on the benign feature matrix of the region of interest image in each frame, benign responsibility frames are extracted until the benign feature entropy corresponding to the set of benign responsibility frames composed of all the benign responsibility frames reaches the minimum value.

[0162] As can be seen from equation (1) above, the probability Y0 of the region of interest image belonging to the benign category is determined only by the relative difference between the benign feature weight parameter and the malignant feature weight parameter and the feature matrix X of the region of interest image. Therefore, this invention obtains the benign feature matrix of the region of interest image based on the feature matrix of the region of interest image and the difference between the benign feature weight parameter and the malignant feature weight parameter corresponding to the static medical image classification neural network model. Then, based on the benign feature matrix of each frame of the region of interest image, benign responsibility frames are extracted until the benign feature entropy corresponding to the benign responsibility frame set composed of all the benign responsibility frames reaches the minimum value, thereby accurately extracting benign responsibility frames with a large amount of benign contribution information.

[0163] In one exemplary embodiment, obtaining the benign feature matrix of the region of interest image based on the feature matrix of the region of interest image and the difference between the benign feature weight parameters and the malignant feature weight parameters corresponding to the static medical image classification neural network model includes:

[0164] The benign feature matrix of the region of interest image is obtained according to the following formulas (8) and (9):

[0165]

[0166]

[0167] In equation (8), [FB] i In equation (9), the benign feature matrix of the region of interest image in the i-th frame is represented. This represents the feature value of the j-th feature dimension in the feature matrix of the i-th frame's region of interest image. This represents the benign feature weight of the j-th feature dimension corresponding to the static medical image classification neural network model. This represents the malignancy feature weight of the j-th feature dimension corresponding to the static medical image classification neural network model. This represents the benign feature value of the j-th feature dimension in the benign feature matrix of the i-th frame region of interest image.

[0168] It should be noted that, as those skilled in the art will understand, Indicates from 0 and Take the larger one, that is, if If it is greater than 0, then Pick like If less than 0, then Take 0.

[0169] Furthermore, in an exemplary implementation, the step of extracting benign responsibility frames based on the benign feature matrix of each frame's region of interest image, until the benign feature entropy corresponding to the set of benign responsibility frames constituted by all the benign responsibility frames reaches its minimum value, includes:

[0170] For each frame of the region of interest image, the benign feature values ​​of all feature dimensions in the benign feature matrix of the region of interest image are summed to obtain the total benign feature value of the region of interest image.

[0171] Based on the total benign feature value of the region of interest image in each frame, benign responsibility frames are extracted until the benign feature entropy corresponding to the set of benign responsibility frames composed of all the benign responsibility frames reaches the minimum value.

[0172] Specifically, combining formulas (8) and (9) above, the total malignant feature value of the i-th frame region of interest image can be expressed by the following formula:

[0173]

[0174] Therefore, by extracting benign responsibility frames based on the total benign feature values ​​of the region of interest images in each frame, not only can the extraction efficiency of benign responsibility frames be further improved, but also the extraction of benign responsibility frames with similar features can be effectively prevented.

[0175] Please continue to refer to this. Figure 8 The diagram illustrates the specific process for extracting benign responsibility frames according to an embodiment of the present invention. Figure 8As shown, in an exemplary implementation, the step of extracting benign responsibility frames based on the total benign feature values ​​of the region of interest images in each frame, until the benign feature entropy corresponding to the set of benign responsibility frames constituted by all the benign responsibility frames reaches its minimum value, includes:

[0176] Step B1: Sort the total benign feature values ​​of the region of interest images in each frame, and determine the region of interest image with the largest total benign feature value as the benign responsibility frame;

[0177] Step B2: Combine all the benign responsibility frames and each non-benign responsibility frame into a second image set, and calculate the total benign feature value for each second image set. The total benign feature value of the second image set is equal to the sum of the benign feature values ​​of all feature dimensions in the benign feature matrix obtained by max pooling the benign feature matrix of all frames in the second image set. The non-benign responsibility frames are the region of interest images that have not yet been determined as benign responsibility frames.

[0178] Step B3: Determine whether the benign feature entropy corresponding to the second image set with the smallest total benign feature value is less than the benign feature entropy corresponding to the benign responsibility frame set composed of all benign responsibility frames;

[0179] If yes, proceed to step B4; otherwise, proceed to step B5.

[0180] Step B4: Identify all region-of-interest images in the second image set with the smallest total benign feature value as benign responsibility frames, and return to step B2.

[0181] Step B5: End the extraction of benign responsibility frames.

[0182] Specifically, the max pooling operation on the benign feature matrices of all frames in the second image set's region of interest (ROI) means taking the maximum benign eigenvalue in the column direction (i.e., the direction of the feature dimension) of the benign feature matrices of all frames in the second image set, resulting in a 1×k benign feature matrix where the benign eigenvalue of each feature dimension is the maximum benign eigenvalue of the benign feature matrices of all frames in the second image set in that feature dimension. The benign feature matrix obtained by the max pooling operation integrates the benign information contributed by each frame in the second image set's ROI. That is, for image set A (A = [frame...]... a ,frame b ,…frame n The total benign feature value of the image set A is equal to the following relationship:

[0183]

[0184] Therefore, the responsibility frame extraction method provided by the present invention first identifies the region of interest image with the largest total benign feature value as the first benign responsibility frame in the benign responsibility frame set (i.e., the benign responsibility frame set), and then combines each of the remaining region of interest images that are not identified as benign responsibility frames with the first benign responsibility frame to form a second image set (at this time, each second image set includes the first benign responsibility frame and a region of interest image that is not identified as a benign responsibility frame), and calculates the total benign feature value of each second image set. Then, the region of interest image in the second image set with the largest total benign feature value that is not identified as a benign responsibility frame is the second benign responsibility frame in the benign responsibility frame set. Next, the first benign responsibility frame, the second benign responsibility frame, and each of the remaining frames whose region of interest was not identified as benign responsibility frames are grouped into a second image set (each second image set includes the first benign responsibility frame, the second benign responsibility frame, and a region of interest image not identified as benign responsibility frame). By calculating the total benign feature value of each second image set, the second image set with the largest total benign feature value can be found. If the benign feature entropy of the second image set with the largest total benign feature value is greater than or equal to that of the first benign responsibility frame and... If the benign feature entropy of the benign responsibility frame set consisting of the second benign responsibility frame is less than the benign feature entropy of the benign responsibility frame set consisting of the first and second benign responsibility frames, then the extracted benign responsibility frames are taken as the final benign responsibility frames. If the benign feature entropy of the second image set with the largest total benign feature value is less than the benign feature entropy of the benign responsibility frame set consisting of the first and second benign responsibility frames, then the region of interest image in the second image set with the largest total benign feature value that was not identified as a benign responsibility frame is identified as the third benign responsibility frame in the benign responsibility frame set. The above steps are repeated until the benign feature entropy of the second image set with the largest total benign feature value is no longer less than the benign feature entropy of the benign responsibility frame set consisting of all benign responsibility frames. Since visually identical region of interest images usually share similar benign feature matrices, adding similar region of interest images will not have a significant impact on the total benign feature value of the image set. Therefore, the benign responsibility frame extraction method described above will not repeatedly select similar benign responsibility frames.

[0185] Specifically, in one exemplary implementation, the benign feature entropy of the image set is calculated according to the following formulas (12) and (13):

[0186] H0(A)=-p0(A)×log2 p0(A) (12)

[0187]

[0188] In the formula, H0(A) represents the benign feature entropy of image set A, MScoreA represents the total malignant feature value of image set X, and BScoreA represents the total benign feature value of image set A.

[0189] Corresponding to the above-mentioned method for extracting responsibility frames, this invention also provides a method for classifying medical videos. Please refer to [link / reference]. Figure 9 The diagram illustrates a flowchart of a medical video classification method according to an embodiment of the present invention. Figure 9 As shown, the medical video classification method includes the following steps:

[0190] Step S210: Using the responsibility frame extraction method described above, extract the malign responsibility frames and / or benign responsibility frames from the acquired medical video to be classified.

[0191] Step S220: Classify the medical video to be classified based on the feature matrix of the extracted malicious responsibility frames and / or benign responsibility frames.

[0192] Therefore, the medical video classification method provided by the present invention classifies the acquired medical video to be classified by extracting the malignant responsibility frames and / or benign responsibility frames using the responsibility frame extraction method described above. This can effectively reduce the interference of noise frames in the medical video, effectively improve the accuracy of medical video classification, and thus effectively reduce the misdiagnosis rate of doctors.

[0193] In one exemplary implementation, classifying the medical video to be classified based on the feature matrix of the extracted malicious and / or benign responsibility frames includes:

[0194] Max pooling is performed on the feature matrices of the extracted malicious and / or benign responsibility frames to obtain the feature matrices of the responsibility frame set.

[0195] The medical videos to be classified are classified based on the feature matrix of the set of responsible frames.

[0196] Specifically, by performing max pooling on the feature matrices of all malicious and / or benign responsibility frames in the column direction, a feature matrix with the largest eigenvalue in each feature dimension can be obtained, i.e., the feature matrix of the responsibility frame set. Therefore, based on the obtained feature matrix of the responsibility frame set, medical videos can be accurately classified. It should be noted that, to further improve the accuracy of medical video classification results, the responsibility frame extraction method described above can be used to simultaneously extract malicious and benign responsibility frames from the acquired medical video to be classified. Max pooling is then performed on the feature matrices of the extracted malicious and benign responsibility frames to obtain the feature matrix of the responsibility frame set. Since the feature matrix of the responsibility frame set obtained by max pooling the feature matrices of the extracted malicious and benign responsibility frames combines both malicious and benign information, the accuracy of medical video classification results can be further improved.

[0197] In one exemplary implementation, classifying the medical video to be classified based on the feature matrix of the set of responsible frames includes:

[0198] The feature matrix of the set of responsible frames is input into the video classification model to classify the medical video to be classified.

[0199] Therefore, by inputting the feature matrix contributed by all responsible frames (including malicious and / or benign responsible frames) into a pre-trained video classification model, the final classification of medical videos can be performed.

[0200] Furthermore, the video classification model is a random forest classifier. The random forest classifier consists of multiple classification trees, each classifying the input feature matrix. The random forest classifier then votes on the classification results from all trees to ultimately determine the benign or malignant nature of the medical video (i.e., the benign or malignant nature of lesions in the medical video). Please refer to [reference needed]. Figure 10 The diagram illustrates a method for video classification using a random forest classifier, as provided in one embodiment of the present invention. Figure 10 As shown, by performing max pooling on the feature matrices of the obtained malignant and benign responsibility frames, the feature matrix of the resulting responsibility frame set is input into a random forest classifier to obtain the classification result of the medical video, that is, whether the medical video is a benign medical video (i.e., the lesions in the medical video are benign) or a malignant medical video (i.e., the lesions in the medical video are malignant).

[0201] It should be noted that, as those skilled in the art will understand, in some other embodiments, the video classification model can also be a classification model other than the random forest classifier, and the present invention does not limit this. Furthermore, as those skilled in the art will understand, the random forest classifier is pre-trained; specifically, it can be trained using a medical video training set (which includes the feature matrix of the responsible frames of medical videos and their corresponding classification labels) to obtain the video classification model.

[0202] In one exemplary embodiment, after extracting malicious and / or benign responsibility frames from the acquired medical video to be classified, the medical video classification method further includes:

[0203] Display the extracted malicious and / or benign liability frames, and receive user confirmation or deletion operations for the extracted malicious and / or benign liability frames, as well as receive user operations to add liability frames.

[0204] Correspondingly, the medical video to be classified is classified according to the feature matrix of the doctor's adjusted responsibility frame.

[0205] Therefore, by receiving user confirmation or deletion of extracted malicious and / or benign liability frames, as well as receiving user-added liability frames, it can not only better adapt to doctors' personalized preferences, but also generate additional risk predictions by selecting liability frames recognized and recommended by doctors, thereby further improving the accuracy of medical video classification.

[0206] Please continue to refer to this. Figure 11 The diagram illustrates, illustratively, a software interface diagram of a doctor adjusting the responsibility frame according to an embodiment of the present invention. Figure 11 As shown, the extracted malignant and / or benign responsibility frames can be displayed in the responsibility section recommendation window in the software interface. The acquired medical videos to be classified can also be displayed in the video playback window in the software interface. Doctors can access the neighboring frames (i.e., the neighboring region of interest images) near the responsibility frame through the "Previous Frame" and "Next Frame" buttons, and can choose to accept or reject the current frame (i.e. the currently accessed region of interest image) as the responsibility frame. The system will automatically record the responsibility frames after the doctor's confirmation.

[0207] The inventors of this invention acquired 13,702 2D ultrasound images of breast nodules (including 9,177 images from 2,457 patients with benign pathology and 4,545 images from 991 patients with malignant pathology) and 2,141 breast ultrasound videos (including 1,227 videos from 560 patients with benign pathology and 914 videos from 412 patients with malignant pathology) for training and validation of a static medical image classification neural network model and a video classification model. The performance of the medical video classification method provided by this invention was evaluated using AUROC (Area Under the Subject Curve), accuracy, sensitivity, and specificity metrics. The validation results of the five-fold cross-validation (dividing the dataset into five equal parts, using one part as the test set in each round, and the rest as the training set) are shown in Table 1 below, and the test set results are shown in Table 2 below.

[0208] Table 1. Results of five-fold cross-validation for classifying benign and malignant breast nodules

[0209]

[0210]

[0211] Table 2. Results of the benign and malignant classification test set for breast nodules

[0212]

[0213] As shown in Tables 1 and 2, the AUROC, accuracy, sensitivity, and specificity of breast nodule benign and malignant classification using responsibility frames (including malignant responsibility frames and / or benign responsibility frames) extracted by the responsibility frame extraction method provided by this invention are significantly better than those of breast nodule benign and malignant classification using responsibility frames manually selected by doctors.

[0214] Based on the same inventive concept, the present invention also provides an electronic device, please refer to... Figure 12 The diagram illustrates a block structure of an electronic device according to an embodiment of the present invention. Figure 12 As shown, the electronic device includes a processor 101 and a memory 103. The memory 103 stores a computer program. When the computer program is executed by the processor 101, it implements the responsibility frame extraction method or the medical video classification method described above. Since the electronic device provided by this invention belongs to the same inventive concept as the responsibility frame extraction method or the medical video classification method described above, the electronic device provided by this invention has all the advantages of the responsibility frame extraction method or the medical video classification method described above. Therefore, the beneficial effects of the electronic device provided by this invention will not be elaborated further here.

[0215] like Figure 12 As shown, the electronic device also includes a communication interface 102 and a communication bus 104, wherein the processor 101, the communication interface 102, and the memory 103 communicate with each other via the communication bus 104. The communication bus 104 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 104 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 102 is used for communication between the aforementioned electronic device and other devices.

[0216] The processor 101 referred to in this invention can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 101 is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and lines.

[0217] The memory 103 can be used to store the computer program. The processor 101 implements various functions of the electronic device by running or executing the computer program stored in the memory 103 and calling the data stored in the memory 103.

[0218] The memory 103 may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0219] This invention also provides a readable storage medium storing a computer program. When executed by a processor, the computer program can implement the responsibility frame extraction method or the medical video classification method described above. Since the readable storage medium provided by this invention belongs to the same inventive concept as the responsibility frame extraction method or the medical video classification method described above, it possesses all the advantages of these methods. Therefore, the beneficial effects of the readable storage medium provided by this invention will not be elaborated upon here.

[0220] The readable storage medium of embodiments of the present invention can be any combination of one or more computer-readable media. The readable medium can be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires, portable computer hard disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device.

[0221] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0222] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0223] Compared with existing technologies, the responsibility frame extraction method, medical video classification method, electronic device, and storage medium provided by this invention have the following advantages:

[0224] (1) The responsibility frame extraction method provided by this invention first uses a target detection neural network model to extract the region of interest (ROI) of each frame of medical image in the acquired medical video to obtain the ROI image corresponding to each frame of medical image; then, it uses the skeleton network of a static medical image classification neural network model to extract features from each ROI image to obtain the feature matrix of each ROI image; finally, based on the feature matrix of each ROI image, it extracts malignant responsibility frames until the malignant feature entropy corresponding to the set of malignant responsibility frames formed by all the malignant responsibility frames reaches the minimum value; and / or based on the feature matrix of each ROI image, it extracts benign responsibility frames until the benign feature entropy corresponding to the set of benign responsibility frames formed by all the benign responsibility frames reaches the minimum value. Therefore, this invention can automatically extract malignant responsibility frames and / or benign responsibility frames from medical videos that contribute different important features to medical video classification (e.g., classification of benign and malignant nodule videos), thereby improving the efficiency and accuracy of medical video classification (e.g., classification of benign and malignant nodule videos) and better assisting doctors in improving the efficiency and accuracy of diagnosis. Furthermore, this invention can automatically determine the required number of malignant and / or benign responsibility frames, further improving the efficiency and accuracy of medical video classification (e.g., classification of benign and malignant nodule videos). Additionally, the responsibility frame extraction method provided by this invention first uses a target detection neural network model to extract the region of interest (ROI) image from each frame of the acquired medical video, and then extracts malignant and / or benign responsibility frames based on the feature matrix of each ROI image. This effectively reduces image noise interference during the extraction process, further improving the efficiency and accuracy of malignant and / or benign responsibility frame extraction.

[0225] (2) The medical video classification method provided by this invention first extracts malignant and / or benign responsibility frames from the acquired medical video to be classified using the responsibility frame extraction method described above, and then classifies the medical video to be classified based on the feature matrix of the extracted malignant and / or benign responsibility frames. Therefore, the medical video classification method provided by this invention, by classifying the acquired medical video to be classified based on the malignant and / or benign responsibility frames extracted using the responsibility frame extraction method described above, can effectively reduce the interference of noise frames in the medical video, effectively improve the accuracy of medical video classification, and thus effectively reduce the misdiagnosis rate of doctors.

[0226] (3) Since the electronic device and readable storage medium provided by the present invention belong to the same inventive concept as the responsibility frame extraction method or the medical video classification method described above, the electronic device and readable storage medium provided by the present invention have all the advantages of the responsibility frame extraction method or the medical video classification method described above. Therefore, the beneficial effects of the electronic device and readable storage medium provided by the present invention will not be described in detail here.

[0227] It should be noted that the apparatus and methods disclosed in the embodiments herein can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments herein. In this regard, each block in a flowchart or block diagram may represent a module, program, or part of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system to perform the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0228] In addition, the functional modules in the various embodiments of this article can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0229] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure are within the protection scope of the present invention. Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for extracting responsibility frames, characterized in that, include: An object detection neural network model is used to extract the region of interest (ROI) from each frame of the acquired medical video to obtain the ROI image corresponding to each frame of the medical video. The backbone network of the static medical image classification neural network model is used to extract features from each frame of the region of interest image to obtain the feature matrix of each frame of the region of interest image; Based on the feature matrix of the region of interest image of each frame, the malicious responsibility frames are extracted until the malicious feature entropy corresponding to the set of malicious responsibility frames composed of all the malicious responsibility frames reaches the minimum value. and / or Based on the feature matrix of the region of interest image of each frame, benign responsibility frames are extracted until the benign feature entropy corresponding to the benign responsibility frame set composed of all the benign responsibility frames reaches the minimum value. The step of extracting malicious responsibility frames based on the feature matrix of each frame's region of interest (ROI) image until the malignant feature entropy corresponding to the set of all malicious responsibility frames reaches its minimum value includes: for each ROI image frame, obtaining the malignant feature matrix of the ROI image based on the feature matrix of the ROI image and the difference between the malignant feature weight parameters and benign feature weight parameters corresponding to the static medical image classification neural network model; for each ROI image frame, summing the malignant feature values ​​of all feature dimensions in the malignant feature matrix of the ROI image frame to obtain the total malignant feature value of the ROI image; step A1: sorting the total malignant feature values ​​of each ROI image frame, and determining the ROI image with the largest total malignant feature value as the malicious responsibility frame; step A2: combining all malicious responsibility frames with each non-malicious... Each frame responsible for the crime is grouped into a first image set, and the total malignant feature value of each first image set is calculated. The total malignant feature value of the first image set is equal to the sum of the malignant feature values ​​of all feature dimensions in the malignant feature matrix obtained by max pooling the malignant feature matrices of all frames in the first image set. The non-malignant frames are the region-of-interest images that have not been identified as malignant frames. Step A3: Determine whether the malignant feature entropy corresponding to the first image set with the smallest total malignant feature value is less than the malignant feature entropy corresponding to the set of malignant frames composed of all malignant frames. If yes, proceed to step A4; if no, proceed to step A5. Step A4: Identify all frames in the region-of-interest images of the first image set with the smallest total malignant feature value as malignant frames and return to step A2. Step A5: End the extraction of malignant frames. The step of extracting benign responsibility frames based on the feature matrices of each region of interest (ROI) image until the benign feature entropy corresponding to the set of benign responsibility frames constituted by all the benign responsibility frames reaches its minimum value includes: for each ROI image, obtaining the benign feature matrix of the ROI image based on the feature matrix of the ROI image and the difference between the benign feature weight parameters and the malignant feature weight parameters corresponding to the static medical image classification neural network model; for each ROI image, summing the benign feature values ​​of all feature dimensions in the benign feature matrix of the ROI image to obtain the total benign feature value of the ROI image; step B1, sorting the total benign feature values ​​of each ROI image, and determining the ROI image with the largest total benign feature value as the benign responsibility frame; step B2, combining all the benign responsibility frames with each non-benign responsibility frame... Each set of images is composed of frames with a benign responsibility, and the total benign feature value of each set is calculated. The total benign feature value of each set is equal to the sum of the benign feature values ​​of all feature dimensions in the benign feature matrix obtained by max-pooling the benign feature matrices of all frames in the region of interest (ROI) within the set. The non-benign responsibility frames are ROI images that have not yet been identified as benign responsibility frames. Step B3: Determine whether the benign feature entropy corresponding to the set of images with the smallest total benign responsibility value is less than that of all frames with a benign responsibility value. The benign feature entropy corresponding to the benign responsibility frame set composed of the responsibility frames; if yes, proceed to step B4, otherwise proceed to step B5; Step B4: Determine all region of interest images in the second image set with the smallest total benign feature value as benign responsibility frames, and return to proceed to step B2; Step B5: End the extraction of benign responsibility frames. The entropy of the malignant features of the image set is calculated using the following formula: In the formula, MScoreA represents the malignant feature entropy of image set A, MScoreA represents the total malignant feature value of image set A, and BScoreA represents the total benign feature value of image set A. Calculate the benign feature entropy of the image set using the following formula: In the formula, Let MScoreA represent the benign feature entropy of image set A, MScoreA represent the total malignant feature value of image set A, and BScoreA represent the total benign feature value of image set A.

2. The responsibility frame extraction method according to claim 1, characterized in that, The step of obtaining the malignant feature matrix of the region of interest image based on the feature matrix of the region of interest image and the difference between the malignant feature weight parameters and the benign feature weight parameters corresponding to the static medical image classification neural network model includes: The malignant feature matrix of the region of interest image is obtained according to the following formula: In the formula, This represents the feature value of the j-th feature dimension in the feature matrix of the i-th frame's region of interest image. This represents the malignancy feature weight of the j-th feature dimension corresponding to the static medical image classification neural network model. This represents the benign feature weight of the j-th feature dimension corresponding to the static medical image classification neural network model. Let represent the malignant feature value of the j-th feature dimension in the malignant feature matrix of the i-th frame's region of interest image. Represents the malignant feature matrix of the region of interest image in the i-th frame; and / or The step of obtaining the benign feature matrix of the region of interest image based on the feature matrix of the region of interest image and the difference between the benign feature weight parameters and the malignant feature weight parameters corresponding to the static medical image classification neural network model includes: The benign feature matrix of the region of interest image is obtained according to the following formula: In the formula, This represents the feature value of the j-th feature dimension in the feature matrix of the i-th frame's region of interest image. This represents the benign feature weight of the j-th feature dimension corresponding to the static medical image classification neural network model. This represents the malignancy feature weight of the j-th feature dimension corresponding to the static medical image classification neural network model. Let represent the benign feature value of the j-th feature dimension in the benign feature matrix of the i-th frame's region of interest image. Let represent the benign feature matrix of the region of interest image in the i-th frame.

3. The responsibility frame extraction method according to claim 1, characterized in that, The step of using a target detection neural network model to extract the region of interest (ROI) from each frame of the acquired medical video to obtain the ROI image corresponding to each frame of the medical video includes: An object detection neural network model is used to extract the region of interest (ROI) from each frame of the acquired medical video to obtain the location information of the ROI corresponding to each frame of the medical video. Based on the location information of the region of interest corresponding to each frame of medical images, the corresponding region is cropped from each frame of medical images to obtain the region of interest image corresponding to each frame of medical images.

4. The responsibility frame extraction method according to claim 3, characterized in that, Before extracting features from the region of interest in each frame of the static medical image classification neural network model using its backbone network, the method further includes: For each frame of the region of interest image: The larger of the width and height dimensions of the region of interest image is used as the target side length. The region of interest image is padded to adjust the smaller of the width and height dimensions of the region of interest image to the target side length dimension; The region of interest image, adjusted to the target side length, is magnified or reduced to adjust its size to a preset size.

5. A medical video classification method, characterized in that, include: Using the responsibility frame extraction method described in any one of claims 1 to 4, malignant responsibility frames and / or benign responsibility frames are extracted from the acquired medical video to be classified; as well as The medical video to be classified is classified based on the feature matrix of the extracted malicious and / or benign responsibility frames.

6. The medical video classification method according to claim 5, characterized in that, The step of classifying the medical video to be classified based on the feature matrix of the extracted malicious and / or benign responsibility frames includes: Max pooling is performed on the feature matrices of the extracted malicious and / or benign responsibility frames to obtain the feature matrices of the responsibility frame set. The medical videos to be classified are classified based on the feature matrix of the set of responsible frames.

7. The medical video classification method according to claim 5, characterized in that, After extracting the malicious and / or benign responsibility frames from the acquired medical videos to be classified, the medical video classification method further includes: Display the extracted malicious and / or benign liability frames, and receive user confirmation or deletion operations for the extracted malicious and / or benign liability frames, as well as receive user operations to add liability frames.

8. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the responsibility frame extraction method of any one of claims 1 to 4 or the medical video classification method of any one of claims 5 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, implements the responsibility frame extraction method according to any one of claims 1 to 4 or the medical video classification method according to any one of claims 5 to 7.

Images