Data detection method, breast nodule detection model training method and electronic device

Abstract

The embodiments of the present invention provide a data detection method, a breast nodule detection model training method and an electronic device, which relate to the field of medical image processing technology. By inputting the ultrasound video to be detected into the trained breast nodule detection model, the detection frame information of each frame image of the ultrasound video to be detected is obtained, and each detection frame of each frame image is placed into multiple detection frame queues according to the position relationship. For each detection detection frame queue, it is determined whether the detection frame queue is a target detection frame queue that meets the first preset condition; if so, the position area corresponding to the detection frame in the target detection frame queue is marked as a breast nodule. In this way, the breast nodule detection model can be used to detect the ultrasound video, which can effectively reduce the false detection on the breast ultrasound video, and then the detection frame queue is used for screening and filtering, which can further reduce the false detection.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a data detection method, a breast nodule detection model training method and an electronic device. Background Art

[0002] Ultrasonic breast scanning is an important method for screening and diagnosing breast nodules. At present, the detection method for ultrasonic breast nodules is generally based on the detection or segmentation model trained with retained image data. The detection method is generally a one-stage target detection method, a two-stage target detection method or a segmentation model based on natural images.

[0003] However, when these existing target detection methods or segmentation methods are directly used on breast ultrasound video data, many false detections will occur, because the breast in ultrasound video is more complex than other parts. For example, breast hyperplasia, fat masses in the glandular layer, etc. will be misidentified, and large areas of disordered glands will also be misidentified, resulting in a large number of false detections in these detection methods. Summary of the invention

[0004] In view of this, an object of embodiments of the present invention is to provide a data detection method, a breast nodule detection model training method and an electronic device to at least partially improve the above-mentioned problems.

[0005] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a data detection method, including:

[0007] Input the ultrasound video to be detected into the trained breast nodule detection model to obtain the detection frame information of each frame image of the ultrasound video to be detected; the detection frame information includes the number and position information of the detection frame, and the detection frame is a pre-mark for the breast nodule;

[0008] Putting the detection frames of the image of each frame into a plurality of detection frame queues according to the position relationship;

[0009] For each of the detection frame queues, determining whether the detection frame queue is a target detection frame queue that satisfies a first preset condition; the first preset condition is that the number of detection frames in the detection frame queue is greater than one half of the length of the detection frame queue;

[0010] If yes, the position area corresponding to the detection frame in the target detection frame queue is marked as a breast nodule;

[0011] If not, the detection box in the target detection box queue is deleted without marking.

[0012] Optionally, placing the detection frames of the images of the frames into a plurality of detection frame queues according to a position relationship includes:

[0013] Putting each of the detection frames of the image of the first frame into a corresponding number of detection frame queues respectively;

[0014] Processing the images except the image of the first frame in sequence, and determining whether the number of detection frames in each detection frame queue is equal to a preset number;

[0015] If yes, delete the head detection frame of the detection frame queue;

[0016] According to the positional relationship between each detection frame of the image currently being processed and the tail detection frame of each detection frame queue, each detection frame of the image currently being processed is placed in the corresponding detection frame queue.

[0017] Optionally, placing each detection frame of the image currently being processed into a corresponding detection frame queue according to a positional relationship between each detection frame of the image currently being processed and a tail detection frame of each detection frame queue, comprises:

[0018] sequentially determining whether the overlap between each of the detection frames of the image currently being processed and the tail detection frame of each of the detection frame queues meets a second preset condition;

[0019] If so, put the detection frame into the corresponding detection frame queue;

[0020] If not, put the detection frame into a new detection frame queue;

[0021] Determine in sequence whether there is a new detection frame to be put into each of the detection frame queues;

[0022] If not, put an empty data into the detection frame queue.

[0023] Optionally, placing each detection frame of the image currently being processed into a corresponding detection frame queue according to a positional relationship between each detection frame of the image currently being processed and a tail detection frame of each detection frame queue, comprises:

[0024] Calculating the overlap between each detection frame of the image currently being processed and the tail detection frame of each detection frame queue;

[0025] sorting the overlaps that meet the second preset condition, and placing the detection frames with high overlaps into the corresponding detection frame queue;

[0026] Putting the detection frames whose overlap does not meet the second preset condition into a new detection frame queue;

[0027] Determine in sequence whether there is a new detection frame to be put into each of the detection frame queues;

[0028] If not, put an empty data into the detection frame queue.

[0029] Optionally, the breast nodule detection model includes a first basic detection model and a first video feature fusion module, and the method further includes a step of training the breast nodule detection model, including:

[0030] The first basic detection model is trained using the retained image data to obtain a mature first basic detection model; the first basic detection model includes a first backbone network and a first detection head module;

[0031] Acquire an ultrasound video sample set; the ultrasound video sample set includes a plurality of ultrasound videos, and the ultrasound video includes a real annotation frame of a breast nodule;

[0032] Extracting sample images of each frame of each ultrasound video, and extracting a preset number of sample images in the same ultrasound video whose image similarity is lower than a preset threshold as a batch;

[0033] Inputting the batch into the first backbone network to obtain the preset number of first feature matrices;

[0034] Inputting each of the first feature matrices into the first video feature fusion module to obtain the preset number of second feature matrices after fusion processing;

[0035] Input each of the second feature matrices into the first detection head module, and obtain the prediction box information of each of the sample images through upsampling operation and linear layer;

[0036] Based on the prediction box information and the annotation box, loss information is calculated, and the breast nodule detection model is iteratively trained according to the loss information until a preset condition is met.

[0037] Optionally, the calculation formula for the image similarity is:

[0038]

[0039] in, is the pixel mean of image x, is the pixel mean of image y, is the pixel variance of image x, is the pixel variance of image y, is the pixel covariance of image x and image y, c1 and c2 are constants.

[0040] Optionally, the step of inputting each of the first feature matrices into the first video feature fusion module to obtain the preset number of second feature matrices after fusion processing includes:

[0041] Concatenate the first feature matrices according to the channel dimension to obtain a concatenated feature matrix;

[0042] Simultaneously fusing the spliced ​​feature matrices through a convolution operation to obtain a fused feature matrix, and performing channel pooling on the spliced ​​feature matrix to obtain a compressed feature vector;

[0043] Multiplying the compressed feature vector by the fused feature matrix to obtain a weight feature matrix for changing the regional weight;

[0044] The weight feature matrix is ​​divided into the preset number of second feature matrices.

[0045] Optionally, the prediction box information includes coordinate information of the prediction box and a prediction box category, and the calculation formula of the loss information is:

[0046]

[0047] in, , and They are the weights of the three loss functions respectively;

[0048]

[0049] in, is the probability that the detection box is a nodule category, is the parameter used to reduce the proportion of simple sample loss;

[0050]

[0051] in, is the predicted box probability, is the intersection-over-union ratio between the predicted box and the labeled box, is the parameter used to reduce the proportion of simple sample loss;

[0052]

[0053]

[0054]

[0055] in, is the prediction box, is the annotation box, is the intersection-over-union ratio between the annotation box and the prediction box, is the square of the Euclidean distance between the center point of the annotation box and the center point of the prediction box, is the square of the shortest diagonal distance between the annotation box and the prediction box, and is the width and height corresponding to the annotation box, and are the width and height corresponding to the predicted box.

[0056] In a second aspect, an embodiment of the present invention provides a breast nodule detection model training method, which is applied to a breast nodule detection model training system. The breast nodule detection model training system includes a second basic detection model and a second video feature fusion module. The method includes:

[0057] The second basic detection model is trained using the retained image data to obtain a mature second basic detection model; the second basic detection model includes a second backbone network and a second detection head module;

[0058] Acquire an ultrasound video sample set; the ultrasound video sample set includes a plurality of ultrasound videos, and the ultrasound video includes a real annotation frame of a breast nodule;

[0059] Extracting sample images of each frame of each ultrasound video, and extracting a preset number of sample images in the same ultrasound video whose image similarity is lower than a preset threshold as a batch;

[0060] Inputting the batch into the second backbone network to obtain the preset number of first feature matrices;

[0061] Inputting each of the first feature matrices into the second video feature fusion module to obtain the preset number of second feature matrices after fusion processing;

[0062] Input each of the second feature matrices into the second detection head module, and obtain the prediction box information of each of the sample images through upsampling operation and linear layer;

[0063] Based on the prediction box information and the annotation box, loss information is calculated, and the breast nodule detection model is iteratively trained according to the loss information until a preset condition is met.

[0064] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, it implements any one of the above-mentioned first aspect and / or the method described in the second aspect.

[0065] The embodiment of the present invention provides a data detection method, a breast nodule detection model training method and an electronic device. By inputting the ultrasound video to be detected into the trained breast nodule detection model, the detection frame information of each frame image of the ultrasound video to be detected is obtained, and each detection frame of each frame image is placed into multiple detection frame queues according to the position relationship. For each detection detection frame queue, it is determined whether the detection frame queue is a target detection frame queue that meets the first preset condition; if so, the position area corresponding to the detection frame in the target detection frame queue is marked as a breast nodule. In this way, the breast nodule detection model can be used to detect the ultrasound video, which can effectively reduce the false detection on the breast ultrasound video, and then the false detection can be further reduced by screening and filtering through the detection frame queue.

[0066] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention;

[0069] Figure 2 A schematic diagram of a flow chart of a data detection method provided by an embodiment of the present invention;

[0070] Figure 3 A schematic diagram of the process of step S220 provided in an embodiment of the present invention;

[0071] Figure 4 A schematic diagram of the structural frame of a breast nodule detection model provided by an embodiment of the present invention;

[0072] Figure 5 A schematic diagram of a process for training a breast nodule detection model provided by an embodiment of the present invention;

[0073] Figure 6 A schematic diagram of the structural block diagram of a breast nodule detection model training system provided by an embodiment of the present invention;

[0074] Figure 7 A schematic flow chart of a breast nodule detection model training method provided in an embodiment of the present invention.

[0075] Icon: 100-electronic device; 101-memory; 102-communication interface; 103-processor; 104-bus; 30-breast nodule detection model; 31-first basic detection model; 311-first backbone network; 312-first detection head module; 32-first video feature fusion module; 40-breast nodule detection model training system; 41-second basic detection model; 411-second backbone network; 412-second detection head module; 42-second video feature fusion module. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0077] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0079] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0080] As described in the background technology, ultrasonic breast scanning is an important method for screening and diagnosing breast nodules. At present, the detection method for ultrasonic breast nodules is generally based on the detection or segmentation model trained with retained image data. The detection method is generally a one-stage target detection method based on natural images, a two-stage target detection method or a segmentation model.

[0081] However, these existing target detection methods or segmentation methods are all based on models trained with image data or natural image video data. If they are directly used on breast video data, many false detections will be found. For example, breast hyperplasia masses and fat masses in the glandular layer will be misidentified, and large areas of disordered glands will also be misidentified, resulting in a large number of false detections in these detection methods.

[0082] Based on the above situation, an embodiment of the present invention provides a data detection method, a breast nodule detection model training method and an electronic device. By inputting the ultrasound video to be detected into a trained breast nodule detection model, the detection frame information of each frame image of the ultrasound video to be detected is obtained, and each detection frame of each frame image is placed into multiple detection frame queues according to the position relationship for screening and filtering. In this way, the breast nodule detection model can be used to detect the ultrasound video, which can effectively reduce the false detection on the breast ultrasound video. Screening and filtering by the detection frame queue can further reduce the false detection.

[0083] To implement the process steps and functions of each example of the present invention, please refer to Figure 1 , Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, and the memory 101 and the processor 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing.

[0084] The electronic device 100 may be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understandable that the electronic device 100 is not limited to a physical server, but may also be a virtual machine on a physical server, a virtual machine built on a cloud platform, etc., which can provide the same functions as the server or virtual machine. The operating system of the electronic device 100 may be, but is not limited to, a Windows system, a Linux system, etc.

[0085] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0086] The communication connection between the electronic device 100 and an external device is achieved through at least one communication interface 102 (which can be wired or wireless).

[0087] The processor 103 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the embodiment of the present invention can be completed by the hardware integrated logic circuit in the processor 103 or the instructions in the form of software. The processor 103 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0088] Understandably, Figure 1 The structure shown is for illustration only. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0089] The following is an exemplary description of the data detection method provided by the present invention. Specifically, Figure 2 A flow chart of a data detection method provided by an embodiment of the present invention is shown in FIG. Figure 2 , the execution subject of this method can be the above Figure 1 The electronic device 100 shown in FIG. 1 includes: Figure 2 The following steps are described:

[0090] S210: Input the ultrasound video to be detected into the trained breast nodule detection model to obtain the detection frame information of each frame of the ultrasound video to be detected, wherein the detection frame information includes the number and position information of the detection frame, and the detection frame is a pre-mark for the breast nodule.

[0091] S220: Put the detection frames of each frame image into multiple detection frame queues according to their positional relationships.

[0092] The first preset condition is that the number of detection frames in the detection frame queue is greater than half of the length of the detection frame queue. For example, if the length of the detection frame queue is set to 15, then the number of detection frames in the detection frame queue is greater than 7.5, which meets the first preset condition.

[0093] S230: for each detection frame queue, determine whether the detection frame queue is a target detection frame queue that meets the first preset condition. If yes, execute step S240; if yes, execute step S250.

[0094] S240: Mark the location area corresponding to the detection box in the target detection box queue as a breast nodule.

[0095] S250: Delete the detection box in the target detection box queue without marking it.

[0096] The ultrasound video to be detected can be obtained by ultrasonic breast scanning of patients or physical examination personnel, and the breast nodule detection model can be a detection model obtained based on video training. Through the breast nodule detection model, each frame of the image in the ultrasound video to be detected can be detected to obtain the detection frame information of each frame of the image. One frame of the image can include multiple detection frames, each detection frame contains position information, and the position of the detection frame is the pre-marking of the breast nodule.

[0097] After marking each frame of the ultrasound video to be detected by the breast nodule detection model, the detection frame of each frame is arranged into multiple detection frame queues, and then for each detection frame queue, it is determined whether the detection frame queue is a target detection frame queue that meets the first preset condition. If so, it means that the detection frame in the detection frame queue is a real breast nodule, and the position area corresponding to the detection frame in the target detection frame queue is marked as a breast nodule. If not, it means that the detection frame in the detection frame queue is not a real breast nodule, and the detection frame in the target detection frame queue is deleted without marking.

[0098] This data detection method uses a breast nodule detection model to perform target detection on the ultrasound video to be detected, and then screens and filters the detected targets through a detection frame queue, so that the breast nodule detection model can be used to detect the ultrasound video, which can effectively reduce false detections on the breast ultrasound video. Screening and filtering through the detection frame queue can further reduce false detections.

[0099] In one possible implementation, a fixed length can be set for the detection box queue, see Figure 3 , step S220 may include the following steps:

[0100] S221: Put each detection frame of the first frame image into a corresponding number of detection frame queues.

[0101] The image of the ultrasound video is continuous. First, each detection frame of the image of the first frame is placed into a plurality of detection frame queues, and one detection frame is placed in one detection frame queue.

[0102] For example, if the first frame image has three detection frames, the three detection frames are placed in three detection frame queues respectively.

[0103] S222: Process the images except the first frame in sequence, and determine whether the number of detection frames in each detection frame queue is equal to a preset number.

[0104] After placing the detection frame of the first frame image, the subsequent images are processed in sequence, that is, the second frame, the third frame, and so on until the last frame.

[0105] When processing each frame, it is first determined whether the number of detection frames in each detection frame queue is equal to a preset number, where the preset number is a fixed length set for the detection frame queue. For example, the preset number may be 15.

[0106] S223: If yes, delete the head detection frame of the detection frame queue.

[0107] If a detection frame queue is full when processing the image of the current frame, the head detection frame of the detection frame queue will be deleted first. The head detection frame is the detection frame that enters the queue first in the detection frame queue. If the detection frame queue is not full, no processing will be performed.

[0108] S224: According to the positional relationship between each detection frame of the currently processed image and the tail detection frame of each detection frame queue, each detection frame of the currently processed image is placed into the corresponding detection frame queue.

[0109] By comparing the positional relationship between each detection frame of the currently processed image and the tail detection frame of each detection frame queue, the detection frames with similar positions can be matched and put in, where the tail detection frame is the detection frame that enters the queue last in the detection frame queue.

[0110] There are many ways to implement step S224. In one possible implementation, step S224 may include the following steps:

[0111] S2241: sequentially determine whether the overlap between each detection frame of the currently processed image and the tail detection frame of each detection frame queue meets the second preset condition. If yes, execute step S2242; if not, execute step S2243.

[0112] The overlap degree may be an expression of the center point position offset and the intersection-and-union ratio of the two detection frames. Specifically, the smaller the center point position offset and the larger the intersection-and-union ratio, the higher the overlap degree. The second preset condition may be set as the overlap degree being greater than a certain threshold.

[0113] S2242: Put the detection frame into the corresponding detection frame queue.

[0114] S2243: Put the detection frame into a new detection frame queue.

[0115] S2244: Determine in turn whether there is a new detection frame in each detection frame queue. If not, execute step S2245.

[0116] S2245: Put an empty data into the detection frame queue.

[0117] For example, following the example in step S221 above, the first frame image has 3 detection frames, which are placed in the 3 detection frame queues. Now the second frame image is processed. If the second frame image includes 4 detection frames, the 4 detection frames of the second frame image are judged and processed in turn.

[0118] First, for the first detection frame, calculate the overlap with the last detection frame in the existing three detection frame queues in turn. When the overlap meets the second preset condition, put the detection frame into the corresponding detection frame queue, and the detection frame queue that is put into the detection frame will no longer participate in the calculation in the current processing, and no more detection frames will be added, that is, the detection frame queue will only add one detection frame at most when processing each frame of the image. If the overlap between the detection frame and the last detection frame in the existing three detection frame queues does not meet the second preset condition, then put the detection frame into a new detection frame queue.

[0119] Then, the same method is used to process the next three detection boxes.

[0120] After processing the four detection frames in sequence, it is necessary to check the first three detection frame queues separately to determine whether there are new detection frames added in the current frame image processing. If no new detection frames are added, fill the detection frame queue with empty data.

[0121] The final processing is completed. For example, the three detection frames of the first frame image are placed in three detection frame queues, queue 1: 1, queue 2: 1, queue 3: 1, where 1 represents the detection frame.

[0122] There are 4 detection frames in the second frame image, and 2 detection frames match queue 1 and queue 2. Then put these 2 detection frames into queue 1 and queue 2, and start 2 new queues for the remaining 2 detection frames. If there is no detection frame in queue 3, queue 3 will be filled with empty data, and finally, queue 1: 11, queue 2: 11, queue 3: 10, queue 4: 1, queue 5: 1. Among them, 0 represents empty data.

[0123] In another possible implementation, step S224 may include the following steps:

[0124] S2241`: Calculate the overlap between each detection frame of the currently processed image and the tail detection frame of each detection frame queue.

[0125] S2242`: Sort the overlaps that meet the second preset condition, and put the detection frames with high overlaps into the corresponding detection frame queue.

[0126] S2243`: Put the detection frames whose overlap does not meet the second preset condition into a new detection frame queue.

[0127] S2244`: Determine in turn whether there is a new detection frame added to each detection frame queue; if not, execute step S2245`.

[0128] S2245`: Put an empty data into the detection box queue.

[0129] Take an example to illustrate the implementation method. In the above example, three detection frames of the first frame image are placed in three detection frame queues, and the second frame image has four detection frames. In this implementation method, all the overlaps are first calculated, that is, 4 detection frames multiplied by 3 detection frame queues, and 12 overlaps are calculated. The overlaps that meet the second preset condition are sorted. If there are 6 overlaps that meet the second preset condition, first put the corresponding detection frame with the highest overlap into the corresponding detection frame queue. If the corresponding detection frame with the second highest overlap has been placed in the previous detection frame queue or the corresponding detection frame queue with the second highest overlap has been placed in the detection frame, it means that the detection frame or detection frame queue with the second highest overlap has been used, then the detection frame or detection frame queue with the second highest overlap will not be processed. If the detection frame and the detection frame queue with the second highest overlap are not used, then the detection frame will be placed in the detection frame queue until the overlaps that meet the second preset condition are processed.

[0130] Steps S2243` to S2245` are the same as the above steps S2243 to S2245 and will not be repeated here.

[0131] Optionally, in the target detection frame queue that meets the first preset condition, the position area corresponding to the last detection frame in the target detection frame queue can be marked as a breast nodule.

[0132] Furthermore, in order to make the breast nodule detection model more accurate in pre-marking breast nodules, in a possible implementation, see Figure 4 , Figure 5 , Figure 4 A schematic diagram of the structure of a breast nodule detection model provided by an embodiment of the present invention, Figure 5 A schematic diagram of a flow chart of training a breast nodule detection model provided in an embodiment of the present invention, wherein the breast nodule detection model 30 may include a first basic detection model 31 and a first video feature fusion module 32, and the data detection method further includes a step of training the breast nodule detection model, including the following steps:

[0133] S310: Use the retained image data to train the first basic detection model to obtain a mature first basic detection model; the basic detection model includes a first backbone network 311 and a first detection head module 312.

[0134] The retained image data refers to some existing image sample data.

[0135] Optionally, for the retained image data, a mosaic data augmentation method can be used to randomly crop and scale multiple retained image data and then splice them together to increase the model's ability to detect image boundaries and small objects.

[0136] Optionally, the first basic detection model can be a yolov9 model, which has higher sensitivity and lower false detection rate in the retained image data test, and because it is a single-stage detection model, the overall single image reasoning speed can be very fast.

[0137] S320: Obtain an ultrasound video sample set; the ultrasound video sample set includes multiple ultrasound videos, and the ultrasound videos include real annotation frames of breast nodules.

[0138] The ultrasound video sample set may be sample data of ultrasound examination, and breast nodules have been truly marked by professionals such as doctors.

[0139] S330: extracting sample images of each frame of each ultrasound video, and extracting a preset number of sample images in the same ultrasound video whose image similarity is lower than a preset threshold as a batch.

[0140] In each sample frame of an ultrasound video, a group of sample images with image similarity lower than a preset threshold are extracted as a batch for training. Multiple frames with different image similarities are used as a batch input model for training. The features of these frames are fused to make it easier for the model to learn the dynamic changes of breast nodules in the video.

[0141] Optionally, the preset threshold may be set to 0.7, and the preset number may be set to 16.

[0142] S340: Input the batch into the first backbone network to obtain a preset number of first feature matrices.

[0143] For example, if the batch size is 16, the batch is input into the first backbone network of the yolov9 model to obtain 16 first feature matrices.

[0144] The first backbone network is the key part responsible for feature extraction, extracting useful features from the input image. These features can then be used for object detection tasks.

[0145] S350: Input each first feature matrix into a first video feature fusion module to obtain a preset number of second feature matrices that have undergone fusion processing.

[0146] The first video feature fusion module fuses the features of each first feature matrix, which enables the breast nodule detection model to better identify the appearance and disappearance of nodules in the breast ultrasound video.

[0147] S360: Input each second feature matrix into the first detection head module, and obtain the prediction box information of each sample image through upsampling operation and linear layer.

[0148] S370: Based on the prediction box information and the annotation box, the loss information is calculated, and the breast nodule detection model is iteratively trained according to the loss information until the preset conditions are met.

[0149] Optionally, the calculation formula for the image similarity in step S330 may be:

[0150]

[0151] in, is the pixel mean of image x, is the pixel mean of image y, is the pixel variance of image x, is the pixel variance of image y, is the pixel covariance of image x and image y, c1 and c2 are constants, optionally, c1=1×10 -4 , c2=9×10-4 .

[0152] In a possible implementation, in order to better capture the process of nodule changes in the breast ultrasound video, the above step S350 may include the following steps:

[0153] S3501: Concatenate the first feature matrices according to the channel dimension to obtain a concatenated feature matrix.

[0154] S3502: Synchronously fusing the concatenated feature matrices through a convolution operation to obtain a fused feature matrix, and performing channel pooling on the concatenated feature matrix to obtain a compressed feature vector.

[0155] S3503: Multiply the compressed feature vector and the fused feature matrix to obtain a weight feature matrix for changing the region weight.

[0156] S3504: Divide the weight feature matrix into a preset number of second feature matrices.

[0157] After each first feature matrix is ​​input into the first video feature fusion module, a splicing operation is first performed to splice all frame image feature matrices according to the channel dimension, and then all output results are subjected to a convolution operation to fuse the feature matrices of all channels together to obtain a fused feature matrix. The spliced ​​feature matrix is ​​synchronously channel pooled and feature compressed. The feature vector after channel pooling is multiplied by the fused feature matrix, which can strengthen the feature weight of the meaningful area and reduce the feature weight of the meaningless area. Then the feature matrix is ​​divided into batch size, such as the 16 parts mentioned above.

[0158] Optionally, the prediction box information may include coordinate information of the prediction box and the prediction box category, and the calculation formula of the loss information may be:

[0159]

[0160] in, , and are the weights of the three loss functions respectively.

[0161]

[0162] in, is the probability that the detection box is a nodule category, is a parameter used to reduce the proportion of simple sample loss; in this embodiment, there is only one breast nodule category, so .

[0163]

[0164] in, is the predicted box probability, is the intersection-over-union ratio between the predicted box and the labeled box, is a parameter used to reduce the proportion of simple sample loss; in this embodiment, there is only one breast nodule category, so .

[0165]

[0166]

[0167]

[0168] in, is the prediction box, is a callout box. is the intersection-over-union ratio between the annotation box and the prediction box, is the square of the Euclidean distance between the center point of the annotation box and the center point of the prediction box, is the square of the shortest diagonal distance between the annotation box and the prediction box, and is the width and height of the annotation box. and It is the width and height corresponding to the prediction box.

[0169] Furthermore, the present invention also provides a method for training a breast nodule detection model. Figure 6 , Figure 7 , Figure 6 A schematic diagram of the structural block diagram of a breast nodule detection model training system provided by an embodiment of the present invention, Figure 7 A flow chart of a method for training a breast nodule detection model provided in an embodiment of the present invention is applied to a breast nodule detection model training system 40, wherein the breast nodule detection model training system 40 includes a second basic detection model 41 and a second video feature fusion module 42. The execution body of the method can be the above-mentioned Figure 1 The electronic device 100 shown in FIG. 1 includes: Figure 7 The following steps are described:

[0170] S410: Use the retained image data to train the basic detection model to obtain a mature basic detection model; the basic detection model includes a second backbone network 411 and a second detection head module 412.

[0171] The retained image data refers to some existing image sample data.

[0172] Optionally, for the retained image data, a mosaic data augmentation method can be used to randomly crop and scale multiple retained image data and then splice them together to increase the model's ability to detect image boundaries and small objects.

[0173] Optionally, the second basic detection model can be a yolov9 model, which has higher sensitivity and lower false detection rate in the retained image data test, and because it is a single-stage detection model, the overall single image reasoning speed can be very fast.

[0174] S420: Obtain an ultrasound video sample set; the ultrasound video sample set includes multiple ultrasound videos, and the ultrasound videos include real annotation frames of breast nodules.

[0175] The ultrasound video sample set may be sample data of ultrasound examination, and breast nodules have been truly marked by professionals such as doctors.

[0176] S430: extracting sample images of each frame of each ultrasound video, and extracting a preset number of sample images in the same ultrasound video whose image similarity is lower than a preset threshold as a batch.

[0177] In each sample frame of an ultrasound video, a group of sample images with image similarity lower than a preset threshold are extracted as a batch for training. Multiple frames with different image similarities are used as a batch input model for training. The features of these frames are fused to make it easier for the model to learn the dynamic changes of breast nodules in the video.

[0178] Optionally, the preset threshold may be set to 0.7, and the preset number may be set to 16.

[0179] S440: Input the batch into the second backbone network to obtain a preset number of first feature matrices.

[0180] For example, if the batch size is 16, the batch is input into the second backbone network of the yolov9 model to obtain 16 first feature matrices.

[0181] The second backbone network is the key part responsible for feature extraction, extracting useful features from the input image. These features can then be used for object detection tasks.

[0182] S450: Input each first feature matrix into a second video feature fusion module to obtain a preset number of second feature matrices that have undergone fusion processing.

[0183] The second video feature fusion module fuses the features of each first feature matrix, which enables the breast nodule detection model to better identify the appearance and disappearance of nodules in the breast ultrasound video.

[0184] S460: Input each second feature matrix into the second detection head module, and obtain the prediction box information of each sample image through upsampling operation and linear layer.

[0185] S470: Based on the prediction box information and the annotation box, the loss information is calculated, and the breast nodule detection model is iteratively trained according to the loss information until the preset conditions are met.

[0186] This method uses multiple frames with different image similarities as a batch input model for training, and fuses the features of these frames to make it easier for the model to learn the dynamic changes of breast nodules in the video, and better identify the appearance and disappearance of nodules in breast ultrasound videos. Compared with target detection methods based on images only, this method can effectively reduce false detections in breast ultrasound videos.

[0187] Optionally, the calculation formula for the image similarity in step S430 may be:

[0188]

[0189] in, is the pixel mean of image x, is the pixel mean of image y, is the pixel variance of image x, is the pixel variance of image y, is the pixel covariance of image x and image y, c1 and c2 are constants, optionally, c1=1×10 -4 , c2=9×10 -4 .

[0190] In a possible implementation, in order to better capture the process of nodule changes in the breast ultrasound video, the above step S450 may include the following steps:

[0191] S4501: Concatenate the first feature matrices according to the channel dimension to obtain a concatenated feature matrix.

[0192] S4502: Synchronously fusing the concatenated feature matrices through a convolution operation to obtain a fused feature matrix, and performing channel pooling on the concatenated feature matrix to obtain a compressed feature vector.

[0193] S4503: Multiply the compressed feature vector and the fused feature matrix to obtain a weight feature matrix for changing the regional weight.

[0194] S4504: Divide the weight feature matrix into a preset number of second feature matrices.

[0195] After each first feature matrix is ​​input into the second video feature fusion module, a splicing operation is first performed to splice all frame image feature matrices according to the channel dimension, and then all output results are subjected to a convolution operation to fuse the feature matrices of all channels together to obtain a fused feature matrix. The spliced ​​feature matrix is ​​simultaneously channel pooled and feature compressed. The feature vector after channel pooling is multiplied by the fused feature matrix, which can strengthen the feature weight of the meaningful area and reduce the feature weight of the meaningless area. Then the feature matrix is ​​divided into batch size, such as the 16 parts mentioned above.

[0196] Optionally, the prediction box information may include coordinate information of the prediction box and the prediction box category, and the calculation formula of the loss information may be:

[0197]

[0198] in, , and are the weights of the three loss functions respectively.

[0199]

[0200] in, is the probability that the detection box is a nodule category, is a parameter used to reduce the proportion of simple sample loss; in this embodiment, there is only one breast nodule category, so .

[0201]

[0202] in, is the predicted box probability, is the intersection-over-union ratio between the predicted box and the labeled box, is a parameter used to reduce the proportion of simple sample loss; in this embodiment, there is only one breast nodule category, so .

[0203]

[0204]

[0205]

[0206] in, is the prediction box, is the annotation box, is the intersection-over-union ratio between the annotation box and the prediction box, is the square of the Euclidean distance between the center point of the annotation box and the center point of the prediction box, is the square of the shortest diagonal distance between the annotation box and the prediction box, and is the width and height corresponding to the annotation box, and are the width and height corresponding to the predicted box.

[0207] In summary, a data detection method, a breast nodule detection model training method and an electronic device provided by an embodiment of the present invention, by inputting the ultrasound video to be detected into the trained breast nodule detection model, the detection frame information of each frame image of the ultrasound video to be detected is obtained, and each detection frame of each frame image is placed in a plurality of detection frame queues according to the position relationship. For each detection detection frame queue, it is determined whether the detection frame queue is a target detection frame queue that meets the first preset condition. If so, the position area corresponding to the detection frame in the target detection frame queue is marked as a breast nodule. Compared with the target detection method with only an image, the false detection on the breast ultrasound video can be effectively reduced. At the same time, through the video frame feature fusion module, the detection model obtained by training can more accurately detect the process of nodule changes in the breast ultrasound video, and better identify the situation when the nodule appears and disappears in the breast ultrasound video. At the same time, the method adopts a single-stage target detection model, so the overall detection speed can meet the real-time detection of breast ultrasound nodules. Through the subsequent multi-frame filtering algorithm, false detection can be further reduced, and the nodule-like tissue structure that is difficult to distinguish on a single frame can be filtered.

[0208] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0209] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0210] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a computer-readable storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0211] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0212] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A data detection method, characterized in that: include: Inputting the ultrasound video to be detected into the trained breast nodule detection model to obtain detection frame information of each frame image of the ultrasound video to be detected; The detection frame information includes the number and position information of the detection frames, and the detection frames are pre-marked for breast nodules; Placing the detection frames of the images of each frame into a plurality of detection frame queues according to the position relationship, including: placing the detection frames of the image of the first frame into the detection frame queues of the corresponding number respectively; processing the images except the image of the first frame in sequence, and judging whether the number of detection frames in each detection frame queue is equal to the preset number; if so, deleting the head detection frame of the detection frame queue; placing the detection frames of the image being processed currently into the corresponding detection frame queues according to the position relationship between the detection frames of the image being processed currently and the tail detection frames of the detection frame queues; For each of the detection frame queues, determining whether the detection frame queue is a target detection frame queue that satisfies a first preset condition; the first preset condition is that the number of detection frames in the detection frame queue is greater than one half of the length of the detection frame queue; If yes, the position area corresponding to the detection frame in the target detection frame queue is marked as a breast nodule; If not, the detection box in the target detection box queue is deleted without marking; The method of placing each detection frame of the image being processed into a corresponding detection frame queue according to the positional relationship between each detection frame of the image being processed and the tail detection frame of each detection frame queue comprises: determining in sequence whether the overlap between each detection frame of the image being processed and the tail detection frame of each detection frame queue meets a second preset condition; if so, placing the detection frame into the corresponding detection frame queue; if not, placing the detection frame into a new detection frame queue; determining in sequence whether each detection frame queue has a new detection frame to be placed; if not, placing an empty data into the detection frame queue; or, Calculate the degree of overlap between each detection frame of the image currently being processed and the detection frame at the end of each detection frame queue; sort the overlaps that meet the second preset condition, and put the detection frames with high overlaps into the corresponding detection frame queues; put the detection frames whose overlaps do not meet the second preset condition into a new detection frame queue; determine in turn whether each detection frame queue has a new detection frame to be put in; if not, put an empty data into the detection frame queue.

2. The method according to claim 1, characterized in that The breast nodule detection model includes a first basic detection model and a first video feature fusion module. The method also includes a training step of the breast nodule detection model, including: The first basic detection model is trained using the retained image data to obtain a mature first basic detection model; the first basic detection model includes a first backbone network and a first detection head module; Acquire an ultrasound video sample set; the ultrasound video sample set includes a plurality of ultrasound videos, and the ultrasound video includes a real annotation frame of a breast nodule; Extracting sample images of each frame of each ultrasound video, and extracting a preset number of sample images in the same ultrasound video whose image similarity is lower than a preset threshold as a batch; Inputting the batch into the first backbone network to obtain the preset number of first feature matrices; Inputting each of the first feature matrices into the first video feature fusion module to obtain the preset number of second feature matrices after fusion processing; Input each of the second feature matrices into the first detection head module, and obtain the prediction box information of each of the sample images through upsampling operation and linear layer; Based on the prediction box information and the annotation box, loss information is calculated, and the breast nodule detection model is iteratively trained according to the loss information until a preset condition is met.

3. The method according to claim 2, characterized in that The calculation formula of the image similarity is: in, is the pixel mean of image x, is the pixel mean of image y, is the pixel variance of image x, is the pixel variance of image y, is the pixel covariance of image x and image y, c1 and c2 are constants.

4. The method according to claim 2, characterized in that: The step of inputting each of the first feature matrices into the first video feature fusion module to obtain the preset number of second feature matrices after fusion processing includes: Concatenate the first feature matrices according to the channel dimension to obtain a concatenated feature matrix; Simultaneously fusing the spliced ​​feature matrices through a convolution operation to obtain a fused feature matrix, and performing channel pooling on the spliced ​​feature matrix to obtain a compressed feature vector; Multiplying the compressed feature vector by the fused feature matrix to obtain a weight feature matrix for changing the regional weight; The weight feature matrix is ​​divided into the preset number of second feature matrices.

5. The method according to claim 2, characterized in that: The prediction box information includes the coordinate information of the prediction box and the prediction box category. The calculation formula of the loss information is: in, , and They are the weights of the three loss functions respectively; in, is the probability that the detection box is a nodule category, is the parameter used to reduce the proportion of simple sample loss; in, is the predicted box probability, is the intersection-over-union ratio between the predicted box and the labeled box, is the parameter used to reduce the proportion of simple sample loss; in, is the prediction box, is the annotation box, is the intersection-over-union ratio between the annotation box and the prediction box, is the square of the Euclidean distance between the center point of the annotation box and the center point of the prediction box, is the square of the shortest diagonal distance between the annotation box and the prediction box, and is the width and height corresponding to the annotation box, and are the width and height corresponding to the predicted box.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

Images