A method, system, electronic device and storage medium for classifying and processing ultrasound image lesion attributes based on video sequences

Abstract

The present invention discloses a method, system, electronic device and storage medium for classifying and processing ultrasound image lesion attributes based on video sequences, and belongs to the field of intelligent medical diagnosis technology. The method avoids the method of directly judging the attribute category by using a target detector. In order to adapt to attribute classification scenarios in different situations, the lesion and the location of the lesion are first determined, and the tracking sequence of the lesion target under different frames is obtained. Then, according to the lesion target attribute category and the overall confidence corresponding to the lesion target attribute category, the lesion attribute is classified, the diagnosis accuracy on difficult attributes is improved, the doctor's diagnosis process is imitated, and the multi-frame result information is integrated to give the final diagnosis result, thereby improving the accuracy of the lesion attribute diagnosis result.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent medical diagnosis, and in particular, relates to a method, system, electronic device and storage medium for classifying and processing ultrasound image lesion attributes based on video sequences. Background Art

[0002] Ultrasound image lesion attribute classification refers to further determining the attributes of lesions located at specific locations, such as the orientation attributes of thyroid nodules, which include vertical and horizontal; the posterior echo attributes, which include echo enhancement and echo uniformity. Determining the attributes of lesions is of great significance for doctors to give specific diagnosis results of lesions in the later stage.

[0003] At present, most automatic ultrasound image diagnosis technologies are generally based on single-frame image recognition. This working method requires doctors to find the best observation angle, fix the ultrasound image by interacting with the ultrasound device, and then start the automatic diagnosis software to identify the lesion. Obviously, this method has a strong dependence on the doctor's scanning technique. The better the scanning angle, the higher the accuracy of the diagnosis. In addition, when diagnosing lesions, doctors often need to comprehensively observe the information of the lesions from different angles to give a diagnosis result in order to improve the accuracy of the diagnosis. However, this method cannot integrate the scanning information from different angles, and the judgment given is somewhat one-sided. At present, some technologies have made partial improvements, such as increasing the speed of the model so that the algorithm can diagnose on the scanned image in real time and give a diagnosis result. Although this method can give a diagnosis result in real time, due to the number of training data and the difference in fitting of the model itself, the model has a certain error rate, which makes the diagnosed attribute classification unstable, which will cause certain interference to the doctor's diagnosis result. Summary of the invention

[0004] In view of the problem that the attribute categories output by the current diagnostic software have errors, which lead to the impact of diagnostic results, the present invention avoids the method of using a target detector to directly judge the attribute category. In order to adapt to attribute classification scenarios in different situations, the lesion and the location of the lesion are first determined, and the tracking sequence of the lesion target in different frames is obtained. Then, the lesion attributes are classified according to the lesion target attribute category and the overall confidence corresponding to the lesion target attribute category. This improves the diagnostic accuracy of difficult attributes, imitates the doctor's diagnostic process, integrates multi-frame result information, and gives a final diagnostic result, thereby improving the accuracy of the lesion attribute diagnosis result.

[0005] In order to achieve the above object, the present invention adopts the following technical solution.

[0006] A first aspect of the present invention provides a method for classifying and processing ultrasound image lesion attributes based on a video sequence, the method comprising:

[0007] S102: obtaining an ultrasound image to be tested, and extracting one or more lesion images in the ultrasound image to be tested as lesion targets;

[0008] S104: Tracking the lesion target in the ultrasound image to be tested, and obtaining multiple tracking sequences corresponding to the lesion target;

[0009] S106: Acquire attribute information of the lesion target in the tracking sequence, and add the attribute information of the lesion target to the tracking sequence;

[0010] S108: Acquire the attribute information of the current lesion target in the tracking sequence, calculate the overall confidence corresponding to the lesion target in the tracking sequence; determine the attribute category of the lesion target according to the overall confidence, and output it.

[0011] As a preferred solution, step S102 includes:

[0012] The target detection model is used to extract one or more lesion images in the ultrasound image to be tested as lesion targets, and the lesion location and lesion type of the lesion target are obtained; the lesion detection model is trained using labeled bounding box samples.

[0013] As a preferred solution, step S106 includes:

[0014] Using a classifier model to identify the attribute category of the lesion target in the tracking sequence, and calculating the target attribute category confidence corresponding to the attribute category of the lesion target;

[0015] The attribute confidence of the lesion target in the ultrasound image, the bounding box of the lesion target and the attribute information category of the lesion target are all added to the tracking sequence.

[0016] As a preferred solution, the step S108 further includes:

[0017] According to the confidence of high attribute classification of lesion target, the difference of attribute categories of the previous and next frames in ultrasound image, and the overlap of bounding boxes of the previous and next frames in ultrasound image, an overall confidence mathematical model is constructed;

[0018] The lesion target attribute category and the overall confidence corresponding to the lesion target attribute category are calculated.

[0019] As a preferred solution, the overall confidence is compared with a first threshold and a second threshold, wherein the first threshold is greater than the second threshold;

[0020] If the overall confidence is greater than a first threshold, placing the lesion target attribute category corresponding to the overall confidence in an output queue;

[0021] If the overall confidence is less than the first threshold but greater than the second threshold, retaining the lesion target attribute category corresponding to the overall confidence in the tracking sequence;

[0022] If the overall confidence is less than a second threshold, the lesion target attribute category corresponding to the overall confidence is deleted from the tracking sequence.

[0023] As a preferred solution, the overall confidence mathematical model is constructed as follows:

[0024]

[0025] Among them, w j Represents the overall confidence of the lesion target correspondence in the tracking sequence; The value indicating whether the lesion target appears continuously in the tracking sequence; Represents the confidence weight of the lesion target attribute; represents the lesion target overlap weight; γ represents the weight bias.

[0026] A second aspect of the present invention provides a system for classifying and processing ultrasound image lesion attributes based on video sequences, comprising:

[0027] A lesion recognition module, which is used to obtain an ultrasonic image to be tested, and extract one or more lesion images in the ultrasonic image to be tested as lesion targets;

[0028] A lesion tracking module, which is used to track the lesion target in the ultrasonic image to be tested and obtain multiple tracking sequences corresponding to the lesion target;

[0029] An attribute recognition module, which is used to obtain attribute information of the lesion target in the tracking sequence and add the attribute information of the lesion target to the tracking sequence;

[0030] The judgment module is used to obtain the attribute information of the current lesion target in the tracking sequence, calculate the overall confidence corresponding to the lesion target in the tracking sequence; determine the attribute category of the lesion target according to the overall confidence, and output it.

[0031] As a preferred solution, the judgment module includes a first judgment unit, a second judgment unit and a third judgment unit;

[0032] A first judgment unit is used to compare the overall confidence with a first threshold, wherein the first threshold is greater than a second threshold; if the first threshold is greater than the first threshold, placing the lesion target attribute category corresponding to the overall confidence in an output queue;

[0033] a second judgment unit, configured to compare the overall confidence with a first threshold and a second threshold, and if the overall confidence is less than the first threshold but greater than the second threshold, retaining the lesion target attribute category corresponding to the overall confidence in the tracking sequence;

[0034] The third judgment unit is used to compare the overall confidence with a second threshold value, and if the overall confidence is less than the second threshold value, delete the lesion target attribute category corresponding to the overall confidence from the tracking sequence.

[0035] A third aspect of the present invention provides an electronic device, comprising a processor, an input device, an output device and a memory, wherein the processor, input device, output device and memory are connected in sequence, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method as described above.

[0036] A fourth aspect of the present invention provides a readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method as described above.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The embodiment of the present invention avoids the method of directly judging the attribute category by using the target detector. In order to adapt to the attribute classification scenarios of different situations, the lesion and the location of the lesion are first determined, the tracking sequence of the lesion target in different frames is obtained, and then the lesion attribute is classified according to the lesion target attribute category and the confidence corresponding to the lesion target attribute category, so as to improve the diagnosis accuracy on difficult attributes, imitate the doctor's diagnosis process, integrate the multi-frame result information, give the final diagnosis result, and improve the accuracy of the lesion attribute diagnosis result; in addition, in the process of constructing the confidence mathematical model, firstly, the confidence factor of the classifier is taken into account, so that the target with high attribute classification confidence contributes more, and secondly, the bounding box modeling process considers that the morphological change of the target in two consecutive frames should not be large. If it is too large, the probability that the two belong to the same target is lower, and the contribution to the overall confidence is lower; thirdly, if the targets in the previous and next frames do not belong to the same attribute category, the probability of correct classification of this attribute is lower, and the contribution of the overall confidence should be reduced. Through these three points, the stability and continuity of the confidence of the final tracking sequence are guaranteed, and the accuracy of the classification result can be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other purposes, features and advantages of the present application will become more apparent by describing the embodiments of the present application in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps. In the accompanying drawings:

[0040] Figure 1 A flow chart of a method for classifying and processing ultrasound image lesion attributes based on video sequences provided by an embodiment of the present invention;

[0041] Figure 2 An ultrasonic image of a thyroid nodule provided by an embodiment of the present invention;

[0042] Figure 3 A block diagram of a system for classifying and processing ultrasound image lesion attributes based on video sequences provided by an embodiment of the present invention;

[0043] Figure 4 A block diagram of an electronic device according to an embodiment of the present application is illustrated;

[0044] Figure 5 A structural block diagram of an improved classifier structure provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0046] Exemplary Methods

[0047] like Figure 1 As shown, this example discloses a method for classifying and processing ultrasound image lesion attributes based on a video sequence, the method comprising the following steps:

[0048] S102: Obtain an ultrasonic image to be tested, and extract one or more lesion images in the ultrasonic image to be tested as lesion targets.

[0049] Specifically, the ultrasound image to be measured in this example utilizes an ultrasound beam to scan the human body, and the reflected signal is received and processed to obtain an image of an organ or a specific area in the body. The ultrasound image to be measured may be an ultrasound image of the thyroid gland, breast and other parts, and the ultrasound image may be acquired by an ultrasound-based medical imaging device, such as reading a video stream from an ultrasound device display interface, and decoding the video stream into continuous thyroid ultrasound images frame by frame. It may also be acquired in non-real time, for example, by receiving a thyroid ultrasound image pre-stored in a server or other device, or by receiving a thyroid ultrasound image transmitted from other devices. Figure 2 Shown is a thyroid nodule, which has a hypoechoic appearance.

[0050] Extracting the lesion image of the ultrasound image to be tested uses a target detection model based on deep learning to first identify the lesion category in the ultrasound image and then accurately locate the lesion. The target detection model here is, for example, a detection network model such as yolo-v5, faster-rcnn, etc. It should be noted that this example requires a bounding box as a subsequent calculation of the overall confidence, so it is necessary to train using ultrasound image samples with labeled bounding boxes. For example, a thyroid nodule detector needs to use a large number of samples containing thyroid nodule bounding boxes to train the neural network so that the detector has the ability to detect thyroid nodules.

[0051] S104: Tracking the lesion target in the ultrasonic image to be detected, and obtaining multiple tracking sequences corresponding to the lesion target.

[0052] Specifically, a target tracker is used here to track the lesion target, and the target tracker is used to track the multiple lesion targets obtained. At this time, the lesion target becomes the tracking target of the target tracker, and the trajectory of the tracking target within the time t1 to t2 and the tracking target result are obtained to obtain the tracking queue. It should be understood that the target detector is used to detect whether the predicted position of the next frame is the target, and then the new detection result is used to update the training set and then update the target detector. When training the target detector, the target area is generally selected as the positive sample, and the surrounding area of ​​the target is the negative sample. Of course, the closer the area to the target is, the greater the possibility of being a positive sample.

[0053] Here, target trackers such as IOU and KCF are used to track the detected lesion targets. When the IOU single target tracker is used, it mainly judges whether the two belong to the same target based on the overlap of the bounding boxes of the two targets. When the KCF single target tracker is used, it mainly judges whether the two belong to the same target based on the similarity of the target features in the bounding boxes of the two targets. At the same time, multiple different lesions may be detected in an ultrasound image. The single target tracker can determine whether the multiple lesion targets detected in the previous and next frames are the same lesion target during continuous ultrasound scanning.

[0054] S106: Acquire the attribute information of the lesion target in the tracking sequence, and add the attribute information of the lesion target to the tracking sequence.

[0055] Specifically, a classifier model is used here to identify the attribute category of the latest lesion target in the tracking sequence, and the target attribute category confidence corresponding to the attribute category of the lesion target is calculated; the classifier here can adopt mainstream target classification networks such as resnet, vgg, inception, etc.; the attribute confidence of the lesion target in the ultrasound image, the bounding box of the lesion target and the attribute information category of the lesion target are added to the tracking sequence.

[0056] S108: Acquire the attribute information of the current lesion target in the tracking sequence, calculate the overall confidence corresponding to the lesion target in the tracking sequence; determine the attribute category of the lesion target according to the overall confidence, and output it.

[0057] Specifically, an overall confidence mathematical model is pre-constructed here, and then the overall confidence corresponding to the lesion target in the tracking sequence is calculated, and the attribute category of the lesion target is judged according to the size of the overall confidence, so as to output the attribute category of the current lesion target. It should be understood that the judgment based on the size of the overall confidence can be achieved through a single threshold or through multiple threshold controls, thereby improving the output accuracy of the attribute category of the current lesion target.

[0058] This example can adapt to attribute classification scenarios in different situations. It first determines the lesion and the location of the lesion, obtains the tracking sequence of the lesion target in different frames, and then classifies the lesion attributes according to the lesion target attribute category and the overall confidence corresponding to the lesion target attribute category. It improves the diagnostic accuracy of difficult attributes, imitates the doctor's diagnosis process, integrates multi-frame result information, gives the final diagnosis result, and improves the stability of the lesion attribute diagnosis result. The difficult attributes here refer to some attributes that are difficult to determine the attribute type from a single ultrasound image, and need to be judged by integrating multi-frame information, that is, combining the tracking sequence of the lesion image and the corresponding attribute characteristics, such as internal calcification attributes. The calcification point cannot be observed at some angles, and different angles are required for observation and judgment.

[0059] As a preferred implementation, this example controls the accuracy of the attribute category of the lesion target by setting multiple groups of thresholds.

[0060] Firstly, the influence of the attribute information of the lesion target on the overall confidence is considered, and a mathematical model of the overall confidence is constructed according to the high attribute classification confidence of the lesion target, the different attribute categories of the front and back frames in the ultrasound image, and the overlap of the bounding boxes of the front and back frames in the ultrasound image; the attribute category of the lesion target and the overall confidence corresponding to the attribute category of the lesion target are calculated.

[0061] Specifically, assume that in step S106, the target tracker outputs N tracking sequences corresponding to the lesion target, T = {t 1 ,t 2 ,…,t N}, T represents the target tracking sequence, and the number of targets in each target tracking sequence is D = {d 1 ,d 2 ,…,d N According to the results of the target detector and the target classifier, each target in the target tracking sequence has three kinds of information: lesion target attribute category C j , lesion target bounding box b j , lesion target attribute category confidence P j , where j is the jth target in a tracking sequence.

[0062] The confidence mathematical model construction process is considered as follows:

[0063] (1) Considering that the higher the confidence of the target attribute category, the higher the accuracy of the target attribute category judgment will be, the logarithm is used to transform the confidence. Since the confidence cannot be a negative value, the result is taken as the maximum value.

[0064]

[0065] in, represents the confidence weight of the lesion target attribute, ɑ is a fixed weight value, P j Represents the confidence of the lesion target attribute category.

[0066] (2) Considering the change rate of the bounding box of the lesion target in several consecutive video frames, it should not be very large. The IOU is the intersection over union of two bounding boxes, which calculates the ratio of the intersection and union of two bounding boxes. It is mainly used to quantify the overlap of the bounding boxes of the target in the previous and next frames. β is a fixed weight value, and the sum of ɑ and β is 1.

[0067]

[0068] b j-1 Represents the coordinate information of the j-1th target bounding box in the tracking sequence of the lesion target (x 1 ,y 1 ,x 2 ,y 2 ), x 1 ,y 1 Indicates the horizontal and vertical coordinates of the upper left corner of the bounding box, x 2 ,y 2 Indicates the horizontal and vertical coordinates of the lower right corner of the bounding box; b j Represents the coordinate information of the jth target bounding box in the lesion target sequence; Represents the lesion target overlap weight.

[0069] (3) Considering that the attribute categories of the lesion targets that appear continuously in consecutive video frames are the same, they are more likely to be the expected output attribute categories, while the probability that the target attributes that appear continuously are the expected output attributes is low. Therefore, the following method is used to distinguish the contributions of the two.

[0070]

[0071] c j-1 represents the attribute category of the j-1th target in the target sequence, An assignment indicating whether the lesion target appears continuously in the tracking sequence.

[0072] If an attribute category does not appear continuously in the next frame, the overall confidence will be reduced by a fixed value γ.

[0073] Combining the above three points, the first confidence modeling takes into account the confidence factor of the classifier, so that the target with high attribute classification confidence contributes more. The second bounding box modeling process considers that the shape change of the target in two consecutive frames should not be large. If it is too large, the probability that the two belong to the same target will be lower, and the contribution to the overall confidence will be lower. Third, if the targets in the previous and next frames do not belong to the same attribute category, the probability of correct classification of this attribute will be lower, and the contribution to the overall confidence should be reduced. These three points ensure the stability and continuity of the confidence of the final tracking sequence, which can greatly improve the accuracy of the classification results.

[0074] The overall confidence model of the target sequence is as follows:

[0075]

[0076] Among them, w j Represents the overall confidence of the lesion target correspondence in the tracking sequence; The value indicating whether the lesion target appears continuously in the tracking sequence; Represents the confidence weight of the lesion target attribute; represents the lesion target overlap weight; γ represents the weight bias.

[0077] It should be noted that after the confidence of different attributes is calculated, the attribute category with the largest overall confidence is taken for output. After the overall confidence is output, a threshold needs to be set for comparison. In this example, the first threshold thr can be set specifically according to the attribute category of the lesion target, such as 5, 6, 7 or 8; the second threshold is 0.

[0078] comparing the overall confidence with a first threshold and a second threshold, wherein the first threshold is greater than the second threshold;

[0079] If it is greater than the first threshold, the lesion target attribute category corresponding to the overall confidence is placed in the output queue; and the result is output.

[0080] If the overall confidence is less than the first threshold but greater than the second threshold, the lesion target attribute category corresponding to the overall confidence is retained in the tracking sequence, so as to facilitate the subsequent overall confidence calculation.

[0081] If the overall confidence is less than a second threshold, the lesion target attribute category corresponding to the overall confidence is deleted from the tracking sequence.

[0082] Specifically, through the above overall confidence calculation, the attribute target of category 0 has not appeared for multiple consecutive frames, and the attribute confidence of this category is less than 0; the attribute target of category 1 has appeared for three consecutive frames, and the attribute confidence of this category is 5.1; the attribute target of category 2 has not appeared for three consecutive frames, and the attribute confidence of this category is 1.2. After overall judgment, the confidence of category 1 is the highest and is greater than the threshold value of 3.0, so it is output and displayed. The attribute of category 0 is less than 0, so all targets of this attribute in this queue are deleted. The confidence of the target of category 2 is less than the threshold value of 3.0, but greater than 0, so it is still retained in the queue; after processing, the first target sequence queue is updated to {1,1,2,1,1,1}.

[0083] As another preferred implementation, the classifier model in this example is different from the traditional classifier. The traditional classifier can only classify a single attribute. Here, it is a multi-task classifier improved on the basis of the traditional classifier, that is, it can classify different attributes at the same time. For example, nodule echo and nodule edge are two attributes. The nodule echo is subdivided into high echo, low echo and no echo, and the nodule boundary is divided into clear and unclear.

[0084] like Figure 5 As shown in the figure, the specific classifier structure is improved as follows, where CNN is the feature extraction network of the recognition network, which is the same as the feature extraction network of the traditional deep classifier, and FC is the fully connected layer, which is the same as the fully connected layer of the traditional deep classifier. Finally, the category is output. If two attributes are to be identified, such as nodule boundary and nodule echo. The classifier before the improvement requires two separate CNN networks, which are trained separately using their respective attribute data. The improved classifier only needs one CNN network, which can be trained using two attribute data at the same time. In this way, the CNN network of the improved classifier can learn the information of the two data. Experiments have also proved that with the same amount of data, the classification accuracy of the improved classifier on each attribute is 6% higher than that of the classifier before the improvement.

[0085] In this example, the traditional classifier model is transformed into a network structure to enable it to have the function of multi-task simultaneous recognition. The advantage of this is that different tasks can promote each other, enhance the ability of the backbone network in the classifier to extract the features of the image, and achieve mutual promotion and improvement of the recognition ability of the target.

[0086] Exemplary Systems

[0087] like Figure 3 As shown, a system for classifying and processing ultrasound image lesion attributes based on video sequences is characterized by comprising:

[0088] A lesion recognition module 20 is used to obtain an ultrasonic image to be tested, and extract one or more lesion images in the ultrasonic image to be tested as lesion targets;

[0089] A lesion tracking module 30, which is used to track the lesion target in the ultrasonic image to be detected, and obtain multiple tracking sequences corresponding to the lesion target;

[0090] An attribute recognition module 40, which is used to obtain attribute information of the lesion target in the tracking sequence and add the attribute information of the lesion target to the tracking sequence;

[0091] The judgment module 50 is used to obtain the attribute information of the current lesion target in the tracking sequence, calculate the overall confidence corresponding to the lesion target in the tracking sequence; determine the attribute category of the lesion target according to the overall confidence, and output it.

[0092] The judgment module 50 includes a first judgment unit, a second judgment unit and a third judgment unit;

[0093] A first judgment unit is used to compare the overall confidence with a first threshold, wherein the first threshold is greater than a second threshold; if the first threshold is greater than the first threshold, placing the lesion target attribute category corresponding to the overall confidence in an output queue;

[0094] a second judgment unit, configured to compare the overall confidence with a first threshold and a second threshold, and if the overall confidence is less than the first threshold but greater than the second threshold, retaining the lesion target attribute category corresponding to the overall confidence in the tracking sequence;

[0095] The third judgment unit is used to compare the overall confidence with a second threshold value, and if the overall confidence is less than the second threshold value, delete the lesion target attribute category corresponding to the overall confidence from the tracking sequence.

[0096] In order to adapt to attribute classification scenarios in different situations, this example first determines the lesion and its location, obtains the tracking sequence of the lesion target in different frames, and then classifies the lesion attributes according to the lesion target attribute category and the confidence corresponding to the lesion target attribute category, thereby improving the diagnostic accuracy of difficult attributes, imitating the doctor's diagnosis process, integrating multi-frame result information, and giving the final diagnosis result, thereby improving the stability of the lesion attribute diagnosis result.

[0097] Exemplary Electronic Devices

[0098] Below, reference Figure 4The electronic device according to the embodiment of the present application is described. The electronic device can be the mobile device itself, or a stand-alone device independent of the mobile device, which can communicate with the mobile device to receive the collected input signals from the mobile device and send the selected target decision behavior to the mobile device.

[0099] Figure 4 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0100] like Figure 4 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0101] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0102] The memory 12 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the decision-making behavior decision-making method of each embodiment of the present application described above and / or other desired functions.

[0103] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). For example, the input device 13 may include various devices such as cameras, CT, MRI, and ultrasound imaging equipment. The input device 13 may also include, for example, a keyboard, a mouse, etc. The output device 14 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0104] Of course, to simplify, Figure 4 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0105] Exemplary computer program products and computer-readable storage media

[0106] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the decision-making behavior decision-making method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0107] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0108] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the decision-making behavior decision-making method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0109] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0110] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0111] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0112] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0113] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0114] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for classifying and processing ultrasound image lesion attributes based on video sequences. It is characterized in that The method comprises: S102: obtaining an ultrasound image to be tested, and extracting one or more lesion images in the ultrasound image to be tested as lesion targets; S104: Tracking the lesion target in the ultrasound image to be tested, and obtaining multiple tracking sequences corresponding to the lesion target; S106: Acquire attribute information of the lesion target in the tracking sequence, and add the attribute information of the lesion target to the tracking sequence; S108: Acquire the attribute information of the current lesion target in the tracking sequence, calculate the overall confidence corresponding to the lesion target in the tracking sequence; determine the attribute category of the lesion target according to the overall confidence, and output it; The overall confidence mathematical model is constructed as follows: Among them, w j Represents the overall confidence of the lesion target correspondence in the tracking sequence; The value indicating whether the lesion target appears continuously in the tracking sequence; Represents the confidence weight of the lesion target attribute; represents the lesion target overlap weight; γ represents the weight bias.

2. According to the method for classifying and processing ultrasound image lesion attributes based on video sequences according to claim 1, It is characterized in that The step S102 includes: The target detection model is used to extract one or more lesion images in the ultrasound image to be tested as lesion targets, and the lesion location and lesion type of the lesion target are obtained; the lesion detection model is trained using labeled bounding box samples.

3. The method for classifying and processing ultrasound image lesion attributes based on video sequences according to claim 1, It is characterized in that The step S106 includes: Using a classifier model to identify the attribute category of the lesion target in the tracking sequence, and calculating the target attribute category confidence corresponding to the attribute category of the lesion target; The attribute confidence of the lesion target in the ultrasound image, the bounding box of the lesion target and the attribute information category of the lesion target are all added to the tracking sequence.

4. The method for classifying and processing ultrasound image lesion attributes based on video sequences according to claim 3, It is characterized in that The step S108 further includes: According to the confidence of high attribute classification of lesion target, the difference of attribute categories of the previous and next frames in ultrasound image, and the overlap of bounding boxes of the previous and next frames in ultrasound image, an overall confidence mathematical model is constructed; The lesion target attribute category and the overall confidence corresponding to the lesion target attribute category are calculated.

5. The method for classifying and processing ultrasound image lesion attributes based on video sequences according to claim 3, It is characterized in that comparing the overall confidence with a first threshold and a second threshold, wherein the first threshold is greater than the second threshold; If the overall confidence is greater than a first threshold, placing the lesion target attribute category corresponding to the overall confidence in an output queue; If the overall confidence is less than the first threshold but greater than the second threshold, retaining the lesion target attribute category corresponding to the overall confidence in the tracking sequence; If the overall confidence is less than a second threshold, the lesion target attribute category corresponding to the overall confidence is deleted from the tracking sequence.

6. A system for classifying and processing ultrasound image lesion attributes based on video sequences. It is characterized in that include A lesion recognition module, which is used to obtain an ultrasonic image to be tested, and extract one or more lesion images in the ultrasonic image to be tested as lesion targets; A lesion tracking module, which is used to track the lesion target in the ultrasonic image to be tested and obtain multiple tracking sequences corresponding to the lesion target; An attribute recognition module, which is used to obtain attribute information of the lesion target in the tracking sequence and add the attribute information of the lesion target to the tracking sequence; A judgment module, which is used to obtain the attribute information of the current lesion target in the tracking sequence, calculate the overall confidence corresponding to the lesion target in the tracking sequence; determine the attribute category of the lesion target according to the overall confidence, and output it; The overall confidence mathematical model is constructed as follows: Among them, w j Represents the overall confidence of the lesion target correspondence in the tracking sequence; The value indicating whether the lesion target appears continuously in the tracking sequence; Represents the confidence weight of the lesion target attribute; represents the lesion target overlap weight; γ represents the weight bias.

7. The ultrasonic image lesion attribute classification processing system based on video sequence according to claim 6, It is characterized in that The judgment module includes a first judgment unit, a second judgment unit and a third judgment unit; A first judgment unit is used to compare the overall confidence with a first threshold, wherein the first threshold is greater than a second threshold; if the first threshold is greater than the first threshold, placing the lesion target attribute category corresponding to the overall confidence in an output queue; a second judgment unit, configured to compare the overall confidence with a first threshold and a second threshold, and if the overall confidence is less than the first threshold but greater than the second threshold, retaining the lesion target attribute category corresponding to the overall confidence in the tracking sequence; The third judgment unit is used to compare the overall confidence with a second threshold value, and if the overall confidence is less than the second threshold value, delete the lesion target attribute category corresponding to the overall confidence from the tracking sequence.

8. An electronic device, It is characterized in that The method comprises a processor, an input device, an output device and a memory, wherein the processor, the input device, the output device and the memory are connected in sequence, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 5.

9. A readable storage medium, It is characterized in that The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 5.

Images