A method, device, equipment and storage medium for tracking cranial brain lesions in ultrasound images
By using the craniocerebral feature extraction module and similarity matrix calculation in ultrasound dynamic images, efficient and automated tracking of craniocerebral lesions in neonatal dynamics is achieved, and the problems of low accuracy and instability in the existing technology are solved, and the prediction of lesion location is improved Accuracy and stability.
Patent Information
- Application Number
- CN202411088880.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-09
AI Technical Summary
The prior art is difficult to efficiently and automatically track neonatal cranial brain lesions in ultrasound images, resulting in poor accuracy and instability.
By obtaining the reference image marked with the lesion area, the previous frame image and the image to be tracked from the ultrasound dynamic image, the feature vector of the image block is obtained by using the cranial feature extraction module, the similarity matrix between the feature vectors is calculated, and the similarity matrix is predicted based on the similarity matrix. The location of the lesion is combined and the location is fusion to achieve accurate tracking of the lesion.
It realizes rapid and accurate tracking of craniocerebral lesions, improves the prediction accuracy and stability of lesions location, and avoids the shortcomings of a single prediction result.
Smart Images

Figure CN118628532B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for tracking cranial and brain lesions in ultrasound images. Background Art
[0002] In recent years, with the improvement of medical technology, the survival rate of premature infants and low-weight newborns has increased year by year, but this is accompanied by an increase in the incidence and detection rate of neonatal craniocerebral injury. As a non-invasive imaging examination method, brain ultrasound examination is of great significance for the diagnosis of neonatal craniocerebral diseases. However, due to factors such as skull artifacts and blood flow obstruction, the clarity of ultrasound images may be limited, which brings difficulties to doctors' diagnosis. In addition, cranial ultrasound examination involves multiple scanning directions, and the relationship between lesions and normal structures needs to be considered comprehensively. There is currently no efficient automatic tracking method for lesions. With the rapid development of artificial intelligence technology, deep learning has achieved remarkable results in the field of medical image analysis, especially self-supervised learning methods, which can use the structure and feature differences of the image itself for learning without relying on a large amount of manually labeled data. However, there is currently no effective method for tracking craniocerebral lesions in dynamic ultrasound images, and the tracking results obtained are not stable enough and have poor accuracy. Summary of the invention
[0003] To this end, the embodiments of the present application provide a method, device, equipment and storage medium for tracking cranial and cerebral lesions in ultrasound images, thereby achieving accurate tracking of the location of cranial and cerebral lesions.
[0004] In a first aspect, the present application provides a method for tracking cranial and brain lesions in ultrasound images.
[0005] This application is achieved through the following technical solutions:
[0006] A method for tracking cranial lesions in ultrasound images, the tracking method comprising:
[0007] Determine an image to be tracked, a previous frame of the image to be tracked, and a reference image from an ultrasonic dynamic image of a cranial lesion; the reference image is marked with a lesion area;
[0008] Inputting the image to be tracked, the previous frame image and the reference image into a trained cranial brain feature extraction module to divide them into a number of image blocks, and obtaining a feature vector corresponding to the image block of each image;
[0009] Calculating a first similarity matrix between a feature vector of the image to be tracked and a feature vector of a reference image, and determining a first predicted position of a lesion in the image to be tracked based on the first similarity matrix and a lesion region in the reference image;
[0010] Calculating a second similarity matrix between a feature vector of the image to be tracked and a feature vector of a previous frame of image, and determining a second predicted position of the lesion in the image to be tracked based on the second similarity matrix and the lesion region in the previous frame of image;
[0011] The first predicted position and the second predicted position are fused to obtain the predicted lesion position of the image to be tracked.
[0012] In a preferred example of the present application, it can be further configured that the image to be tracked, the previous frame image and the reference image are input into a trained cranial brain feature extraction module, and the step of obtaining the feature vector corresponding to each image includes:
[0013] Input the image to be tracked, the previous frame image and the reference image as input images into a trained cranial brain feature extraction module;
[0014] The cranial brain feature extraction module divides the input image into N image blocks of the same size. The cranial brain feature extraction module analyzes all image blocks through the encoder encoder-vit to obtain the The dimension corresponding to the image block is The eigenvector of , and Represent the height and width of the input image respectively, represents the size of the image block, is the embedding dimension of encoder encoder-vit.
[0015] In a preferred example of the present application, it can be further configured that the brain feature extraction module is obtained based on MAE model training.
[0016] In a preferred example of the present application, it can be further configured that the MAE model training step includes:
[0017] A public ultrasound dynamic image set is obtained, the original images in the ultrasound dynamic image set are masked, and the masked images are input into the MAE model. The MAE model uses an encoder to extract image features, and the decoder reconstructs the masked image based on the image features and mask marks to obtain a reconstructed image. The mean square error between the pixels of the reconstructed image and the original image is used as the loss function to optimize the MAE model. When the preset performance indicators are reached, the trained MAE model is used as a cranial and brain feature extraction module.
[0018] In a preferred example of the present application, it can be further configured that the step of calculating a first similarity matrix between a feature vector of the image to be tracked and a feature vector of the reference image comprises:
[0019] Calculate the first cosine similarity between the feature vector of each image block in the image to be tracked and the feature vector of each image block in the reference image; and construct a first similarity matrix based on the first cosine similarity.
[0020] In a preferred example of the present application, it can be further configured that the step of calculating the second similarity matrix between the feature vector of the image to be tracked and the feature vector of the previous frame image includes:
[0021] Calculate the second cosine similarity between the feature vector of each image block in the image to be tracked and the feature vector of each image block in the previous frame image;
[0022] A second similarity matrix is constructed based on the second cosine similarity.
[0023] ,
[0024] in, Represents the cosine similarity between the feature vectors of the mth image block of the i-th frame image and the nth image block of the j-th frame image, Represents the feature vector of the mth image block of the i-th frame image, Represents the feature vector of the nth image block of the jth frame image, represents the inner product, Represents the modular product of two eigenvectors.
[0025] In a preferred example of the present application, it can be further set that the calculation formula of the first cosine similarity or the second cosine similarity is:
[0026] In a preferred example of the present application, it can be further set as follows:
[0027] The step of determining a first predicted position of a lesion in the image to be tracked based on the first similarity matrix and the lesion area in the reference image comprises:
[0028] Determining whether a first cosine similarity in the first similarity matrix is greater than a preset similarity threshold, and if the first cosine similarity is greater than the preset similarity threshold, determining that the two image blocks corresponding to the first cosine similarity belong to the same type of tissue;
[0029] A first target image block in the image to be tracked that belongs to the same tissue as the image block of the lesion area in the reference image is searched to obtain a first moving trajectory of the first target image block, and a first predicted position is generated based on the first moving trajectory.
[0030] In a preferred example of the present application, it can be further configured that the step of determining the second predicted position of the lesion in the image to be tracked based on the second similarity matrix includes:
[0031] Determining whether the second cosine similarity in the second similarity matrix is greater than a preset similarity threshold, and if the second cosine similarity is greater than the preset similarity threshold, determining that the two image blocks corresponding to the second cosine similarity belong to the same type of tissue;
[0032] A second target image block in the image to be tracked that belongs to the same tissue as the image block of the lesion area in the previous frame of image is searched, a second movement trajectory of the second target image block is counted, and a second predicted position is generated based on the second movement trajectory.
[0033] In a preferred example of the present application, it can be further configured that the step of fusing the first predicted position and the second predicted position to obtain the predicted lesion position of the image to be tracked includes:
[0034] The step of fusing the first predicted position and the second predicted position to obtain the predicted lesion position of the image to be tracked includes:
[0035] The first predicted position and predicted location of the second lesion Perform weighted superposition to generate the predicted lesion location of the image to be tracked :
[0036] ,
[0037] in, is the weight coefficient of the first predicted position, , is the weight coefficient of the second predicted position, , .
[0038] In a second aspect, the present application provides a device for tracking cranial and cerebral lesions in ultrasound images.
[0039] This application is achieved through the following technical solutions:
[0040] A tracking device for cranial brain lesions in an ultrasound image, the tracking device is used to execute the tracking method for cranial brain lesions in an ultrasound image described in the first aspect, the tracking device comprising:
[0041] An image acquisition module, used to determine the image to be tracked, the previous frame image of the image to be tracked and a reference image from the ultrasonic dynamic image of the cranial brain lesion; the reference image is marked with the lesion area;
[0042] A feature extraction module, used to obtain feature vectors corresponding to image blocks of the image to be tracked, the previous frame of the image to be tracked, and the reference image;
[0043] A similarity calculation module, used for calculating a first similarity matrix between a feature vector of the image to be tracked and a feature vector of a reference image, and calculating a second similarity matrix between a feature vector of the image to be tracked and a feature vector of a previous frame of image;
[0044] a position prediction module, configured to determine a first predicted position of a lesion in the image to be tracked based on the first similarity matrix, and to determine a second predicted position of the lesion in the image to be tracked based on the second similarity matrix and a lesion region in a previous frame of image;
[0045] The position fusion module is used to fuse the first predicted position and the second predicted position to obtain the predicted lesion position of the image to be tracked.
[0046] On the third aspect, the present application is realized through the following technical solutions:
[0047] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any one of the above-mentioned methods for tracking cranial and brain lesions in ultrasound images are implemented.
[0048] In a fourth aspect, the present application provides a computer-readable storage medium.
[0049] This application is achieved through the following technical solutions:
[0050] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps of any one of the above-mentioned methods for tracking cranial brain lesions in ultrasound images.
[0051] In summary, compared with the prior art, the technical solution provided in the embodiment of the present application has at least the following beneficial effects:
[0052] The present application obtains a reference image, a previous frame image and an image to be tracked with the lesion area marked from the ultrasonic dynamic image of the cranial and cerebral lesions; obtains the feature vectors corresponding to the image blocks of the above-mentioned images by using the cranial and cerebral feature extraction module; further calculates the first similarity matrix between the feature vector of the image to be tracked and the feature vector of the reference image, and obtains the first predicted position of the image to be tracked based on the first similarity matrix; calculates the second similarity matrix between the feature vector of the image to be tracked and the feature vector of the previous frame image, and obtains the second predicted position of the image to be tracked based on the second similarity matrix and the lesion area in the previous frame image; finally, the first predicted position and the second predicted position are fused to obtain the predicted lesion position of the image to be tracked. The present application can only mark the lesion position in the reference image, and the model can automatically obtain two predicted positions based on the reference image, the previous frame image and the image to be tracked, and further fuse the two predicted positions to obtain the final predicted lesion position, which can realize automatic fast and accurate tracking of the lesion position. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 A schematic diagram of a flow chart of a method for tracking cranial brain lesions in ultrasound images provided in one embodiment of the present application;
[0054] Figure 2 A schematic diagram of the structure of a tracking device for cranial and cerebral lesions in ultrasound images provided in one embodiment of the present application;
[0055] Figure 3 A schematic structural diagram of a device for tracking cranial and cerebral lesions in ultrasound images provided in yet another embodiment of the present application;
[0056] Description of reference numerals:
[0057] Image acquisition module-01, feature extraction module-02, similarity calculation module-03, first similarity calculation unit-031, second similarity calculation unit-032, position prediction module-04, first trajectory prediction unit-041, second trajectory prediction unit-042, position fusion module-05. DETAILED DESCRIPTION
[0058] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make modifications to the present embodiment without any creative contribution as needed, but such modifications are protected by the patent law as long as they are within the scope of the claims of the present application.
[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0060] In addition, the term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.
[0061] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with basically the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on quantity and execution order.
[0062] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.
[0063] S10: Determine a reference image, a previous frame image and an image to be tracked from the ultrasonic dynamic image of the cranial brain lesion; wherein the reference image is marked with the lesion area.
[0064] The patient's brain is scanned in multiple scanning directions using an ultrasonic device, and the dynamic image is previewed on the ultrasonic device. An image containing a cranial lesion and having a high image quality is selected as an ultrasonic dynamic image, and the ultrasonic dynamic image is converted into a digital image file format for export. The ultrasonic dynamic image is opened using image processing software, such as medical image processing software, AdobePhotoshop software, or ImageJ software, and a frame of image is determined as a reference image, and annotation tools such as Pair software, ITK-SNAP software, and 3D Slicer software are used to annotate the edge of the lesion area in the reference image. Specifically, the ultrasound doctor can operate the software to manually annotate or automatically annotate through the software, and then the ultrasound doctor can verify to ensure the accuracy of the lesion area. Among them, the reference image is an image with a lesion area that needs to be tracked, which is used for subsequent lesion tracking and comparison; the image to be tracked is an image that needs to analyze the change in the position of the lesion; the previous frame image is the previous frame image of the image to be tracked, and is an image between the time series frames of the reference image and the image to be tracked.
[0065] S20: Input the image to be tracked, the previous frame image and the reference image into the trained brain feature extraction module to divide them into several image blocks, and obtain the feature vector corresponding to the image block of each image.
[0066] Among them, the cranial feature extraction module is trained based on the MAE (Masked AutoEncoders) model. The reference image, the previous frame image and the image to be tracked are input into the trained MAE model. The MAE model first divides each input image into several image blocks to obtain a high-dimensional abstract feature vector with the same dimension. The MAE model learns a large amount of image data during the training process, which makes the feature vectors it extracts have a certain generalization ability. Even when faced with new and unseen images to be tracked, the MAE model can extract effective feature vectors, thereby achieving accurate tracking of the lesion area.
[0067] S30: Calculate a first similarity matrix between the feature vector of the image to be tracked and the feature vector of the reference image, and determine a first predicted position of the lesion in the image to be tracked based on the first similarity matrix.
[0068] The first similarity matrix is composed of the first cosine similarity between the feature vector of the image to be tracked and the feature vector of the reference image. According to the first similarity matrix, it is determined which areas of the image to be tracked and the areas marked as lesions in the reference image belong to the same type of tissue, and then the first predicted position of the lesion area in the previous frame image is determined according to the lesion area in the reference image.
[0069] S40: Calculate a second similarity matrix between the feature vector of the image to be tracked and the feature vector of the previous frame image, and determine a second predicted position of the lesion in the image to be tracked based on the second similarity matrix and the lesion region in the previous frame image.
[0070] The second similarity matrix is composed of the second cosine similarity between the feature vector of the image to be tracked and the feature vector of the previous frame image. According to the second similarity matrix, it is determined which areas in the image to be tracked and the lesion area in the previous frame image belong to the same tissue, and then the second predicted position in the image to be tracked is determined according to the lesion area in the previous frame image. Among them, the lesion area in the previous frame image is a predicted lesion position obtained based on the reference image and the image of the previous frame of the previous frame image.
[0071] Compared with directly using the pixel information of the original image, using high-dimensional abstract feature vectors for calculation and matching can improve matching accuracy while reducing computational complexity and improving tracking efficiency. This is because the high-dimensional feature vectors have effectively compressed and encoded the image, making it possible to use less resources and time to complete the calculation.
[0072] S50: Fusing the first predicted position and the second predicted position to obtain the predicted lesion position of the image to be tracked.
[0073] Considering that a single prediction result may have slight errors due to noise, interference or limitations of the model itself, by fusing the first predicted position and the second predicted position, the effective information of the two prediction results can be integrated to make up for the shortcomings of a single prediction result, thereby improving the prediction accuracy of the lesion location.
[0074] In some embodiments, the step of determining the image to be tracked, the previous frame image of the image to be tracked, and the reference image from the ultrasonic dynamic image of the cranial brain lesion, and marking the lesion area in the reference image includes: dividing the ultrasonic dynamic image into ultrasonic sequence images; obtaining the first frame image in the ultrasonic sequence image as the reference image, and marking the lesion area in the reference image. Specifically, firstly, the scanned ultrasonic dynamic image is divided into ultrasonic sequence images according to the time sequence. , where n is the number of all images in the ultrasound sequence. Get the first frame image As the reference image, the image to be tracked is , the previous frame of the image to be tracked As the previous frame image. The present application obtains the predicted position of a lesion in an image to be tracked based on a reference image, obtains the predicted position of a lesion in an image to be tracked based on the previous frame image, fuses the two predicted positions, and determines the position of the lesion in the final image to be tracked, which can more comprehensively capture the dynamic changes of the lesion and further improve the prediction accuracy of the lesion position; at the same time, it can avoid the problem of missing lesions due to single image prediction, and effectively improve the accuracy and stability of the prediction results.
[0075] In some embodiments, the step of using the cranial brain feature extraction module to obtain a feature vector corresponding to the image includes: inputting a reference image, a previous frame image, and an image to be tracked as input images into a trained cranial brain feature extraction module;
[0076] The cranial brain feature extraction module divides the input image into N image blocks of the same size. The cranial brain feature extraction module uses the encoder encoder-vit to analyze all the image blocks to obtain the corresponding The dimension of an image block is The eigenvector of , and Represent the height and width of the input image respectively, represents the size of the image block, is the embedding dimension of encoder encoder-vit. Specifically, first divide the input reference image, the previous frame image and the image to be tracked into N The image blocks are of size d, and the image blocks do not overlap. At the same time, the brain feature extraction module will add position coding information to each image block, and then input the image blocks with position coding information into the encoder. The encoder uses a multi-head attention mechanism to analyze the semantic information and position information between image blocks, and outputs a high-dimensional feature vector with a dimension of d for each image block. The brain feature extraction module can extract useful information such as texture features from image blocks and filter out redundant information of the original image. For example, the input size is The reference image, the previous frame image and the image to be tracked are divided into 256 frames of size The image blocks are then encoded with positions ranging from 0 to 255 in order from left to right in each row. The position codes are used to locate the movement trajectory of the image blocks in the subsequent step.
[0077] In some embodiments, the pre-training process of the MAE model includes: obtaining a public ultrasound dynamic image set, dividing the obtained ultrasound dynamic image set into a training set and a verification set according to a ratio of 8:2. Masking the original image in the ultrasound dynamic image set in the training set, inputting the mask image into the MAE model, the MAE model extracts image features using an encoder, and the decoder reconstructs the mask image based on the image features and the mask mark to obtain a reconstructed image, and the mean square error between the pixels of the reconstructed image and the original image is used as a loss function to optimize the MAE model, and when the preset performance index is reached, the trained MAE model is used as a cranial feature extraction module. Specifically, first, a public data set containing cranial ultrasound images is obtained from a medical image database, and the public data set covers various morphological and pathological changes of the cranial brain, thereby improving the generalization ability of the model. The original image is divided into image blocks, and the image blocks in the original image are randomly set to preset mask values, such as 0 or 1, according to a certain mask ratio. The masking process is to enable the MAE model to learn from the features of the remaining image information and then restore the complete image. The mask image is input into the MAE model, and the MAE model calls the encoder encoder-vit to extract features from the visible part of the mask image. Then the image features and mask tags are input into the decoder together. The decoder reconstructs the complete image based on the above information. The difference between the pixel values of the reconstructed image and the original image is calculated, and the mean square error (MSE) is used as the loss function. The parameters of the MAE model are optimized through the back propagation algorithm, such as the learning rate, image block size, embedding vector space dimension, latent variable dimension, number of Transformer Blocks, etc., to minimize the loss function. When the accuracy of the model on the validation set reaches more than 95%, the training is stopped. The trained MAE model is used for subsequent feature extraction. This makes the constructed cranial feature extraction module more robust, able to more comprehensively capture the dynamic changes of the lesions, and effectively improve the accuracy and stability of the prediction results.
[0078] In some embodiments, before the image to be extracted features is input into the cranial feature extraction module, the method further includes:
[0079] The target area image containing the lesion area is intercepted with the same rectangular sampling frame, and the target area image is subjected to noise reduction processing. Specifically, the cropping function of image processing software such as OpenCV and PIL is used to first determine the approximate location of the lesion area, and a rectangular sampling frame that can completely contain the lesion area is set to intercept the target area image. Further, the wavelet transform method is used to set a suitable threshold to remove the noise component in the target area image. This can provide high-quality image input for subsequent cranial brain feature extraction and improve the tracking accuracy of the lesion location.
[0080] In some embodiments, the step of calculating the first similarity matrix between the feature vector of the image to be tracked and the feature vector of the reference image includes: calculating the first cosine similarity between the feature vector of each image block in the image to be tracked and the feature vector of each image block in the reference image; and constructing the first similarity matrix based on the first cosine similarity. The image to be tracked is divided into 256 Image blocks , , … , and their corresponding eigenvectors are expressed as , , … ; The reference image is divided into 256 Image blocks , , … , the feature vectors corresponding to each image block are , , … . An image patch in the image to be tracked Take the following example to illustrate the calculation of the image block of the image to be tracked. The eigenvector of The 256 cosine similarities of all 256 image block feature vectors of the reference image are used to construct a cosine similarity matrix based on the 256 cosine similarities:
[0081] ;
[0082] Among them, represents the similarity matrix, Image patch representing the image to be tracked and the image patch of the reference image The first cosine similarity of Image patch representing the image to be tracked and the image patch of the reference image The first cosine similarity of Image patch representing the image to be tracked and the image patch of the reference image The first cosine similarity of Image patch representing the image to be tracked and the image patch of the reference image The first cosine similarity.
[0083] In some embodiments, the step of determining a first predicted position of a lesion in the image to be tracked based on the first similarity matrix includes:
[0084] Determine whether the first cosine similarity in the first similarity matrix is greater than a preset similarity threshold. If the first cosine similarity is greater than the preset similarity threshold, determine that the two image blocks corresponding to the first cosine similarity belong to the same type of tissue; find the first target image block in the image to be tracked that belongs to the same type of tissue as the image block of the lesion area in the reference image, obtain the first moving trajectory of the first target image block, and generate a first predicted position based on the first moving trajectory. Among them, the preset similarity threshold refers to the threshold of cosine similarity, which can be adjusted according to the prediction target accuracy. Preferably, the preset similarity threshold θ=0.9 in this application. For example, the first cosine similarity of the feature vector of the image block whose position is encoded as 156 in the image to be tracked and the feature vector of the image block whose position is encoded as 163 in the reference image is , It is considered that the two image blocks belong to the same type of tissue, and the image block with position code 163 in the reference image and the image block with position code 156 in the image to be tracked are image blocks of the same tissue type. Determine that the image blocks of the lesion area in the image to be tracked and the reference image belong to all the first target image blocks of the same type of tissue. Based on the first target image block and the lesion area, the movement trajectory of the lesion area can be determined, and then the first predicted position of the lesion area in the image to be tracked can be determined. .
[0085] In some embodiments, in the step of calculating the second similarity matrix between the feature vector of the image to be tracked and the feature vector of the previous frame image: calculating the second cosine similarity between the feature vector of each image block in the image to be tracked and the feature vector of each image block in the previous frame image; and constructing the second similarity matrix based on the second cosine similarity. The image to be tracked is divided into 256 Image blocks , , … , and its corresponding eigenvector is expressed as , , … ;Will The previous frame image is divided into 256 Image blocks , , … , and their corresponding eigenvectors are expressed as , , … Among them, based on an image block in the image to be tracked Take the following example to illustrate the calculation of the image block of the image to be tracked. The eigenvector of The 256 cosine similarities of the feature vectors of all 256 image blocks of the previous frame image are used to construct a similarity matrix based on the 256 cosine similarities, which is expressed as:
[0086] ;
[0087] in represents the similarity matrix, Image patch representing the image to be tracked and the image block of the previous frame The second cosine similarity of Image patch representing the image to be tracked and the image block of the previous frame The second cosine similarity of Image patch representing the image to be tracked and the image block of the previous frame The second cosine similarity of Image patch representing the image to be tracked and the image block of the previous frame The second cosine similarity.
[0088] In some embodiments, the step of determining a second predicted position of the lesion in the image to be tracked based on the second similarity matrix and the lesion region in the previous frame of image includes:
[0089] Determine whether the second cosine similarity in the second similarity matrix is greater than a preset similarity threshold. If the second cosine similarity is greater than the preset similarity threshold, determine that the two image blocks corresponding to the second cosine similarity belong to the same type of tissue; find the second target image block in the image to be tracked that belongs to the same type of tissue as the image block of the lesion area in the previous frame image, count the second movement trajectory of the second target image block, and generate a second predicted position based on the second movement trajectory. Among them, the preset similarity threshold is the threshold of cosine similarity, which can be adjusted according to the prediction target accuracy. Preferably, the preset similarity threshold θ=0.9 in the present application. For example, the second cosine similarity of the feature vector of the image block whose position is encoded as 188 in the image to be tracked and the feature vector of the image block whose position is encoded as 176 in the previous frame image is , it is considered that the image block 188 of the image to be tracked and the image block 176 of the previous frame image belong to the same type of tissue, and the image block with the position code 176 in the previous frame image and the image block with the position code 188 in the image to be tracked are image blocks belonging to the same tissue type. Determine all the second target image blocks of the image to be tracked and the previous frame image that belong to the lesion area and belong to the same type of tissue. Based on the second target image block, the movement trajectory of the lesion area can be determined, and then the second predicted position of the lesion area in the image to be tracked can be determined. .
[0090] In some embodiments, the calculation formula of the first cosine similarity or the second cosine similarity is:
[0091] ,
[0092] in, Represents the cosine similarity between the feature vectors of the mth image block of the i-th frame image and the nth image block of the j-th frame image, Represents the feature vector of the mth image block of the i-th frame image, Represents the feature vector of the nth image block of the jth frame image, represents the inner product, Represents the modular product of two eigenvectors.
[0093] In some embodiments, the step of fusing the obtained first predicted position and the second predicted position to obtain the predicted lesion position of the image to be tracked includes:
[0094] The first predicted position and predicted location of the second lesion Perform weighted superposition to generate the predicted lesion location of the image to be tracked :
[0095] ,
[0096] in, is the weight coefficient of the predicted position of the first lesion, , is the weight coefficient of the predicted position of the second lesion, , In some embodiments, , .
[0097] Another embodiment of the present application also provides a tracking device for cranial brain lesions in ultrasound images, and the tracking device is used to execute the above tracking method, such as Figure 2As shown, the tracking device includes: an image acquisition module 01, a feature extraction module 02, a similarity calculation module 03, a position prediction module 04, and a position fusion module 05.
[0098] The image acquisition module 01 is used to determine the image to be tracked, the previous frame image of the image to be tracked and the reference image from the ultrasonic dynamic image of the cranial brain lesion; the lesion area is marked in the reference image.
[0099] The feature extraction module 02 is used to obtain feature vectors corresponding to image blocks of the image to be tracked, the previous frame of the image to be tracked, and the reference image.
[0100] The similarity calculation module 03 is used to calculate a first similarity matrix between the feature vector of the image to be tracked and the feature vector of the reference image and to calculate a second similarity matrix between the feature vector of the image to be tracked and the feature vector of the previous frame image.
[0101] The position prediction module 04 is used to determine the first predicted position of the lesion in the image to be tracked based on the first similarity matrix and the lesion area in the reference image, and to determine the second predicted position of the lesion in the image to be tracked based on the second similarity matrix and the lesion area in the previous frame image.
[0102] The position fusion module 05 is used to fuse the first predicted position and the second predicted position to obtain the predicted lesion position of the image to be tracked.
[0103] In some embodiments, the feature extraction module 02 is trained by the MAE model.
[0104] In some embodiments, Figure 3 As shown, the similarity calculation module 03 also includes a first similarity calculation unit 031 and a second similarity calculation unit 032 .
[0105] The first similarity calculation unit 031 is used to calculate the first cosine similarity between the feature vector of each image block in the image to be tracked and the feature vector of each image block in the reference image; and to construct a first similarity matrix based on the first cosine similarity. The second similarity calculation unit 032 is used to calculate the second cosine similarity between the feature vector of each image block in the image to be tracked and the feature vector of each image block in the previous frame image; and to construct a second similarity matrix based on the second cosine similarity.
[0106] In some embodiments, the position prediction module 04 further includes a first trajectory prediction unit 041 and a second trajectory prediction unit 042 .
[0107] Among them, the first trajectory prediction unit 041 is used to determine whether the first cosine similarity in the first similarity matrix is greater than a preset similarity threshold. If the first cosine similarity is greater than the preset similarity threshold, it is determined that the two image blocks corresponding to the first cosine similarity belong to the same type of tissue; find the first target image block in the image to be tracked that belongs to the same type of tissue as the image block of the lesion area in the reference image, obtain the first moving trajectory of the first target image block, and generate the first predicted position based on the first moving trajectory.
[0108] The second trajectory prediction unit 042 is used to determine whether the second cosine similarity in the second similarity matrix is greater than a preset similarity threshold. If the second cosine similarity is greater than the preset similarity threshold, it is determined that the two image blocks corresponding to the second cosine similarity belong to the same type of tissue; search for a second target image block in the image to be tracked that belongs to the same type of tissue as the image block of the lesion area in the previous frame image, count the second movement trajectory of the second target image block, and generate a second predicted position based on the second movement trajectory.
[0109] The specific definition of the tracking device for cranial and cerebral lesions in ultrasound images provided in this embodiment can be found in the embodiment of the tracking method for cranial and cerebral lesions in ultrasound images above, and will not be repeated here. Each module in the above-mentioned tracking device for cranial and cerebral lesions in ultrasound images can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0110] The embodiment of the present application provides a computer device, which may include a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the processor executes the steps of the tracking method for cranial brain lesions in ultrasound images as in any of the above embodiments.
[0111] The working process, working details and technical effects of the computer device provided in this embodiment can be found in the above embodiment of the method for tracking cranial and brain lesions in ultrasound images, which will not be described in detail here.
[0112] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for tracking cranial brain lesions in ultrasound images in any of the above embodiments are implemented. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive and / or a memory stick, etc. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
[0113] The working process, working details and technical effects of the computer-readable storage medium provided in this embodiment can be found in the above embodiment of the method for tracking cranial and brain lesions in ultrasound images, and will not be described in detail here.
[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0115] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0116] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system described in the present application is divided into different functional units or modules to complete all or part of the functions described above.
Claims
1. A method for tracking cranial brain lesions in ultrasound images, characterized in that: The tracking method includes: Determine an image to be tracked, a previous frame of the image to be tracked, and a reference image from an ultrasonic dynamic image of a cranial lesion; the reference image is marked with a lesion area; Inputting the image to be tracked, the previous frame of the image to be tracked and the reference image into a trained cranial brain feature extraction module to divide them into a number of image blocks, and obtaining a feature vector corresponding to the image block of each image; Calculating a first similarity matrix between a feature vector of the image to be tracked and a feature vector of a reference image, and determining a first predicted position of a lesion in the image to be tracked based on the first similarity matrix and a lesion region in the reference image; Calculating a second similarity matrix between a feature vector of the image to be tracked and a feature vector of a previous frame of the image to be tracked, and determining a second predicted position of the lesion in the image to be tracked based on the second similarity matrix and a lesion region in the previous frame of the image to be tracked; The first predicted position and the second predicted position are merged to obtain a predicted lesion position of the image to be tracked; Wherein, determining a first predicted position of a lesion in the image to be tracked based on the first similarity matrix and the lesion region in the reference image includes: judging whether a first cosine similarity in the first similarity matrix is greater than a preset similarity threshold, and if the first cosine similarity is greater than the preset similarity threshold, determining that two image blocks corresponding to the first cosine similarity belong to the same type of tissue; Searching for a first target image block in the image to be tracked that is of the same tissue as the image block of the lesion area in the reference image, obtaining a first moving trajectory of the first target image block, and generating a first predicted position based on the first moving trajectory; Determining a second predicted position of the lesion in the image to be tracked based on the second similarity matrix and the lesion region in the previous frame of the image to be tracked, comprising: judging whether a second cosine similarity in the second similarity matrix is greater than a preset similarity threshold, and if the second cosine similarity is greater than the preset similarity threshold, determining that two image blocks corresponding to the second cosine similarity belong to the same type of tissue; Searching for a second target image block in the image to be tracked that is of the same tissue as the image block of the lesion area in the previous frame of the image to be tracked, obtaining a second moving trajectory of the second target image block, and generating a second predicted position based on the second moving trajectory; The calculation formula of the first cosine similarity or the second cosine similarity is: , in, Represents the cosine similarity between the feature vectors of the mth image block of the i-th frame image and the nth image block of the j-th frame image, Represents the feature vector of the mth image block of the i-th frame image, Represents the feature vector of the nth image block of the jth frame image, represents the inner product, Represents the modular product of two eigenvectors.
2. The method for tracking cranial brain lesions in ultrasonic images according to claim 1, characterized in that: The steps of inputting the image to be tracked, the previous frame image of the image to be tracked and the reference image into a trained cranial brain feature extraction module to divide them into a plurality of image blocks, and obtaining the feature vectors corresponding to the image blocks of each image include: The image to be tracked, the previous frame of the image to be tracked and the reference image are input as input images into the trained cranial brain feature extraction module; The cranial brain feature extraction module divides the input image into N image blocks of the same size. The cranial brain feature extraction module analyzes all image blocks through the encoder encoder-vit to obtain the The dimension corresponding to the image block is The eigenvector of ; where , and Represent the height and width of the input image respectively, represents the size of the image block, is the embedding dimension of encoder encoder-vit.
3. The method for tracking cranial brain lesions in ultrasonic images according to claim 2, characterized in that: The brain feature extraction module is obtained based on MAE model training.
4. The method for tracking cranial brain lesions in ultrasonic images according to claim 3, characterized in that: The MAE model training steps include: A public ultrasound dynamic image set is obtained, the original images in the ultrasound dynamic image set are masked, and the masked images are input into the MAE model. The MAE model uses an encoder to extract image features, and the decoder reconstructs the masked image based on the image features and mask marks to obtain a reconstructed image. The mean square error between the pixels of the reconstructed image and the original image is used as the loss function to optimize the MAE model. When the preset performance indicators are reached, the trained MAE model is used as a cranial and brain feature extraction module.
5. The method for tracking cranial brain lesions in ultrasonic images according to claim 1, characterized in that: The step of calculating a first similarity matrix between the feature vector of the image to be tracked and the feature vector of the reference image comprises: Calculating the first cosine similarity between the feature vector of each image block in the image to be tracked and the feature vector of each image block in the reference image; A first similarity matrix is constructed based on the first cosine similarity.
6. The method for tracking cranial brain lesions in ultrasonic images according to claim 5, characterized in that: The step of calculating a second similarity matrix between a feature vector of the image to be tracked and a feature vector of a previous frame of the image to be tracked comprises: Calculating the second cosine similarity between the feature vector of each image block in the image to be tracked and the feature vector of each image block in the previous frame image of the image to be tracked; A second similarity matrix is constructed based on the second cosine similarity.
7. The method for tracking cranial brain lesions in ultrasonic images according to claim 1, characterized in that: The step of fusing the first predicted position and the second predicted position to obtain the predicted lesion position of the image to be tracked includes: The first predicted position and predicted location of the second lesion Perform weighted superposition to generate the predicted lesion location of the image to be tracked : , in, is the weight coefficient of the first predicted position, , is the weight coefficient of the second predicted position, , .
8. A tracking device for cranial and cerebral lesions in ultrasound images, characterized in that: The tracking device is used to execute the tracking method according to any one of claims 1 to 7, and the tracking device includes: An image acquisition module, used to determine the image to be tracked, the previous frame image of the image to be tracked and a reference image from the ultrasonic dynamic image of the cranial brain lesion; the reference image is marked with the lesion area; A feature extraction module, used to obtain feature vectors corresponding to image blocks of the image to be tracked, the previous frame of the image to be tracked, and the reference image; A similarity calculation module, used for calculating a first similarity matrix between a feature vector of the image to be tracked and a feature vector of a reference image, and calculating a second similarity matrix between a feature vector of the image to be tracked and a feature vector of a previous frame of the image to be tracked; a position prediction module, configured to determine a first predicted position of a lesion in the image to be tracked based on the first similarity matrix, and to determine a second predicted position of the lesion in the image to be tracked based on the second similarity matrix and a lesion region in a previous frame of the image to be tracked; A position fusion module, used for fusing the first predicted position and the second predicted position to obtain a predicted lesion position of the image to be tracked; Among them, the position prediction module is used to determine whether the first cosine similarity in the first similarity matrix is greater than a preset similarity threshold, if the first cosine similarity is greater than the preset similarity threshold, determine that the two image blocks corresponding to the first cosine similarity belong to the same type of tissue, search for a first target image block in the image to be tracked that belongs to the same type of tissue as the image block of the lesion area in the reference image, obtain a first moving trajectory of the first target image block, and generate a first predicted position based on the first moving trajectory; and to determine whether the second cosine similarity in the second similarity matrix is greater than a preset similarity threshold, if the second cosine similarity is greater than the preset similarity threshold, determine that the two image blocks corresponding to the second cosine similarity belong to the same type of tissue; search for a second target image block in the image to be tracked that belongs to the same type of tissue as the image block of the lesion area in the previous frame of the image to be tracked, obtain a second moving trajectory of the second target image block, and generate a second predicted position based on the second moving trajectory; The calculation formula of the first cosine similarity or the second cosine similarity in the position prediction module is: , in, Represents the cosine similarity between the feature vectors of the mth image block of the i-th frame image and the nth image block of the j-th frame image, Represents the feature vector of the mth image block of the i-th frame image, Represents the feature vector of the nth image block of the jth frame image, represents the inner product, Represents the modular product of two eigenvectors.
9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
