Method, apparatus, device, and medium for determining object information of a medical image

Abstract

This disclosure describes a method, apparatus, device, and medium for determining object information in a medical image. The method includes obtaining a multi-channel image based on an image to be processed and auxiliary images; inputting the multi-channel image into an object information determination model to obtain a mask image; the object information determination model includes at least one encoder, a mapping layer, at least one decoder, and a mask determination module; the encoder is configured to take a first feature vector corresponding to the multi-channel image as input to obtain a second feature vector; the mapping layer is configured to map the second feature vector into at least one set of third feature vectors; the decoder is configured to take the third feature vector and a first category vector as input to obtain a mask vector, a second category vector, and a fourth feature vector; and the mask determination module is configured to obtain a mask image based on the fourth feature vector and the mask vector, and determine the object information of the image to be processed based on the mask image. This improves the accuracy of the determined object information.

Description

Technical Field

[0001] This disclosure generally relates to the field of medical image computing, and specifically to a method, apparatus, device, and medium for determining object information in a medical image. Background Technology

[0002] In the medical field, medical images are crucial tools for diagnosing and treating diseases. Medical images can provide detailed information about the internal structure and function of the human body. Extracting useful information from medical images, particularly identifying objects within them, is a vital task of medical image analysis.

[0003] Traditional methods primarily rely on manual annotation or selection of regions of interest to determine object information in medical images. With the development of computer technology, computer-based image processing methods have emerged. Currently, computer image processing methods generally determine object information in medical images through operations such as image enhancement, filtering, and edge detection.

[0004] However, medical images often contain uncertainties that affect image quality (such as significant noise and artifacts). General computer image processing methods for medical images mostly address specific image quality characteristics (e.g., extracting pre-defined features), resulting in poor adaptability to different image quality levels and often leading to inaccurate identification of object information. Therefore, accurately determining object information in medical images has become a pressing problem. Summary of the Invention

[0005] This disclosure is made in view of the above-mentioned state of the prior art, and its purpose is to provide a method, apparatus, device and medium for determining object information in medical images that can improve the accuracy of determined object information.

[0006] Therefore, a first aspect of this disclosure provides a method for determining object information of a medical image, comprising acquiring a to-be-processed image and an auxiliary image associated with the to-be-processed image, wherein the to-be-processed image and the auxiliary image are medical images; obtaining a multi-channel image based on the to-be-processed image and the auxiliary image; and inputting the multi-channel image into a trained object information determination model to obtain a mask image corresponding to the to-be-processed image, wherein the object information determination model includes at least one encoder, a mapping layer, at least one decoder, and a mask determination module, wherein the encoder is configured to take a first feature vector corresponding to the multi-channel image as input to obtain a second feature vector, and the mapping layer is configured to input the first feature vector corresponding to the multi-channel image to obtain a second feature vector. The second feature vector is mapped to at least one set of third feature vectors, each set of third feature vectors corresponding to one of the at least one decoders. Each of the at least one decoders is configured to take the corresponding set of third feature vectors and the first category vector as input to obtain the mask vector, the second category vector, and the fourth feature vector of each decoder. The first category vector of the first decoder in the at least one decoder is a preset category vector. The mask determination module is configured to obtain the mask image based on the fourth feature vector of the at least one decoder and the mask vector; and to determine the object information of the image to be processed based on the mask image.

[0007] In the first aspect of this disclosure, a multi-channel image is obtained based on the image to be processed and auxiliary images. This multi-channel image is then input into a trained object information determination model to obtain a mask image corresponding to the image to be processed. The object information of the image to be processed is then determined based on the mask image. In this case, the multi-channel image contains information from the auxiliary image related to the image to be processed. Processing with the multi-channel image increases the amount of information provided to the object information determination model, which helps improve the model's ability to distinguish objects in medical images, thereby improving the accuracy of the determined object information. Furthermore, by mapping a second feature vector to at least one set of third feature vectors through a mapping layer, one output of the encoder can be transformed into the input of at least one decoder, thereby improving the independence of the input to each decoder and increasing the flexibility of the structural design of the object information determination model.

[0008] Furthermore, in the method for determining object information of a medical image according to the first aspect of this disclosure, the medical image may optionally be an intravascular image. Therefore, the method of this disclosure can be used to determine object information of an intravascular image.

[0009] Furthermore, in the method for determining object information of a medical image according to the first aspect of this disclosure, optionally, the auxiliary image includes at least one frame image before the image to be processed and at least one frame image after the image to be processed, and the image to be processed, the at least one frame image before the image to be processed, and the at least one frame image after the image to be processed are combined to form the multi-channel image. In this case, processing using multi-channel images enables the object information determination model to learn the contextual information of the image to be processed, which can help improve the object information determination model's ability to distinguish objects in medical images.

[0010] Furthermore, in the method for determining object information of a medical image according to the first aspect of this disclosure, optionally, the number of frames in at least one frame before the image to be processed is 1, the number of frames in at least one frame after the image to be processed is 1, and the multi-channel image is a three-channel image; or the number of frames in at least one frame before the image to be processed is 2, the number of frames in at least one frame after the image to be processed is 2, and the multi-channel image is a five-channel image. In this case, processing with a three-channel image enables the object information determination model to learn the contextual information of the image to be processed, which is beneficial to improving the object information determination model's ability to distinguish objects in medical images. In addition, a five-channel image, compared to a three-channel image, can increase the amount of contextual information of the image to be processed contained, which is beneficial to further improving the object information determination model's ability to distinguish objects in medical images.

[0011] Furthermore, in the method for determining object information in a medical image according to the first aspect of this disclosure, optionally, the encoder includes a downsampling module, a feature extraction module, at least one self-attention layer, and an upsampling module. The downsampling module is configured to downsample the first feature vector; the feature extraction module is configured to extract features from the first feature vector to obtain a first intermediate vector, the magnitude of which is the same as the magnitude of the first feature vector; the at least one self-attention layer is configured to use the downsampled first feature vector as input to obtain a second intermediate vector; and the upsampling module is configured to use the first intermediate vector to upsample the second intermediate vector to obtain the second feature vector. In this case, downsampling the first feature vector reduces the amount of data in the first feature vector. When using the downsampled first feature vector as input to obtain the second intermediate vector using the self-attention layer, the computational load of the self-attention layer is reduced, thereby improving computational speed. Furthermore, extracting features from the first feature vector to obtain a first intermediate vector of the same size allows for the extraction of low-level features from the first feature vector. When upsampling the second intermediate vector using the first intermediate vector to obtain the second feature vector, this facilitates the restoration of detailed information from the first feature vector, thereby improving the accuracy of feature representation in the object information determination model. Additionally, the encoder utilizes at least one self-attention layer to encode long-range dependencies in the first feature vector corresponding to the multi-channel image, which is beneficial for determining objects in the medical image, thus improving the accuracy of the determined object information.

[0012] Furthermore, in the method for determining object information of a medical image according to the first aspect of this disclosure, optionally, the mapping layer includes a linear mapping layer configured to multiply the second feature vector by a mapping matrix to obtain the at least one set of third feature vectors, the number of sets of third feature vectors being the same as the number of decoders. In this case, the linear mapping preserves the linearity of the second feature vector, which is beneficial for subsequently using the obtained third feature vectors to obtain a mask image, and also improves the computational speed.

[0013] Furthermore, in the method for determining object information in a medical image according to the first aspect of this disclosure, optionally, the decoder includes at least one mutual attention layer, each set of the third feature vectors includes a key vector and a value vector, and the at least one mutual attention layer receives the corresponding set of the third feature vectors and the first category vector and outputs the decoder's mask vector, the second category vector, and the fourth feature vector. In this case, the decoder utilizes at least one mutual attention layer for decoding, enabling the acquisition of long-distance dependencies in the third feature vectors that are beneficial for determining objects in the medical image, thereby improving the accuracy of the determined object information.

[0014] Furthermore, in the method for determining object information in a medical image according to the first aspect of this disclosure, optionally, the number of decoders is multiple, and the multiple decoders are cascaded. The first category vector corresponding to the first decoder among the multiple decoders is the preset category vector, and the first category vector corresponding to the decoders other than the first one among the multiple decoders is the second category vector output by the previous decoder. The fourth feature vector output by the last decoder among the multiple decoders and all the mask vectors of the multiple decoders are used as input to the mask determination module. In this case, multiple decoders can process the output of the same encoder from different perspectives, enabling feature analysis from different angles, thereby improving the decoding capability of the object information determination model. In addition, by cascading decoders, each decoder other than the first decoder can receive the output of the previous decoder as input, so that each decoder can fuse information from different levels of previous decoders, thereby improving the decoding capability of the object information determination model and improving the accuracy of the determined object information. Furthermore, using all the mask vectors of the multiple decoders as input to the mask determination module can comprehensively utilize the output of the decoders, reduce the influence of a single decoder on the mask image, and improve the robustness of the object information determination model.

[0015] Furthermore, in the method for determining object information of a medical image according to the first aspect of this disclosure, optionally, the mask determination module is configured to: obtain a category prediction vector corresponding to the image to be processed based on the fourth feature vector output by the last decoder among the at least one decoder; fuse all the mask vectors corresponding to the at least one decoder to obtain a fused mask vector; and perform a dot product between the fused mask vector and the category prediction vector to obtain the mask image. In this case, fusing all the mask vectors corresponding to at least one decoder to obtain the fused mask vector can comprehensively utilize the outputs of the decoders, reduce the influence of a single decoder on the fused mask vector, and improve the robustness of the object information determination model. In addition, using the fourth feature vector output by the last decoder to obtain the category prediction vector corresponding to the image to be processed can reduce the computational load compared to using the fourth feature vector output by multiple decoders.

[0016] Furthermore, in the method for determining object information of a medical image according to the first aspect of this disclosure, optionally, the object information of the image to be processed is input into a parametric measurement model to obtain measurement parameters of the object information. The parametric measurement model is configured to determine a preset calculation method for the measurement parameters of the object information of the image to be processed, and to determine the measurement parameters of the object information based on the preset calculation method. In this case, the measurement parameters of the object information can be determined through the parametric measurement model, and results for further analysis of the object information (e.g., the area and diameter of the object) can be obtained.

[0017] A second aspect of this disclosure provides an apparatus for determining object information of a medical image, comprising an image acquisition module, an image processing module, and a result determination module. The image acquisition module is configured to acquire an image to be processed and an auxiliary image associated with the image to be processed, and to obtain a multi-channel image based on the image to be processed and the auxiliary image, wherein the image to be processed and the auxiliary image are medical images. The image processing module is configured to input the multi-channel image into a trained object information determination model to obtain a mask image corresponding to the image to be processed. The object information determination model includes at least one encoder, a mapping layer, at least one decoder, and a mask determination module. The encoder is configured to take a first feature vector corresponding to the multi-channel image as input to obtain a second feature vector. The mapping layer is configured to map the second feature vector into at least one set of third feature vectors. Each set of third feature vectors corresponds to one of the at least one decoders. Each decoder is configured to take the corresponding set of third feature vectors and a first category vector as input to obtain a mask vector, a second category vector, and a fourth feature vector for each decoder. The first category vector of the first decoder in the at least one decoder is a preset category vector. The mask determination module is configured to obtain the mask image based on the fourth feature vector of the at least one decoder and the mask vector. The result determination module is configured to determine the object information of the image to be processed based on the mask image. In this case, the image acquisition module can obtain multi-channel images containing information from auxiliary images related to the image to be processed. Processing with multi-channel images increases the amount of information provided to the object information determination model, which helps improve the model's ability to distinguish objects in medical images, thereby improving the accuracy of the determined object information. Furthermore, by mapping the second feature vector to at least one set of third feature vectors through a mapping layer, one output of the encoder can be transformed into the input of at least one decoder, thereby improving the independence of the input of each decoder and increasing the flexibility of the structural design of the object information determination model.

[0018] A third aspect of this disclosure provides an electronic device including a processor and a memory, the processor executing a program stored in the memory to implement the method for determining object information of a medical image as described in the first aspect of this disclosure.

[0019] A fourth aspect of this disclosure provides a computer-readable storage medium storing at least one instruction that, when executed by a processor, implements the method for determining object information of a medical image as described in the first aspect of this disclosure.

[0020] According to this disclosure, a method, apparatus, device, and medium for determining object information in medical images are provided, which can improve the accuracy of the determined object information. Attached Figure Description

[0021] This disclosure will now be explained in further detail by way of example only with reference to the accompanying drawings.

[0022] Figure 1 This is a schematic diagram illustrating an application scenario of the determination method involved in the examples of this disclosure.

[0023] Figure 2 This is a flowchart illustrating the determination method involved in the example of this disclosure.

[0024] Figure 3 This is a structural block diagram illustrating the object information determination model involved in the example of this disclosure.

[0025] Figure 4 This is a schematic diagram illustrating how a mask image corresponding to the image to be processed is obtained by using an object information determination model in the determination method involved in this disclosure example.

[0026] Figure 5 This is a block diagram illustrating the structure of the encoder involved in the example of this disclosure.

[0027] Figure 6 This is a schematic diagram illustrating how an encoder obtains a second feature vector based on a first feature vector in a determination method involved in this disclosure.

[0028] Figure 7 This is a schematic diagram illustrating the determination of object information and the measurement parameters of the object information in the determination method involved in this disclosure example.

[0029] Figure 8 This is a structural block diagram illustrating the determining device involved in the example of this disclosure. Detailed Implementation

[0030] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same reference numerals are used for the same components, and repeated descriptions are omitted. Furthermore, the drawings are merely schematic diagrams, and the proportions of the components or the shapes of the components may differ from actual figures.

[0031] It should be noted that the terms "comprising" and "having" and any variations thereof in this disclosure, such as a process, method, system, product, or device that includes or has a series of steps or units, are not necessarily limited to those steps or units that are explicitly listed, but may include or have other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0032] This disclosure provides a method for determining object information in medical images (hereinafter referred to as the determination method or approach), which employs machine learning methods to process medical images to determine their object information. The method for determining object information in medical images provided by this disclosure can improve the accuracy of the determined object information.

[0033] The method for determining object information in medical images disclosed herein can also be referred to as a machine learning-based method for determining object information in medical images, a method for extracting object information from medical images, a medical image processing method, or an image segmentation method, etc. The method for determining object information in medical images disclosed herein is applicable to any application scenario requiring the determination of object information in medical images. The apparatus for determining object information in medical images disclosed herein can be simply referred to as a determining apparatus.

[0034] The medical images (hereinafter referred to as images) disclosed herein are images obtained by imaging the internal tissues of the human body or a specific region of the human body. Medical images may include X-ray images, magnetic resonance images, ultrasound images, angiography images, and intracavitary images, etc.

[0035] The term "intraluminal" as used in this disclosure can refer to the cavity between internal organs or tissues of the human body. In some examples, intraluminal images can include intravascular images. Intravascular images can be images of the interior of blood vessels within the human body obtained using medical imaging techniques. In some examples, "intraluminal" may not be limited to the lumen of blood vessels, and may include, for example, the lumen of the esophagus, intestine, or ureter.

[0036] In some examples, categorized by imaging technique, intravascular images can include intravascular ultrasound (IVUS) images and intravascular optical coherence tomography (IVOCT) images. Intravascular ultrasound imaging uses ultrasound waves to image inside blood vessels. Intravascular optical coherence tomography uses optical techniques to achieve high-precision tomographic imaging.

[0037] In some examples, there may be no particular limitation on the method of acquiring medical images. For example, ultrasound signals can be acquired in real time using an intravascular ultrasound system to generate intravascular ultrasound images, or intravascular ultrasound images can be directly read from a storage medium (such as a hard drive) by a computer.

[0038] The object information involved in this disclosure can be any information derived from analyzing image content. Additionally, the object can be any object whose information needs to be identified, such as internal human tissue. In some examples, the object information can be information about internal human tissue. In some examples, the object information can be contour information.

[0039] For ease of description, some examples below use intravascular images as examples of images with similar content, and intravascular ultrasound images are further described as examples of intravascular images. It should be noted that this does not imply limitation of this disclosure, and unless there is a contradiction, the relevant descriptions also apply to other images with similar characteristics.

[0040] The method for determining object information in a medical image, as disclosed herein, will now be described in detail with reference to the accompanying drawings.

[0041] Figure 1 This is a schematic diagram illustrating an application scenario of the determination method involved in the examples of this disclosure.

[0042] In some examples, the determination method disclosed herein can be applied to, for example... Figure 1 In the scenario shown. See also Figure 1 In some examples, the determining device 10 may be connected to or integrated with the acquisition device 20 via a communication link.

[0043] In some examples, where the medical image is an intravascular image, the acquisition device 20 can acquire intravascular image data 30. The intravascular image data 30 can be at least one intravascular image, such as 1, 2, 10, 100, or 500 images. In some examples, the intravascular image data 30 can be a video clip. In some examples, the intravascular image data 30 can be received by the determining device 10.

[0044] In some examples, the determining device 10 can process the image data 30 within the blood vessel lumen to determine object information in the image of the blood vessel lumen to be processed. In this case, after determining the object information, detailed information about the internal tissues of the human body in the image of the blood vessel lumen can be obtained.

[0045] Figure 2 This is a flowchart illustrating the determination method involved in the example of this disclosure.

[0046] See Figure 2In some examples, the determination method includes acquiring the image to be processed and an auxiliary image associated with the image to be processed (step S100), obtaining a multi-channel image 11 based on the image to be processed and the auxiliary image (step S200), inputting the multi-channel image 11 into a trained object information determination model 200 to obtain a mask image 16 corresponding to the image to be processed (step S300), and determining the object information of the image to be processed based on the mask image 16 (step S400). In this case, processing using the multi-channel image 11 increases the amount of information provided to the object information determination model 200, which helps improve the object information determination model 200's ability to distinguish objects in medical images, thereby improving the accuracy of the determined object information.

[0047] See Figure 2 In some examples, in step S100, the image to be processed and an auxiliary image related to the image to be processed can be acquired. The auxiliary image can be used to provide more contextual information related to the image to be processed. In some examples, the auxiliary image can be temporally and spatially related to the image to be processed. For example, the auxiliary image can be an image of an adjacent frame of the image to be processed in a video. The video can be a series of consecutive intracavitary images acquired and recorded using medical imaging technology. Thus, the auxiliary image and the image to be processed can reflect the condition of the same tissue over a period of time.

[0048] In some examples, the image to be processed and the auxiliary image can be medical images. In some examples, the medical image can be an intravascular image. Thus, the object information of the intravascular image can be determined using the method of this disclosure.

[0049] In some examples, the medical image can be an image from a video. In some examples, the video can be a series of consecutive intraluminal images acquired and recorded using medical imaging technology. In some examples, the video can be time-based. Each frame in the video can represent a momentary state within the lumen. Taking intraluminal images of blood vessels as an example, the video can contain information about the temporal and spatial changes of the vessel wall, blood flow, and other possible pathological structures (such as plaques and stenosis). In some examples, where the medical image is an intraluminal image of a blood vessel, the vessel has a first end and a second end, and the video can be acquired from the first end to the second end of the vessel. In some examples, the video can be acquired from the second end to the first end of the vessel. In this case, when integrating information from multiple medical images in the video for processing, the amount of information provided to the object information determination model 200 can be increased, which helps improve the object information determination model 200's ability to distinguish objects in the medical images, thereby improving the accuracy of the determined object information.

[0050] In some examples, in step S200, a multi-channel image 11 can be obtained based on the image to be processed and the auxiliary image. The multi-channel image 11 may contain information about the auxiliary image related to the image to be processed. In some examples, multiple channels in the multi-channel image 11 may correspond to the image to be processed and the auxiliary image respectively, that is, the image to be processed and the auxiliary image are superimposed on the channels. In this case, processing using the multi-channel image 11 increases the amount of information provided to the object information determination model 200, which helps improve the object information determination model 200's ability to distinguish objects in medical images, thereby improving the accuracy of the determined object information. In other words, the multi-channel image 11 can provide more information for the object information determination model 200 to learn.

[0051] In some examples, the auxiliary images may include the first A-frame images of the image to be processed and the last A-frame images of the image to be processed. Here, A represents the frame number of the image. In some examples, A can be 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10, etc. In some examples, the image to be processed, the first A-frame images of the image to be processed, and the last A-frame images of the image to be processed can be combined into a multi-channel image 11. In this case, processing using the multi-channel image 11 enables the object information determination model 200 to learn the contextual information of the image to be processed, which can help improve the object information determination model 200's ability to distinguish objects in medical images.

[0052] In some examples, the frame number of at least one image preceding the image to be processed can be 1, and the frame number of at least one image following the image to be processed can be 1. That is, the auxiliary image can include the frame image preceding the image to be processed and the frame image following the image to be processed. In some examples, the image to be processed, the frame image preceding the image to be processed, and the frame image following the image to be processed can be combined into a three-channel image and used as a multi-channel image 11. In this case, processing using a three-channel image enables the object information determination model 200 to learn the contextual information of the image to be processed, which can help improve the object information determination model 200's ability to distinguish objects in medical images.

[0053] In some examples, the number of frames in at least one image preceding the image to be processed can be two, and the number of frames in at least one image following the image to be processed can also be two. That is, the auxiliary images may include the two images preceding the image to be processed and the two images following the image to be processed. In some examples, the image to be processed, the two images preceding the image to be processed, and the two images following the image to be processed can be combined into a five-channel image and used as a multi-channel image 11. In this case, the five-channel image can increase the amount of contextual information of the image to be processed contained compared to the three-channel image, which can help to further improve the object information determination model 200's ability to distinguish objects in medical images.

[0054] In some examples, the object information determination model 200 can be used to process the multi-channel image 11. In some examples, the object information determination model 200 can be a model for identifying and locating objects contained in medical images.

[0055] In some examples, the object information determination model 200 can be a deep learning model. Deep learning models can include deep neural network models, convolutional neural network models, recurrent neural network models, and self-attention neural network models, etc. In some examples, the object information determination model 200 can include a MedViT (Medical Vision Transformer) model. In this case, compared to a convolutional neural network model that requires processing segmented images separately, the self-attention neural network model can take a vector corresponding to the entire image to be processed as input, and can obtain long-range dependencies that are beneficial for determining objects in medical images, thereby improving the accuracy of the determined object information.

[0056] In some examples, the object information determination model 200 can be a parallel combination of multiple models from the aforementioned deep learning models. In other words, the object information determination model 200 can include multiple different models. The input to each model can be the same. The outputs of multiple models can be fused to obtain the final prediction result.

[0057] In some examples, the object information determination model 200 can be a concatenation of multiple models from the aforementioned deep learning models. In other words, the object information determination model 200 can include multiple different models, where the input of each model except the first can be the output of the previous model. In this case, the flexibility in constructing the object information determination model 200 can be improved, thereby enhancing its applicability to different scenarios.

[0058] In some examples, the object information determination model 200 may include an encoder 210 and a decoder 230. The encoder 210 may be configured to extract features from the image. The decoder 230 may be configured to perform feature analysis on the output of the encoder 210. In this case, feature information of the image can be obtained during the encoding and decoding process, ultimately enabling the determination of object information in the image. Furthermore, the encoder 210 can reduce redundant information in the image, improving image processing efficiency. Additionally, the decoder 230 can recover a portion of the original image information from the encoded image data, which is beneficial for determining object information in the image.

[0059] In some examples, the object information determination model 200 may include a self-attention neural network model. The self-attention neural network model may include an encoder 210 having at least one self-attention layer 213 and a decoder 230 having at least one mutual attention layer.

[0060] In some examples, the object information determination model 200 may be pre-trained.

[0061] In some examples, data augmentation can be performed on the training samples when training the object information determination model 200. In some examples, data augmentation may include at least one of spatial transformation, grayscale transformation, adding noise, and image swapping. In this case, by performing data augmentation on the training samples, the richness of the training samples can be improved, thereby enhancing the generalization ability of the object information determination model 200.

[0062] In some examples, spatial transformations may include rotation, translation, horizontal flipping, and vertical flipping. In this case, the object information determination model 200 can learn images at different spatial locations, thereby improving the robustness of the object information determination model 200 to changes in spatial location.

[0063] In some examples, grayscale transformation may include processes such as grayscale inversion, logarithmic transformation, kinetic transformation, and piecewise linear transformation. In this case, performing grayscale transformation on the training samples can enhance the robustness of the object information determination model 200 to changes in brightness and contrast.

[0064] In some examples, adding noise may include processes such as adding salt-and-pepper noise and adding Gaussian noise. In this case, the complexity of the training samples can be increased, thereby improving the robustness of the object information determination model 200.

[0065] In some examples, image swapping can involve exchanging the order of previous and subsequent frames. In some examples, for medical images depicting the interior of a blood vessel (with a first and second end), video can be captured from the first end to the second end. In other examples, video can be captured from the second end to the first end. The positional order of the images within the blood vessel may differ across multiple video segments. In this context, performing image swapping on the training samples can simulate image order changes in real-world scenarios, improving the object information determination model 200's understanding of the contextual information of the images being processed, thereby enhancing its adaptability to different application scenarios.

[0066] In some examples, when training the object information determination model 200, a loss function can be used to optimize the model. In some examples, the loss function can be used to characterize the difference between the predicted result and the gold standard result. In some examples, the loss function can be used to adjust the network parameters in the object information determination model 200. In some examples, methods for adjusting the network parameters in the object information determination model 200 using a loss function may include the backpropagation algorithm.

[0067] Furthermore, the gold standard result can be a result considered correct. In some examples, the gold standard result can be pre-set. In some examples, the gold standard result can be a medical image 17 with manually processed object information. In some examples, at least one piece of information, including the actual class probability and the actual mask vector, can be obtained from the gold standard result. That is, when training the object information determination model 200, the corresponding true values ​​needed can be obtained directly or indirectly from the gold standard result.

[0068] In some examples, the backpropagation algorithm can be used to adjust network parameters in model 200 based on the results of the loss function and the object information. In some examples, the backpropagation algorithm can first compute the loss function, then compute the gradient of each network parameter with respect to the loss function, and update each parameter using gradient descent to minimize the loss function. In some examples, different optimizers can be used when adjusting network parameters using the backpropagation algorithm. Optimizers can include stochastic gradient descent optimizers, adaptive moment estimation optimizers, or other types of optimizers.

[0069] In some examples, network parameters may include scaling factors, weight matrices, weight parameters, the number of self-attention layers 213, and the number of mutual attention layers, etc.

[0070] In some examples, the loss function may include cross-entropy loss, focusing loss, and dice loss. In this case, using the loss function to optimize object information to determine model 200 can simultaneously take into account multiple losses and optimize object information to determine model 200 from different perspectives, thereby improving the accuracy of the determined object information.

[0071] In some examples, the cross-entropy loss function can be determined based on the predicted class probabilities corresponding to the predicted mask vector (described later) output by at least one decoder 230 and the actual class probabilities determined by the gold standard result. In some examples, the cross-entropy loss function can include the cross-entropy between the predicted class probabilities and the actual class probabilities. In this case, the cross-entropy loss function can take into account the class prediction loss of decoder 230, and decoder 230 can be optimized based on the cross-entropy loss function.

[0072] In some examples, the focus loss function can be determined based on the predicted class probability and the actual class probability corresponding to the predicted fusion mask vector (described later) obtained by the mask determination module 240. The focus loss function may include an adjustable parameter. In this case, optimizing the object information determination model 200 based on the focus loss function can increase the attention of the object information determination model 200 to objects with fewer pixels in the training samples, thereby taking into account the problem of pixel imbalance between positive and negative samples. In addition, for medical images that are intravascular images, when the intravascular image contains small lumens, optimizing the object information determination model 200 based on the focus loss function can increase the attention of the object information determination model 200 to small lumens in the training samples, thereby improving the accuracy of the determined object information.

[0073] In some examples, the dice loss function can be determined based on the predicted fused mask vector obtained by the mask determination module 240 and the actual mask vector determined by the gold standard result. The dice loss function can include the intersection and union of the predicted fused mask vector and the actual mask vector. In some examples, the dice loss function can be used to maximize the intersection of the predicted fused mask vector and the actual mask vector and minimize the union of the predicted fused mask vector and the actual mask vector. In this case, the dice loss function can measure the similarity between the predicted fused mask vector and the actual mask vector, and optimizing the object information determination model 200 based on the dice loss function can make the predicted fused mask vector closer to the actual mask vector.

[0074] See also Figure 2 In some examples, in step S300, the multi-channel image 11 can be input to the trained object information determination model 200 to obtain a mask image 16 corresponding to the image to be processed.

[0075] Figure 3 This is a structural block diagram illustrating the object information determination model 200 involved in the example of this disclosure.

[0076] See Figure 3 In some examples, the object information determination model 200 may include at least one encoder 210, a mapping layer 220, at least one decoder 230, and a mask determination module 240.

[0077] In some examples, encoder 210 can be configured to extract features from an image. In some examples, decoder 230 can be configured to perform feature analysis on an image. In some examples, mapping layer 220 can be configured to convert the output of encoder 210 into the input of decoder 230. In some examples, mask determination module 240 can determine a mask image 16 corresponding to the image to be processed based on the output of decoder 230, and determine object information of the image to be processed based on mask image 16.

[0078] In some examples, there can be multiple encoders 210. For example, there can be 2, 3, 4, 5, or 6 encoders 210. In this case, multi-level feature extraction of the first feature vector 13 (described later) can be performed, which can improve the encoding capability of the object information determination model 200.

[0079] In some examples, the number of decoders 230 can be multiple. For example, the number of decoders 230 can be 2, 3, 4, 5, or 6, etc. In this case, multi-level feature analysis of the third feature vector (described later) can be performed, which can improve the decoding capability of the object information determination model 200.

[0080] In some examples, a single encoder 210 may correspond to multiple decoders 230. The output of a single encoder 210 can be transformed into the input of multiple decoders 230 through a mapping layer 220.

[0081] Figure 4 This is a schematic diagram illustrating how the object information determination model 200 obtains a mask image 16 corresponding to the image to be processed in the determination method involved in this disclosure example.

[0082] See Figure 4 In some examples, the object information determination model 200 may also include an image serialization module. The image serialization module can be configured to preprocess the multi-channel image 11. In some examples, the image serialization module can perform image segmentation and serialization processing on the multi-channel image 11 to obtain a first feature vector 13 corresponding to the multi-channel image 11. In this case, the serialized input method facilitates the object information determination model 200 in processing the multi-channel image 11 using the first feature vector 13 corresponding to the multi-channel image 11, which helps the object information determination model 200 obtain local information in the image to be processed.

[0083] In some examples, the size of the input multi-channel image 11 can be H×W×B, where H represents the number of columns of pixels in the image, W represents the number of rows of pixels in the image, and B represents the number of channels in the image. In some examples, the image can be divided into L image blocks 12, where L = (H / P)×(W / P), where L represents the number of blocks 12 and P represents the size of each image block 12. In some examples, each image block 12 can be converted into a one-dimensional vector of size C, where C = P×P×B, where C represents the size of each one-dimensional vector. In some examples, the L one-dimensional vectors can be arranged and combined sequentially to obtain a first feature vector 13. The size of the first feature vector 13 can be L×C. The first feature vector 13 can correspond to the multi-channel image 11.

[0084] In some examples, encoder 210 may be configured to take a first feature vector 13 corresponding to the multi-channel image 11 as input to obtain a second feature vector 132.

[0085] Figure 5 This is a block diagram illustrating the structure of the encoder 210 as described in this disclosure example.

[0086] See Figure 5 In some examples, encoder 210 may include a downsampling module 211, a feature extraction module 212, at least one self-attention layer 213, and an upsampling module 214. In some examples, downsampling module 211 may downsample the first feature vector 13. In some examples, feature extraction module 212 may preserve the low-level features of the first feature vector 13. In other words, the first feature vector 13 may represent a medical image, and feature extraction module 212 may preserve the high-resolution image information of the unsampled medical image.

[0087] In some examples, the self-attention layer 213 can encode the downsampled first feature vector 131. In some examples, the upsampling module 214 can be configured to upsample the second intermediate vector.

[0088] In some examples, each self-attention layer 213 may include a multi-head self-attention module and a multi-layer perceptron layer module. In some examples, the multi-head self-attention module may obtain a first key vector, a first value vector, and a first query vector based on the downsampled first feature vector 131.

[0089] In some examples, the calculation formula for the multi-head self-attention module can be:

[0090]

[0091] Where Attention(Q,K,V) represents the output of the multi-head self-attention module, softmax represents the softmax function, Q represents the first query vector, and K represents the first key vector. T It can represent the transpose of the first key vector, d k V can represent a scaling factor, and V can represent the first value vector.

[0092] Figure 6 This is a schematic diagram illustrating how encoder 210 obtains a second feature vector 132 based on a first feature vector 13 in the determination method involved in this disclosure.

[0093] See Figure 6 In some examples, the downsampling module 211 can be configured to downsample the first feature vector 13. In this case, downsampling the first feature vector 13 can reduce the amount of data in the first feature vector 13. When the downsampled first feature vector 131 is used as input by the self-attention layer 213 to obtain the second intermediate vector, the computational load of the self-attention layer 213 can be reduced, thereby improving the computational speed.

[0094] In some examples, the downsampling module 211 may use methods such as nearest neighbor downsampling, linear downsampling, bilinear downsampling, mean downsampling, or median downsampling to downsample the first feature vector 13.

[0095] In some examples, the size of the first feature vector 13 can be larger than the size of the downsampled first feature vector 131. For example, the size of the first feature vector 13 can be 32×4096, and the size of the downsampled first feature vector 131 can be 16×2048.

[0096] In some examples, the first feature vector 13 can be input into the downsampling module 211. In some examples, the first feature vector 13 can be directly input into the self-attention layer 213.

[0097] In some examples, the feature extraction module 212 can be configured to extract features from the first feature vector 13 to obtain a first intermediate vector. In this case, the low-level features of the first feature vector 13 can be extracted, and when the second intermediate vector is upsampled using the first intermediate vector to obtain the second feature vector 132, it is beneficial to restore the detailed information of the first feature vector 13, thereby improving the accuracy of feature representation by the object information determination model 200.

[0098] In some examples, the first intermediate vector can be used to record the underlying features of the first feature vector 13. In some examples, a convolutional network algorithm can be used to extract features from the first feature vector 13 to obtain the first intermediate vector.

[0099] In some examples, the feature extraction module 212 can be constructed based on a multi-head self-attention module. In some examples, the formula for calculating the first intermediate vector can be:

[0100]

[0101] Where Q' can represent the first intermediate vector, Q1 can represent the initial query vector, and K1 can represent the key vector of the first feature vector 13. T V1 is the transpose of the key vector of the first feature vector 13, and can represent the value vector of the first feature vector 13. Q1 in the feature extraction module 212 can be randomly initialized.

[0102] In some examples, the first feature vector 13 can be used as both the key vector and the value vector, and the first intermediate vector can be obtained through the calculation formula of the first intermediate vector. In this case, the underlying features of the first feature vector 13 can be extracted through the first intermediate vector.

[0103] In some examples, the size of the first intermediate vector can be the same as the size of the first feature vector 13. For example, the size of the first feature vector 13 can be 32×4096, and the size of the first intermediate vector can also be 32×4096.

[0104] In some examples, the self-attention layer 213 can be configured to take the vector corresponding to the multi-channel image 11 as input and perform feature extraction. That is, the self-attention layer 213 can encode the multi-channel image 11. In this case, the encoder 210, by using at least one self-attention layer 213 for encoding, can obtain long-range dependencies in the first feature vector 13 corresponding to the multi-channel image 11 that are beneficial for determining objects in the medical image, thereby improving the accuracy of the determined object information.

[0105] In some examples, at least one self-attention layer 213 can be configured to take the downsampled first feature vector 131 as input to obtain a second intermediate vector. In this case, the amount of data in the downsampled first feature vector 131 is reduced compared to the original first feature vector 13. When using the self-attention layer 213 to take the downsampled first feature vector 131 as input to obtain the second intermediate vector, the computational cost of the self-attention layer 213 can be reduced, thereby improving the computational speed.

[0106] In some examples, the size of the second intermediate vector can be the same as the size of the downsampled first feature vector 131. For example, the size of the downsampled first feature vector 131 can be 16×2048, and the size of the second intermediate vector can also be 16×2048.

[0107] In some examples, each self-attention layer 213 may include one multi-head self-attention module and one multilayer perceptron layer module. The multi-head self-attention module can be configured to extract features from the input. The multi-head self-attention module may include a first key vector, a first value vector, and a first query vector. The first key vector can be obtained by multiplying the downsampled first feature vector 131 with the weight matrix corresponding to the first key vector. The first value vector can be obtained by multiplying the downsampled first feature vector 131 with the weight matrix corresponding to the first value vector. The first query vector can be obtained by multiplying the downsampled first feature vector 131 with the weight matrix corresponding to the first query vector. The weight matrix in the encoder 210 can be pre-trained.

[0108] In some examples, a multilayer perceptron (MLP) layer module can be configured to extract features from the input. A MLP layer module may include an input layer, hidden layers, and an output layer. The MLP layer module may include weight parameters. These weight parameters may include weights between hidden layers and weights for the output layer. The weight parameters may be pre-trained.

[0109] In some examples, the number of self-attention layers 213 can be 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, or 13, etc. Multiple self-attention layers 213 can be cascaded.

[0110] In some examples, the number of multi-head self-attention modules in each self-attention layer 213 can be multiple, such as 1, 2, 3, 4, or 5. In some examples, the number of multilayer perceptron layer modules in each self-attention layer 213 can be multiple, such as 1, 2, 3, 4, or 5. In some examples, the multi-head self-attention modules can be connected to the multilayer perceptron layer modules. The output of the multi-head self-attention modules can be used as the input of the multilayer perceptron layer modules. In some examples, the output of the multilayer perceptron layer modules can be used as the input of the multi-head self-attention modules.

[0111] Because the second intermediate vector is obtained by the self-attention layer 213 based on the downsampled first feature vector 131, the size of the second intermediate vector can be the same as the size of the downsampled first feature vector 131. Therefore, the resolution of the image corresponding to the first feature vector 13 is greater than the resolution of the image corresponding to the second intermediate vector.

[0112] In some examples, the upsampling module 214 can upsample the second intermediate vector to restore the detailed information of the first feature vector 13. In some examples, the upsampling module 214 can be configured to upsample the second intermediate vector using the first intermediate vector to obtain the second feature vector 132.

[0113] In some examples, the second intermediate vector can be upsampled using the first intermediate vector, which has already recorded the low-level features of the first feature vector 13, to restore the detailed information of the first feature vector 13. In some examples, a convolutional network algorithm can be used to upsample the second intermediate vector using the first intermediate vector to obtain the second feature vector 132.

[0114] In some examples, the upsampling module 214 can be constructed based on a multi-head self-attention module. In some examples, the formula for calculating the second feature vector 132 can be:

[0115]

[0116] Where V” can represent the second feature vector 132, K T ' can represent the transpose of the key vector of the second intermediate vector, and V' can represent the value vector of the second intermediate vector. K T 'and V' can be output from the self-attention layer 213 to the upsampling module 214.

[0117] In some examples, the size of the second feature vector 132 can be the same as the size of the first feature vector 13. For example, if the size of the first feature vector 13 is 32×4096, then the size of the first intermediate vector can be 32×4096, the size of the second intermediate vector can be 16×2048, the size of the key vector of the second intermediate vector can be 16×2048, and the size of the value vector of the second intermediate vector can be 16×2048. In some examples, the size of the second feature vector 132 can be 32×4096. Specifically, the size of the second feature vector 132 can be obtained through vector multiplication using the following calculation process: (32×4096)×(2048×16)×(16×2048)=(32×2×2048)×(2048×16)×(16×2048)=(32×2×16)×(16×2048)=(64×16)×(16×2048)=32×4096.

[0118] In some examples, at least one self-attention layer 213 may be configured after the upsampling module 214. The upsampled second intermediate vector output by the upsampling module 214 can be input to the self-attention layer 213 after the upsampling module 214 to obtain the final second feature vector 132.

[0119] See back Figure 4 In some examples, the mapping layer 220 can convert the output of the encoder 210 into the input of the decoder 230.

[0120] In some examples, the mapping layer 220 can be configured to map the second feature vector 132 into at least one set of third feature vectors. In some examples, each of the at least one set of third feature vectors can correspond to one of the at least one decoder 230. In this case, mapping the second feature vector 132 into at least one set of third feature vectors through the mapping layer 220 can transform an output of the encoder 210 into an input of at least one decoder 230, thereby improving the independence of the input of each decoder 230 and increasing the flexibility of the structural design of the object information determination model 200. In addition, for different scenarios, the number of encoders 210 and decoders 230 can be flexibly selected to meet different requirements for the accuracy and computation time of determining object information, thereby improving the applicability of the object information determination model 200 to different scenarios.

[0121] In some examples, mapping layer 220 may be located in encoder 210. In some examples, mapping layer 220 may be located in decoder 230.

[0122] In some examples, mapping layer 220 may include a linear mapping layer. In some examples, the linear mapping layer may be configured to multiply the second feature vector 132 with the mapping matrix to obtain at least one set of third feature vectors. In this case, the linear mapping preserves the linearity of the second feature vector 132, which is beneficial for subsequently using the obtained third feature vectors to obtain the mask image 16, and also improves the computation speed.

[0123] In some examples, each set of third feature vectors may include a key vector and a value vector. The key vector in each set of third feature vectors can serve as the second key vector in decoder 230 (described later). The value vector in each set of third feature vectors can serve as the second value vector in decoder 230 (described later). The key vector in each set of third feature vectors can be obtained by multiplying the second feature vector 132 with the mapping matrix corresponding to the second key vector. The second value vector can be obtained by multiplying the downsampled second feature vector 132 with the mapping matrix corresponding to the second value vector. The mapping matrix can be pre-trained. The mapping matrix corresponding to the second key vector and the mapping matrix corresponding to the second value vector can be different.

[0124] In some examples, the third eigenvectors of each group can be different. The key vector and value vector in each group of third eigenvectors can also be different.

[0125] In some examples, random noise can be added to the third feature vector. In this case, the independence of the input to each decoder 230 can be improved, and the robustness of the object information determination model 200 can be enhanced.

[0126] In some examples, the number of sets of third feature vectors can be the same as the number of decoders 230. Each set of third feature vectors can correspond to one decoder 230.

[0127] See also Figure 4 In some examples, each of the at least one decoder 230 can be configured to take the third feature vector of the corresponding group and the first category vector 14 as input to obtain the mask vector 15, the second category vector and the fourth feature vector of the respective decoder 230.

[0128] In some examples, the first category vector 14 of the first decoder 230 in at least one decoder 230 can be a preset category vector.

[0129] In some examples, the size of the first category vector 14 can be N×C, where N can represent the number of object categories. For example, for an intravascular image, the object categories could include lumen and external elastic membrane, in which case the number of object categories would be 2. In some examples, N can be a preset value, and the first category vector 14 can be a preset category vector. Thus, the number of object categories can be preset.

[0130] In some examples, the mask vector 15 can be N-valued. In the mask vector 15, each set of identical data values ​​can represent the region to which the object belongs in the medical image.

[0131] In some examples, decoder 230 may include at least one mutual attention layer. In some examples, at least one mutual attention layer may receive a third feature vector of the corresponding group and a first category vector 14 and output a mask vector 15, a second category vector, and a fourth feature vector of decoder 230. In this case, decoder 230 utilizes at least one mutual attention layer for decoding, enabling it to obtain long-distance dependencies in the third feature vector that are beneficial for determining objects in medical images, thereby improving the accuracy of the determined object information.

[0132] In some examples, the mutual attention layer can be configured to take the vector corresponding to the multi-channel image 11 as input and perform feature analysis. That is, the mutual attention layer can decode the multi-channel image 11.

[0133] In some examples, the input to the mutual attention layer may include a second key vector, a second value vector, and a second query vector. The second key vector may be the key vector in the third feature vector of the corresponding group of decoder 230. The second value vector may be the value vector in the third feature vector of the corresponding group of decoder 230. The second query vector may be the first class vector 14.

[0134] In some examples, the output of the mutual attention layer may include a third query vector, a mask vector 15, and a fourth feature vector. In some examples, the second category vector output by the decoder 230 may be the third query vector output by the mutual attention layer.

[0135] In some examples, there can be multiple mutual attention layers. For example, there can be 2, 3, 4, 5, or 6 mutual attention layers.

[0136] In some examples, there can be multiple decoders 230. For example, there can be two, three, four, five, or six decoders 230. In this case, multiple decoders 230 can process the output of the same encoder 210 from different perspectives, enabling feature analysis from different angles, thereby improving the decoding capability of the object information determination model 200.

[0137] In some examples, multiple decoders 230 can be cascaded. In some examples, the first category vector 14 corresponding to the first decoder 230 among the multiple decoders 230 can be a preset category vector. In some examples, the first category vector 14 corresponding to the decoders 230 other than the first one among the multiple decoders 230 can be the second category vector output by the previous decoder 230. In this case, by cascading the decoders 230, each decoder 230 other than the first decoder 230 can receive the output of the previous decoder 230 as input, so that each decoder 230 can fuse information from different levels of previous decoders 230, thereby improving the decoding capability of the object information determination model 200 and improving the accuracy of the determined object information.

[0138] In some examples, decoder 230 can determine mask vector 15 based on the first predicted class probability of the data point. In some examples, decoder 230 can select the object class corresponding to the highest first predicted class probability of the first data point as the object class of the first data point. For example, the first predicted class probability of the first data point can be: the probability that the first data point belongs to the first object class is 80%, and the probability that it belongs to the second object class is 20%. The first object class can be selected as the object class of the first data point in mask vector 15.

[0139] In some examples, decoder 230 can output the predicted class probabilities corresponding to the data points in mask vector 15. In some examples, the mask vector 15 output by decoder 230 is also the prediction mask vector, and the first predicted class probability of the data points in the prediction mask vector is also the predicted class probability of the prediction mask vector.

[0140] See also Figure 4 In some examples, the mask determination module 240 may be configured to obtain a mask image 16 based on a fourth feature vector of at least one decoder 230 and a mask vector 15.

[0141] In some examples, the mask determination module 240 can fuse all mask vectors 15 corresponding to at least one decoder 230 to obtain a fused mask vector. In this case, fusing all mask vectors 15 corresponding to at least one decoder 230 to obtain a fused mask vector can make comprehensive use of the output of the decoder 230, reduce the influence of a single decoder 230 on the fused mask vector, and improve the robustness of the object information determination model 200.

[0142] In some examples, the mask determination module 240 can multiply the fourth feature vectors of each decoder 230 with the weight matrices corresponding to each fourth feature vector to obtain a fused fourth feature vector. The mask determination module 240 can then convert the fused fourth feature vector into a multi-channel prediction image and select the image of the channel corresponding to the image to be processed as the category prediction image. The category prediction image can be represented as a category prediction vector.

[0143] In some examples, the fourth feature vector output by the last decoder 230 among multiple decoders 230 and all mask vectors 15 of the multiple decoders 230 can be used as input to the mask determination module 240. In this case, using all mask vectors 15 of the multiple decoders 230 as input to the mask determination module 240 can comprehensively utilize the outputs of the decoders 230, reduce the influence of a single decoder 230 on the mask image 16, and improve the robustness of the object information determination model 200.

[0144] In some examples, the mask determination module 240 can be configured to obtain a category prediction vector corresponding to the image to be processed based on the fourth feature vector output by the last decoder 230 of at least one decoder 230. For example, if the size of the fourth feature vector is L×C, it can first be restored to L image blocks 12, and then the L image blocks 12 can be sequentially concatenated to form a multi-channel prediction image. In the multi-channel prediction image, the image of the channel corresponding to the image to be processed can be a category prediction image. The category prediction image can be represented by a category prediction vector. That is, the category prediction vector can represent the image to be processed. In this case, obtaining the category prediction vector corresponding to the image to be processed using the fourth feature vector output by the last decoder 230 reduces the computational cost compared to using the fourth feature vectors output by multiple decoders 230.

[0145] In some examples, the mask determination module 240 can obtain the predicted class probability of the fused mask vector by statistically analyzing the second predicted class probability of each data point at the same position in all mask vectors 15 and based on the second predicted class probability.

[0146] In some examples, the mask determination module 240 can select the object category corresponding to the highest second predicted category probability of the data point as the object category of the second data point. For example, in all three mask vectors 15, if the second data point belongs to the first object category in the first mask vector 15 and to the second object category in the second and third mask vectors 15, then the probability that the second data point belongs to the first object category is 33.3%, and the probability that it belongs to the second object category is 66.7%. The second predicted category probability of the second data point can be: the probability that the second data point belongs to the first object category is 33.3%, and the probability that it belongs to the second object category is 66.7%. The second object category can be selected as the object category of the second data point in the fused mask vector, and the value corresponding to the second object category can be used as the value of the second data point in the fused mask vector.

[0147] In some examples, the mask determination module 240 may select the object category corresponding to the highest first predicted category probability above a threshold for the second data point as the object category of the second data point. The threshold may be 60%, 70%, 80%, or 90%, etc. In some examples, the threshold may be preset.

[0148] In some examples, the fused mask vector output by the mask determination module 240 is also the predicted fused mask vector, and the second predicted class probability of the data points in the predicted fused mask vector is also the predicted class probability of the predicted fused mask vector.

[0149] In some examples, the mask determination module 240 can perform a dot product between the fused mask vector and the class prediction vector to obtain the mask image 16. In some examples, the resulting vector obtained by performing a dot product between the fused mask vector and the class prediction vector can represent the mask image 16. That is, the mask image 16 can be represented as a result vector. In other words, performing a dot product between the fused mask vector and the class prediction vector can obtain the mask image 16.

[0150] See back Figure 2 In some examples, in step S400, object information of the image to be processed can be determined based on the mask image 16.

[0151] Figure 7 This is a schematic diagram illustrating the determination of object information and the measurement parameters of the object information in the determination method involved in this disclosure example.

[0152] See Figure 7 In some examples, after the multi-channel image 11 is input into the object information determination model 200 for processing, a medical image 17 with determined object information can be obtained.

[0153] In some examples, object information may include outline information. In some examples, outline information may include outline information of organizational structure and outline information of non-organizational structure.

[0154] In some examples, for medical images that are intravascular images, object information may include the contour information of plaque lesions, such as calcified lesions. In some examples, the contour information of tissue structures may include the boundaries of the lumen and the external elastic membrane. In some examples, the contour information of non-tissue structures may include the boundaries of guidewire artifacts and stent struts.

[0155] See also Figure 7 In some examples, object information from the image to be processed can be input into the parametric measurement model 300 to obtain measurement parameters for the object information. In this case, the parametric measurement model 300 can determine the measurement parameters of the object information, and can provide results for further analysis of the object information (such as the area and diameter of the object).

[0156] In some examples, the parameter measurement model 300 can be configured to determine a preset calculation method for measurement parameters of object information in the image to be processed, and to determine the measurement parameters of the object information based on the preset calculation method. In some examples, measurement parameters may include diameter and area, etc.

[0157] In some examples, for medical images that are intravascular images, measurement parameters may include lumen area, external elastic membrane area, lumen diameter, external elastic membrane diameter, plaque load, and stenosis rate. In some examples, diameter may include minimum diameter, maximum diameter, average diameter, area-equivalent diameter, and circumference-equivalent diameter.

[0158] In some examples, the area measurement parameter can be calculated by using Green's formula to calculate the area enclosed by the closed curve, and then multiplying it by the pixel interval to obtain the physical area. In other examples, the area measurement parameter can be calculated by directly counting the number of pixels, and then multiplying it by the pixel interval to obtain the physical area.

[0159] In some examples, the diameter measurement parameter can be calculated by finding any number of diameters at the two intersections of a straight line through the center of the lumen and the boundary of the lumen, with the minimum value being the minimum lumen diameter and the maximum value being the maximum lumen diameter.

[0160] In some examples, the plaque load measurement parameter can be calculated as follows: Plaque load = (External elastic membrane area - Lumen area) / External elastic membrane area.

[0161] In some examples, the stenosis rate can be calculated as follows: Stenosis rate (%) = (Normal lumen diameter - Current lumen diameter) / Current lumen diameter × 100%. Here, the normal lumen diameter can refer to the diameter of the vessel without stenosis, and the current lumen diameter can refer to the diameter of the vessel under stenotic conditions.

[0162] In some examples, an ideal model can be built for calculating the normal lumen diameter. In other examples, images from a video can be selected and an ideal model built. For instance, the first and last frames of the video can be selected, and the lumen diameter can be linearly predicted based on the lumen diameters in the two images. In still other examples, the average of the lumen diameters from the first and last frames can be taken as the normal lumen diameter.

[0163] In some examples, the image can be selected manually. In some examples, the normal lumen diameter can be determined manually.

[0164] In some examples, the number of images selected can be multiple, such as 2, 3, 4, or 5.

[0165] Figure 8 This is a structural block diagram illustrating the determining device 10 involved in the example of this disclosure.

[0166] The determining device 10 can be configured to implement the method for determining object information of a medical image as described in the examples of this disclosure. See also Figure 8In some examples, the determining device 10 may include an image acquisition module 110, an image processing module 120, and a result determining module 130. It should be noted that, unless contradictory, the above description of the method for determining object information in a medical image also applies to the determining device 10.

[0167] In some examples, the image acquisition module 110 can be configured to acquire an image to be processed and an auxiliary image associated with the image to be processed, and obtain a multi-channel image 11 based on the image to be processed and the auxiliary image. In some examples, the image to be processed and the auxiliary image can be a medical image. In this case, the multi-channel image 11 contains information about the auxiliary image associated with the image to be processed. Processing with the multi-channel image 11 increases the amount of information provided to the object information determination model 200, which helps improve the object information determination model 200's ability to distinguish objects in medical images, thereby improving the accuracy of the determined object information.

[0168] In some examples, the image processing module 120 may be configured to input the multi-channel image 11 into a trained object information determination model 200 to obtain a mask image 16 corresponding to the image to be processed. In some examples, the object information determination model 200 may include at least one encoder 210, a mapping layer 220, at least one decoder 230, and a mask determination module 240. In some examples, the encoder 210 may be configured to take a first feature vector 13 corresponding to the multi-channel image 11 as input to obtain a second feature vector 132.

[0169] In some examples, mapping layer 220 can be configured to map the second feature vector 132 into at least one set of third feature vectors. In some examples, each of the at least one set of third feature vectors can correspond to one of the at least one decoder 230. In this case, mapping the second feature vector 132 into at least one set of third feature vectors via mapping layer 220 can transform an output of encoder 210 into an input of at least one decoder 230, thereby improving the independence of the inputs of each decoder 230 and increasing the flexibility of the structural design of the object information determination model 200.

[0170] In some examples, each of the at least one decoder 230 may be configured to take the third feature vector of the corresponding group and the first category vector 14 as input to obtain the mask vector 15, the second category vector, and the fourth feature vector of each decoder 230. In some examples, the first category vector 14 of the first decoder 230 in the at least one decoder 230 is a preset category vector.

[0171] In some examples, the mask determination module 240 may be configured to obtain a mask image 16 based on a fourth feature vector of at least one decoder 230 and a mask vector 15.

[0172] In some examples, the result determination module 130 can be configured to determine object information of the image to be processed based on the mask image 16.

[0173] Examples of this disclosure also disclose an electronic device that may include a processor and a memory, the processor being able to execute a program stored in the memory to implement one or more steps of the method described above for determining object information of a medical image.

[0174] Examples of this disclosure also disclose a computer-readable storage medium that can store at least one instruction, which, when executed by a processor, can implement one or more steps of the method described above for determining object information of a medical image.

[0175] This disclosure relates to a method for determining object information of a medical image, comprising obtaining a multi-channel image 11 based on an image to be processed and an auxiliary image, inputting the multi-channel image 11 into a trained object information determination model 200 to obtain a mask image 16 corresponding to the image to be processed, and determining the object information of the image to be processed based on the mask image 16.

[0176] In this context, the multi-channel image 11 contains information from auxiliary images related to the image to be processed. Processing with the multi-channel image 11 increases the amount of information provided to the object information determination model 200, which helps improve the object information determination model 200's ability to distinguish objects in medical images, thereby improving the accuracy of the determined object information. Furthermore, by mapping the second feature vector 132 to at least one set of third feature vectors through the mapping layer 220, one output of the encoder 210 can be transformed into the input of at least one decoder 230, thereby improving the independence of the inputs to each decoder 230 and increasing the flexibility of the structural design of the object information determination model 200.

[0177] While the present disclosure has been specifically described above in conjunction with the accompanying drawings and examples, it is to be understood that the foregoing description does not limit the present disclosure in any way. Those skilled in the art can make modifications and variations to the present disclosure as needed without departing from its essential spirit and scope, and all such modifications and variations shall fall within the scope of the present disclosure.

Claims

1. A method for determining object information of a medical image, characterized in that, include: Acquire an image to be processed and an auxiliary image related to the image to be processed, wherein the image to be processed and the auxiliary image are medical images; A multi-channel image is obtained based on the image to be processed and the auxiliary image, wherein the auxiliary image includes at least one frame before the image to be processed and at least one frame after the image to be processed; The multi-channel image is input into a trained object information determination model to obtain a mask image corresponding to the image to be processed. The object information determination model includes at least one encoder, a mapping layer, at least one decoder, and a mask determination module. The encoder is configured to take a first feature vector corresponding to the multi-channel image as input to obtain a second feature vector. The mapping layer is configured to map the second feature vector into at least one set of third feature vectors, each set of third feature vectors corresponding to one of the at least one decoders. Each of the at least one decoder is configured to take the third feature vector and the first category vector of the corresponding group as input to obtain a mask vector, a second category vector, and a fourth feature vector for each decoder. The first category vector of the first decoder in the at least one decoder is a preset category vector. There are multiple decoders, and these decoders are cascaded. The first category vector corresponding to each decoder other than the first one is the second category vector output by the previous decoder. The mask determination module is configured to obtain the mask image based on the fourth feature vector of the at least one decoder and the mask vector. Specifically, the fourth feature vector output by the last decoder among the plurality of decoders and all the mask vectors of the plurality of decoders are used as input to the mask determination module. A category prediction vector corresponding to the image to be processed is obtained based on the fourth feature vector output by the last decoder among the at least one decoder. All the mask vectors corresponding to the at least one decoder are fused to obtain a fused mask vector. The fused mask vector is then multiplied by the category prediction vector to obtain the mask image. The object information of the image to be processed is determined based on the mask image.

2. The method for determining object information in a medical image according to claim 1, characterized in that, The medical images are intravascular images.

3. The method for determining object information in a medical image according to claim 1, characterized in that, The image to be processed, at least one frame before the image to be processed, and at least one frame after the image to be processed are combined to form the multi-channel image.

4. The method for determining object information in a medical image according to claim 3, characterized in that, The frame number of the at least one frame preceding the image to be processed is 1, the frame number of the at least one frame following the image to be processed is 1, and the multi-channel image is a three-channel image; or The number of frames in the at least one frame before the image to be processed is 2, the number of frames in the at least one frame after the image to be processed is 2, and the multi-channel image is a five-channel image.

5. The method for determining object information in a medical image according to claim 1, characterized in that, The encoder includes a downsampling module, a feature extraction module, at least one self-attention layer, and an upsampling module. The downsampling module is configured to downsample the first feature vector. The feature extraction module is configured to extract features from the first feature vector to obtain a first intermediate vector. The magnitude of the first intermediate vector is the same as the magnitude of the first feature vector. The at least one self-attention layer is configured to use the downsampled first feature vector as input to obtain a second intermediate vector. The upsampling module is configured to use the first intermediate vector to upsample the second intermediate vector to obtain the second feature vector.

6. The method for determining object information in a medical image according to claim 1, characterized in that, The mapping layer includes a linear mapping layer configured to multiply the second feature vector by the mapping matrix to obtain the at least one set of third feature vectors, the number of sets of third feature vectors being the same as the number of decoders.

7. The method for determining object information in a medical image according to claim 1, characterized in that, The decoder includes at least one mutual attention layer. Each group of the third feature vectors includes a key vector and a value vector. The at least one mutual attention layer receives the third feature vector of the corresponding group and the first category vector, and outputs the mask vector, the second category vector and the fourth feature vector of the decoder.

8. The method for determining object information in a medical image according to claim 1, characterized in that, Also includes: The object information of the image to be processed is input into the parameter measurement model to obtain the measurement parameters of the object information. The parameter measurement model is configured to determine a preset calculation method for the measurement parameters of the object information of the image to be processed, and to determine the measurement parameters of the object information based on the preset calculation method.

9. An apparatus for determining object information in a medical image, characterized in that, include: Image acquisition module, image processing module, and result determination module. The image acquisition module is configured to acquire an image to be processed and an auxiliary image related to the image to be processed, and to obtain a multi-channel image based on the image to be processed and the auxiliary image. The image to be processed and the auxiliary image are medical images, and the auxiliary image includes at least one frame before the image to be processed and at least one frame after the image to be processed. The image processing module is configured to input the multi-channel image into a trained object information determination model to obtain a mask image corresponding to the image to be processed. The object information determination model includes at least one encoder, a mapping layer, at least one decoder, and a mask determination module. The encoder is configured to take a first feature vector corresponding to the multi-channel image as input to obtain a second feature vector. The mapping layer is configured to map the second feature vector into at least one set of third feature vectors, each of the at least one set of third feature vectors corresponding to one of the at least one decoders. Each of the at least one decoder is configured to take the third feature vector and the first category vector of the corresponding group as input to obtain a mask vector, a second category vector, and a fourth feature vector for each decoder. The first category vector of the first decoder in the at least one decoder is a preset category vector. There are multiple decoders, and these decoders are cascaded. The first category vector corresponding to each decoder other than the first one is the second category vector output by the previous decoder. The mask determination module is configured to obtain the mask image based on the fourth feature vector of the at least one decoder and the mask vector, wherein the fourth feature vector output by the last decoder among the plurality of decoders and all the mask vectors of the plurality of decoders are used as the input of the mask determination module; a category prediction vector corresponding to the image to be processed is obtained based on the fourth feature vector output by the last decoder among the at least one decoder; all the mask vectors corresponding to the at least one decoder are fused to obtain a fused mask vector; and the fused mask vector is multiplied by the category prediction vector to obtain the mask image. The result determination module is configured to determine the object information of the image to be processed based on the mask image.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory, the processor executing a program stored in the memory to implement the method for determining object information of a medical image as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the method for determining object information of a medical image as described in any one of claims 1 to 8.

Images