Medical image marking device and medical image marking method

Abstract

To provide a medical image marking device and a medical image marking method.SOLUTION: A medical image marking device comprises an interface, a memory for storing a plurality of serialized medical images, and a processor coupled to the interface and the memory. The processor is used to receive, via the interface, a first manual mark on a first slice image in the plurality of serialized medical images and a second manual mark on a second slice image in the plurality of serialized medical images, and execute a bidirectional inference mechanism to generate a plurality of result mark labels on a plurality of intermediate slice images in the plurality of serialized medical images on the basis of the first manual mark and the second manual mark. The medical image marking device is used to efficiently and accurately generate the marks on the serialized medical images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This disclosure relates to medical imaging technology. More specifically, this disclosure relates to a method and apparatus for marking serialized medical images. [Background technology]

[0002] Medical imaging technologies, including X-rays, magnetic resonance imaging, and computed tomography scans, are crucial tools in modern medical diagnosis, providing detailed images of the body's internal structures necessary for accurate diagnosis and treatment. However, the increasing volume and complexity of image data pose significant challenges for medical professionals, particularly in the accurate recognition and marking of areas of interest. [Overview of the Initiative] [Problems that the invention aims to solve]

[0003] Marking areas of interest in large volumes of medical images is extremely time-consuming. In practical applications, accurately marking areas of interest often relies on manual operation by radiologists or radiologic technologists. The burden of manually processing serialized images, such as computed tomography scans and magnetic resonance imaging scans, is particularly heavy because specialists must mark each slice obtained from the scan individually. This process is not only extremely time-consuming but can also lead to inconsistencies between markings due to differences in marking techniques, potentially impacting the efficiency of medical image analysis and clinical diagnosis. [Means for solving the problem]

[0004] According to one aspect of the present disclosure, a medical image marking apparatus is disclosed, comprising an interface, a memory for storing a plurality of serialized medical images, and a processor coupled to the interface and the memory, wherein the processor is used to receive a first manual mark on a first slice image of the plurality of serialized medical images and a second manual mark on a second slice image of the plurality of serialized medical images via the interface, and to perform a bidirectional inference mechanism to generate a plurality of result mark labels on a plurality of intermediate slice images of the plurality of serialized medical images, respectively, based on the first manual mark and the second manual mark.

[0005] In some embodiments, the bidirectional inference mechanism performed by the processor includes: running a tracking model based on the first manual mark on the first slice image to generate a plurality of forward predictive marks in forward order on the plurality of intermediate slice images of the plurality of serialized medical images; running the tracking model based on the second manual mark on the second slice image to generate a plurality of backward predictive marks in backward order on the plurality of intermediate slice images of the plurality of serialized medical images; and integrating the plurality of forward predictive marks and the plurality of backward predictive marks to generate the plurality of result mark labels on the plurality of intermediate slice images.

[0006] In some embodiments, the tracking model includes a tracking model for detecting, associating, and tracking the first manual mark on the first slice image between each of the plurality of intermediate slice images in the forward order, and for detecting, associating, and tracking the second manual mark on the second slice image between each of the plurality of intermediate slice images in the reverse order.

[0007] In some embodiments, depending on the K-th slice image in the plurality of serialized medical images, the result mark label on the K-th slice image is generated by calculating a first distance between the first slice image and the K-th slice image and a second distance between the K-th slice image and the second slice image; if the first distance is shorter than the second distance, selecting a forward prediction mark on the K-th slice image as the result mark label on the K-th slice image; and if the second distance is shorter than the first distance, selecting a backward prediction mark on the K-th slice image as the result mark label on the K-th slice image.

[0008] In some embodiments, depending on the K-th slice image in the plurality of serialized medical images, the result mark label on the K-th slice image is generated by calculating a first distance between the first slice image and the K-th slice image and a second distance between the K-th slice image and the second slice image, and generating the result mark label on the K-th slice image according to a weighted sum of forward and backward prediction marks on the K-th slice image based on the first and second distances.

[0009] In some embodiments, the bidirectional inference mechanism performed by the processor includes running a bidirectional XMem model to generate the plurality of result mark labels on the plurality of intermediate slice images based on the first manual mark and the second manual mark.

[0010] In some embodiments, the bidirectional XMem model includes a sensory memory for storing short-term forward hidden representations and short-term backward hidden representations; a working memory for storing forward memory key features, backward memory key features, forward memory value features and backward memory value features; and a long-term memory for storing forward long-term memory key features, backward long-term memory key features, forward long-term memory value features and backward long-term memory value features.

[0011] In some embodiments, a bidirectional XMem model further includes a query encoder for generating forward input queries for forward input medical images and reverse input queries for reverse input medical images; a decoder for generating forward mark shields for forward input medical images and reverse mark shields for reverse input medical images based on the forward input queries, the reverse input queries, read features, the short-term forward hide expressions from the sensory memory, and the short-term reverse hide expressions; and a shield encoder for generating short-term forward hide expressions and short-term reverse hide expressions based on the forward mark shields and reverse mark shields.

[0012] In some embodiments, read features are generated based on an association matrix and read values, the association matrix is ​​calculated by the processor based on the forward input query, the reverse input query, the forward memory key features, the reverse memory key features, the forward long-term memory key features and the reverse long-term memory key features, and the read values ​​are calculated by the processor based on the forward memory value features, the reverse memory value features, the forward long-term memory value features and the reverse long-term memory value features.

[0013] In some embodiments, the multiple serialized medical images include multiple magnetic resonance imaging scan images or multiple computed tomography scan images.

[0014] Another aspect of the present disclosure discloses a medical image marking method comprising: acquiring a plurality of serialized medical images; receiving a first manual mark on a first slice image and a second manual mark on a second slice image of the plurality of serialized medical images; and performing a bidirectional inference mechanism to generate a plurality of result mark labels on a plurality of intermediate slice images of the plurality of serialized medical images, respectively, based on the first manual mark and the second manual mark. [Effects of the Invention]

[0015] The medical image marking apparatus and method can effectively utilize a limited number of user marking data to automatically generate result mark labels on intermediate slice images based on bidirectional tracking. The medical image marking apparatus and method can improve semi-automatic marking efficiency. The bidirectional inference mechanism can provide more reference information through object tracking, thereby improving the accuracy of automatic marking when recognizing unmarked image data.

[0016] It is necessary to explain that the above description and subsequent detailed descriptions are used to illustrate the present invention with embodiments and to support the interpretation and understanding of the content of the invention claimed by the present invention. [Brief explanation of the drawing]

[0017] To make the above and other purposes, features, and embodiments of this disclosure clearer and easier to understand, the attached drawings are described below. [Figure 1] This is a schematic diagram of a medical image marking device according to some embodiments of the present disclosure. [Figure 2] This is a schematic diagram of a first embodiment of the bidirectional inference mechanism executed by the processor according to the present disclosure. [Figure 3] This is a schematic diagram illustrating how the tracking model generates forward prediction marks. [Figure 4] This is a schematic diagram illustrating how the tracking model generates reverse prediction marks. [Figure 5] This is a schematic diagram of a bidirectional XMem model that the processor executes in some embodiments of this disclosure. [Figure 6] This is a schematic diagram of a bidirectional XMem model that the processor executes based on another pair of slice images. [Modes for carrying out the invention]

[0018] The following disclosure provides many different embodiments or examples for carrying out different features of this disclosure. Components and configurations in specific examples are used in the following discussion to simplify this disclosure. Any examples discussed are for illustrative purposes only and do not limit the scope or meaning of this disclosure or its examples. Where appropriate, the same reference numerals are used between drawings and in corresponding text descriptions to represent the same or similar components.

[0019] Please refer to Figure 1, which is a schematic diagram of a medical image marking apparatus 100 according to some embodiments of the present disclosure. In some embodiments, the medical image marking apparatus 100 is used to efficiently generate marks on multiple serialized medical image SIMGs.

[0020] In some embodiments, the serialized medical image SIMG includes multiple consecutive magnetic resonance imaging (MMRI) scan images or multiple consecutive computed tomography (CT) scan images. In the embodiment shown in Figure 1, the serialized medical image SIMG may be a series of abdominal MMRI / CT scan images. The serialized medical image SIMG may also include multiple slice images extracted in chronological order. As shown in Figure 1, the serialized medical image SIMG includes multiple slice images SL1, SL2, SL3, SL4, SL5, SL6, SL7, and SL8. In some embodiments, these slice images SL1-SL8 may be taken from the patient's body at chronological points in time by a MMRI / CT scanner. For example, slice image SL1 is taken first, slice image SL2 is taken after slice image SL1, slice image SL3 is taken after slice image SL2, and so on, until finally slice image SL8 is taken. These slice images SL1-SL8 may each represent consecutive abdominal images located at different heights in the body.

[0021] The serialized medical image SIMG is not limited to the abdominal magnetic resonance imaging / computed tomography (MRI) scan images shown in Figure 1. In other embodiments, the serialized medical image SIMG may be a series of brain MRI scans, a series of chest MRI scans, or other similar medical images. The medical image marking device 100 can be used to process various types of serialized medical image SIMGs and can also be extended to applications in medical films.

[0022] For the sake of brevity, the serialized medical image SIMG shown in Figure 1 includes eight slice images for illustrative purposes. However, the number of slice images in a serialized medical image SIMG is not limited to this in this disclosure. A serialized medical image SIMG may include N slice images, where N is a positive integer greater than 3.

[0023] Each serialized medical image SIMG is extracted sequentially, so adjacent slice images in a serialized medical image SIMG are related to each other. If an object (e.g., an organ or tissue of the body) appears with a specific contour in slice image SL1, the same object may appear with a similar contour in slice image SL2. Therefore, manual markings P for this object can be made on slice image SL1. M1 If provided, manual marking P on slice image SL1. M1 By using this as a hint, similar objects in the subsequent slice images SL2 and SL3 can be separated according to this hint. Therefore, another manual mark P for this object can be placed on slice image SL8. M8 If provided, manual markings P on slice image SL8 are used to separate similar objects in adjacent slice images SL7, SL6, etc. M8 This can be used as a hint.

[0024] In the present disclosure, the medical image marking device 100 aims to improve the accuracy during automatic marking of medical images not marked by a recognition algorithm by effectively utilizing a limited number of manual mark data. As shown in FIG. 1, the medical image marking device 100 includes an interface 120, a processor 140, and a memory 160. As shown in FIG. 1, the memory 160 is used to store serialized medical images SIMG (i.e., slice images SL1 to slice image SL8).

[0025] The interface 120 may include input-output elements (e.g., a touch panel, a keyboard, a mouse, a microphone, a display). As shown in FIG. 1, in some embodiments, the user can operate the interface 120 to input a manual mark P M1 on the first slice image (i.e., slice image SL1) of the serialized medical image SIMG, and operate the interface 120 to input another manual mark P M8 on the last slice image (i.e., slice image SL8) of the serialized medical image SIMG.

[0026] As shown in FIG. 1, the processor 140 is coupled to the interface 120 and the memory 160. The processor 140 is used to receive the manual mark P M1 on the slice image SL1 in the serialized medical image SIMG via the interface 120, and receive the manual mark P M8 on the slice image SL8 in the serialized medical image SIMG. The processor 140 executes a bidirectional inference mechanism to generate result mark labels P2, P3, P4, P5, P6, and P7 on intermediate slice images (e.g., slice images SL2, SL3, SL4, SL5, SL6, and SL7) of the serialized medical image SIMG based on the two manual marks P M1 and P M8 respectively. The detailed method of the bidirectional inference mechanism will be further described in the following embodiments.

[0027] In some embodiments, the two manual marks are not limited to being marked on the first and last slice images SL1 and SL8. In other embodiments, the two manual marks may be marked on other slice image combinations, for example, on slice images SL2 and SL7, or on slice images SL3 and SL6.

[0028] Please also refer to Figure 2. Figure 2 is a schematic diagram of a first embodiment of the bidirectional inference mechanism performed by the processor 140 according to this disclosure. In the embodiment shown in Figure 2, the bidirectional inference mechanism performed by the processor 140 comprises a tracking model 142 and an integration device 144. The tracking model 142 is a manual mark P on the slice image SL1. M1 Based on this, forward prediction marks P are placed on the intermediate slice images (e.g., slice images SL2, SL3…SL7) of the serialized medical image SIMG. F2 , P F3 , P F4 , P F5 , P F6 and P F7 It is used to generate the following. Furthermore, tracking model 142 uses manual marking P on slice image SL8. M8 Based on this, reverse prediction marks P are placed on the intermediate slice images (e.g., slice images SL7, SL6…SL2) of the serialized medical image SIMG. B7 , P B6 , P B5 , P B4 , P B3 and P B2 It is used to generate [something].

[0029] How tracking model 142 predicts the forward mark P F2 , P F3 , P F4 , P F5 , P F6 and P F7See also Figure 3, a schematic diagram illustrating how this occurs. In some embodiments, the tracking model 142 may be implemented by a Track-Anything-Model (TAM). The Track-Anything-Model (TAM) involves manually marking P on slice image SL1 between each of the intermediate slice images (e.g., slice images SL2, SL3…SL7) in a forward order. M1 It is used to detect, associate, and track them.

[0030] In some embodiments, a Track Anising Model (TAM) is a computer vision algorithm for sequential tracking or following of a single object or label in a serialized medical image SIMG. A Track Anising Model (TAM) may be implemented by software program code and may employ software architectures such as convolutional neural networks, recurrent neural networks, or converters. A Track Anising Model (TAM) is pre-trained using a large dataset to understand and predict the movement trajectories and appearance contours of various objects.

[0031] As shown in Figure 3, serialized medical image SIMG and manual mark P M1 This is input into tracking model 142 (e.g., track-anishing model). Tracking model 142 (e.g., track-anishing model) uses manual mark P on slice image SL1. M1 By using this as a hint, forward prediction marks P for objects similar to the connected slice image SL2 can be made. F2 This will cause the event to occur. In this example, manual mark P M1 The liver region appearing in slice image SL1 is marked, and thereby the tracking model 142 predicts the location of the potential liver region in slice image SL2 using forward prediction marks P. F2 This can generate a forward prediction mark P for an object similar to the connected slice image SL3. Similarly, a forward prediction mark P for an object similar to the connected slice image SL3 can be generated. F3 To generate, manually mark P M1and forward prediction mark P F2 This can be input as a hint for the tracking model 142. Similarly, the tracking model 142 places forward prediction marks P on the intermediate slice images (e.g., slice images SL2, SL3, SL4, SL5, SL6, and SL7) of the serialized medical image SIMG. F2 , P F3 , P F4 , P F5 , P F6 and P F7 These can be generated in sequence.

[0032] How tracking model 142 predicts the reverse direction mark P B7 , P B6 , P B5 , P B4 , P B3 and P B2 See also Figure 4, a schematic diagram illustrating how this occurs. In some embodiments, the tracking model 142 may be implemented by a track-anishing model (TAM). The track-anishing model (TAM) involves manually marking P on slice image SL1 between each of the intermediate slice images (e.g., slice images SL7, SL6…SL2) in reverse order. M1 It is used to detect, associate, and track them.

[0033] As shown in Figure 4, the serialized medical image SIMG and manual mark PM8 are input to the tracking model 142 (e.g., the track-anissing model). The tracking model 142 (e.g., the track-anissing model) inputs the manual mark P on the slice image SL8. M8 By using this as a hint, we can predict the inverse mark P for objects similar to the adjacent slice image SL7. B7 This will cause the event to occur. In this example, manual mark P M8 The liver region appearing in slice image SL8 is marked, and thereby the tracking model 142 predicts the reverse direction of the liver region P in slice image SL7. B7This can generate a reverse prediction mark P for an object similar to the adjacent slice image SL6. B6 To generate, manually mark P M8 And the reverse direction prediction mark P B7 This can be input as a hint for the tracking model 142. Similarly, the tracking model 142 can input reverse prediction marks P on the intermediate slice images (e.g., slice images SL7, SL6, SL5, SL4, SL3 and SL2) of the serialized medical image SIMG. B7 , P B6 , P B5 , P B4 , P B3 and P B2 These can be generated in sequence.

[0034] As shown in Figure 2, based on the above bidirectional tracking, the tracking model 142 generates two prediction marks for each intermediate slice image (e.g., slice images SL2 to SL7). For slice image SL2, manual mark P M1 The forward prediction mark P generated based on F2 and manual mark P M8 The reverse prediction mark P is generated based on this. B2 There is a forward prediction mark P for slice image SL3. F3 And the reverse direction prediction mark P B3 There is a forward prediction mark P for slice image SL7. F7 And the reverse direction prediction mark P B7 There is.

[0035] The integration device 144 displays forward prediction marks P for the slice image SL2. F2 And the reverse direction prediction mark P B2 It is used to receive and integrate and generate result mark labels P2 on the slice image SL2. Similarly, the integrating device 144 generates forward predicted marks P on the slice image SL3. F3 And the reverse direction prediction mark P B3It is used to receive and integrate and generate result mark labels P3 on slice image SL3. Similarly, the integrating device 144 generates forward prediction marks P on slice image SL7. F7 And the reverse direction prediction mark P B7 It is used to receive and integrate the data and generate the resulting mark label P7 on the slice image SL7.

[0036] In some embodiments, the result mark labels P2 to P7 may be determined by the integration device 144 based on the distance between the target slice image and the manually marked image.

[0037] For example, with respect to the K-th slice image of a serialized medical image SIMG, the result mark labels on the K slice images are generated by calculating a first distance between slice image SL1 and the K-th slice image, and a second distance between the K-th slice image and slice image SL8. If the first distance is shorter than the second distance, the integrator 144 selects the forward prediction mark on the K-th slice image as the result mark label on the K-th slice image. If the second distance is shorter than the first distance, the integrator 144 selects the backward prediction mark on the K-th slice image as the result mark label on the K-th slice image. Here, K is a positive integer.

[0038] For example, with respect to slice image SL2, the first distance between slice image SL1 and slice image SL2 is "1", and the second distance between slice image SL2 and slice image SL8 is "6". In this case, the first distance is short, meaning that slice image SL2 is closer to slice image SL1. In this case, the forward prediction mark P F2 Select this as the result mark label P2 on slice image SL2.

[0039] With respect to slice image SL3, the first distance between slice image SL1 and slice image SL3 is "2", and the second distance between slice image SL3 and slice image SL8 is "5". In this case, the forward prediction mark P F3Select it as the result mark label P3 on the slice image SL3.

[0040] Regarding the slice image SL7, the first distance between the slice image SL1 and the slice image SL7 is "6", and the second distance between the slice image SL7 and the slice image SL8 is "1". In this case, the second distance is short, that is, the slice image SL7 is closer to the slice image SL8. In this case, the reverse prediction mark P B7 is selected as the result mark label P7 on the slice image SL7.

[0041] In some embodiments, the result mark labels P2 to P7 are the forward prediction marks P F2 ~P F7 and the reverse prediction marks P B2 ~P B7 It is not limited to being selected from. In other embodiments, the result mark labels P2 to P7 are the forward prediction marks P F2 ~P F7 and the reverse prediction marks P B2 ~P B7 It may be determined based on the weighted sum with.

[0042] For example, regarding the K-th slice image of the serialized medical image SIMG, the result mark labels on the K slice images calculate the first distance between the slice image SL1 and the K-th slice image and the second distance between the K-th slice image and the slice image SL8, and may be generated by the weighted sum of the forward prediction mark and the reverse prediction mark on the K-th slice image based on the first distance and the second distance.

[0043] For example, regarding the slice image SL2, the first distance between the slice image SL1 and the slice image SL2 is "1", and the second distance between the slice image SL2 and the slice image SL8 is "6". The result mark label P2 on the slice image SL2 may be generated by the weighted sum of the forward prediction mark P F2 and the reverse prediction mark P B2 as shown below.

Number

[0044] In some embodiments, the forward prediction mark P F2 includes the probability value (e.g., 0% to 100%) of each corresponding pixel on the slice image SL2. These probability values represent whether each pixel is related to the target object or label. The backward prediction mark P B2 also includes the probability value of each pixel on the slice image SL2. The integration device 144 is used to calculate the weighted sum of both the probability values of the forward prediction mark P F2 and the backward prediction mark P B2 . In some examples, if the weighted sum of one pixel exceeds 50%, this pixel is selected as part of the result mark label P2 on the slice image SL2. In some examples, if the weighted sum of one pixel is lower than 50%, this pixel is excluded from the result mark label P2 on the slice image SL2.

[0045] Regarding the slice image SL3, the first distance between the slice image SL1 and the slice image SL3 is "2", and the second distance between the slice image SL3 and the slice image SL8 is "5". The result mark label P3 on the slice image SL3 may be generated by the weighted sum of the forward prediction mark P F3 and the backward prediction mark P B3 as shown below.

Number

[0046] Regarding the slice image SL7, the first distance between the slice image SL1 and the slice image SL7 is "6", and the second distance between the slice image SL7 and the slice image SL8 is "1". The result mark label P7 on the slice image SL7 may be generated by the weighted sum of the forward prediction mark P F7 and the backward prediction mark P B7 as shown below.

number

[0047] The bidirectional inference mechanism shown in Figure 2 includes a tracking model 142 and an integration device 144, and can be seamlessly integrated into a Track Anising Model (TAM) framework. The medical image marking device 100 has a limited number of user marking data (i.e., manual mark P M1 and P M8 By effectively utilizing this, result mark labels P2~P7 on intermediate slice images can be automatically generated based on bidirectional tracking. The medical image marking device 100 can improve semi-automatic marking efficiency. The bidirectional inference mechanism can provide more reference information for object tracking, thereby improving the accuracy of automatic marking when recognizing unmarked image data.

[0048] Manual marking on serialized medical images SIMG M1 and P M8 The result mark labels P2-P7 may be used as training data for training medical-related models, such as organ segmentation models, medical image classification models, and diagnostic support models.

[0049] The bidirectional inference mechanism shown in Figure 2 is a post-fusion (policy-level fusion) method comprising a tracking model 142 and an integration device 144, and its architecture is simple and easy to implement. In some embodiments, the bidirectional inference mechanism shown in Figure 2 may exhibit discontinuous shapes in the generated marks, and further manual adjustments may be necessary to improve the resulting mark labels P2-P7.

[0050] This disclosure provides an alternative method for implementing a bidirectional inference mechanism to overcome the potential challenges of shape discontinuities that may arise when post-fusion integrates marking information. See also Figure 5, which is a schematic diagram of a bidirectional XMem model 146 executed by a processor 140 according to some embodiments of this disclosure. In some embodiments, the processor 140 is used to execute the bidirectional XMem model 146, which, as one method of implementing the bidirectional inference mechanism, is manual marking P M1 and P M8 Based on this, result mark labels P2, P3, P4, P5, P6, and P7 are generated on the intermediate slice images (e.g., slice images SL2, SL3, SL4, SL5, SL6, and SL7).

[0051] XMem is a model proposed to realize object partitioning of long-term films, consisting of short-term and long-term memory systems, referencing the Atkinson-Shiffrin memory model.

[0052] As shown in Figure 5, the bidirectional XMem model 146 is an improved version of XMem, and the bidirectional XMem model 146, using a bidirectional mechanism, can consider information from preceding and succeeding slices. This bidirectional memory retrieval enhances the contextual understanding of the bidirectional XMem model 146. The bidirectional XMem model 146 employs a feature-level fusion policy (not policy-level fusion). This policy enables integration at the feature level, leveraging the ability to consider the temporal and spatial dimensions of the model in the feature representation. Spatial fusion at the feature level allows for tighter and more effective joining of marking information from different perspectives, achieving the multidimensional analysis capabilities of the bidirectional XMem model 146.

[0053] In some embodiments, the bidirectional XMem model 146 has a first manual mark (i.e., manual mark P M1 ) and the second manual mark (i.e., manual mark P) M8 This is used to generate result mark labels on the intermediate slice image based on ).

[0054] In some embodiments, the bidirectional XMem model 146 can process a pair of slice images in a serialized medical image SIMG each time to generate result mark labels on these two slice images. For example, a serialized medical image SIMG (including slice images SL1-SL8), with a manual mark P on slice image SL1. M1 and manual mark P on slice image SL8 M8 After receiving the signal, the bidirectional XMem model 146 can first generate result mark labels P2 and P7 on this pair of slice images SL2 and SL7.

[0055] As shown in Figure 5, the bidirectional XMem model 146 includes a sensory memory 146a, a working memory 146b, and a long-term memory 146c.

[0056] Sensory memory 146a is a short-term memory element for processing real-time data of recent slices from a serialized medical image SIMG. Sensory memory 146a extracts and processes real-time information that is currently critical to slice splitting. As shown in Figure 5, sensory memory 146a is used to store the short-term forward non-display expression h1 and the short-term reverse non-display expression h8. The short-term forward non-display expression h1 is used for manual marking P M1 This occurs based on the following: Short-term forward-hiding expression h1 is a manual mark P on slice image SL1. M1 The distribution of (shield / label) can be represented. The short-term inverse hiding expression h8 represents the manual mark P on the slice image SL8. M8 This occurs based on (shield / label).

[0057] Working memory 146b and long-term memory 146c are used to hold important information that may no longer be present in short-term memory. Working memory 146b and long-term memory 146c are used to maintain the relationship between preceding and succeeding images on a serialized medical image SIMG. Working memory 146b can be considered a dynamic and flexible memory space for processing information to handle tasks in real time, and is used to process the current and recent frames, thereby allowing the system to quickly adapt to changes in input information. Long-term memory 146c is used to store information that is considered important to maintain the preceding and succeeding consistency and the relationship between preceding and succeeding images of the serialized medical image SIMG, compared to working memory 146b. Long-term memory 146c is used to hold important object features and relationship information between preceding and succeeding images on a long-term scale.

[0058] As shown in Figure 5, the working memory 146b has forward memory key feature K WF Reverse direction memory key feature K WB , forward memory value feature V WF and reverse memory value feature V WB It is used to store. Long-term memory 146c is forward long-term memory key feature K LF Reverse Long-Term Memory Key Feature K LB Forward long-term memory value feature V LF and reverse long-term memory feature V LB It is used for memorization.

[0059] In some embodiments, the forward memory key feature K in the working memory 146b WF and forward memory feature V WF This is updated based on forward information data INF1. Initially, forward information data INF1 is based on slice image SL1 and manual mark P M1 This includes the reverse memory key feature K in the working memory 146b. WB and reverse memory value feature V WB This is updated based on the reverse information data INF2. Initially, the reverse information data INF2 is based on the slice image SL8 and the manually marked PM8 Includes.

[0060] Using a bidirectional mechanism, the data stored in the working memory 146b and long-term memory 146c takes into account information from the front and rear slices (e.g., forward information data INF1 and reverse information data INF2). This bidirectional memory retrieval enhances the understanding of front and rear images in the bidirectional XMem model 146.

[0061] As shown in Figure 5, the bidirectional XMem model 146 further comprises a query encoder 146d, a decoder 146e, and a shield encoder 146f. As shown in Figure 5, the query encoder 146d handles forward input queries Q related to the forward input medical image (i.e., slice image SL2). SL2 And reverse input queries Q regarding reverse input medical images (i.e., slice images SL7) SL7 It is used to generate the query encoder 146d converts the input slice images SL2 and SL7 into a high-dimensional feature representation (i.e., forward input query Q SL2 and reverse input query Q SL7 It is used to convert to a forward input query Q. SL2 and reverse input query Q SL7 This is used to query the working memory 146b and the long-term memory 146c to retrieve relevant historical information.

[0062] As shown in Figure 5, in some embodiments, the forward memory key feature K WF Reverse direction memory key feature K WB Forward long-term memory key feature K LF and reverse long-term memory key feature K LB These are connected in series in common to form a memory key Kc. The bidirectional XMem model 146 executed by processor 140 is a forward input query Q SL2 and reverse input query Q SL7 The association matrix W is based on the similarity between the memory key Kc and the association matrix W. AM It is used to perform further calculations. In other words, processor 140 is used for forward input query Q SL2 , reverse input query QSL7 Forward memory key feature K WF Reverse direction memory key feature K WB Forward long-term memory key feature K LF and reverse long-term memory key feature K LB Based on the related matrix W AM Further calculations are performed.

[0063] The bidirectional XMem model 146 is used to integrate memory data retrieved at different time points (medium-term and long-term) to obtain a consistent and comprehensive memory representation, such as a memory key Kc. Unlike the original XMem, the bidirectional XMem model 146 ensures a more complete representation of shape and time information. When shape information from the early, medium, and late points in a serialized image can be combined with each other, it is extremely useful for continuously tracking objects in medical imaging.

[0064] As shown in Figure 5, the bidirectional XMem model 146 performs a memory read operation on the associated matrix W AM And based on the read value Vc, read feature F RO This generates the forward memory feature V. In some embodiments, the processor 140 generates the forward memory feature V. WF , reverse memory value feature V WB Forward long-term memory value feature V LF and reverse long-term memory feature V LB The read value Vc is further calculated based on this. The above memory read operation determines the relationship (relationship matrix W) between various historical information in memory and the features extracted by the query encoder 146d. AM This is used to calculate () and to recognize which features or regions are most relevant to the target object.

[0065] Decoder 146e receives a forward input query Q SL2 , reverse input query Q SL7 , readout feature F ROBased on short-term forward non-display expression h1 and short-term reverse non-display expression h8 from sensory memory 146a, a forward mark shield (i.e., result mark label P2) for a forward input medical image (i.e., slice image SL2) and a reverse mark shield (i.e., result mark label P7) for a reverse input medical image (i.e., slice image SL7) are generated.

[0066] Decoder 146e is used to generate result segmentation shields from integrated features obtained by the query encoder 146d and the memory modules (i.e., sensory memory 146a, working memory 146b, and long-term memory 146c). The input to decoder 146e is a combination of the features of the current slice from the query encoder 146d and the high-dimensional feature representation of the relevant historical data retrieved from the memory modules. This combination ensures that the current observation and the past context together form the image segmentation results (result mark labels P2 and P7).

[0067] The shield encoder 146f is used to generate a short-term forward invisibility expression h2 based on the forward mark shield (i.e., result mark label P2), and a short-term reverse invisibility expression h7 based on the reverse mark shield (i.e., result mark label P7). The short-term forward invisibility expression h2 and the short-term reverse invisibility expression h7 are updated in the sensory memory 146a for later splitting of another pair of slice images.

[0068] In this case, the result mark label P2 for slice image SL2 and the result mark label P7 for slice image SL7 are manually marked on slice image SL1 by the bidirectional XMem model 146. M1 and manual mark P on slice image SL8 M8 It may be generated by referring to [the specified source]. As shown in Figure 5, for subsequent splitting, the result mark label P2 for slice image SL2 is added to the forward information data INF1, and the result mark label P7 for slice image SL7 is added to the reverse information data INF2.

[0069] See also Figure 6, which is a schematic diagram of the bidirectional XMem model 146 that processor 140 executes based on another pair of slice images SL3 and SL6.

[0070] As shown in Figure 6, the sensory memory 146a is used to store the short-term forward non-display expression h2 and the short-term reverse non-display expression h7. The short-term forward non-display expression h2 can represent the distribution of forward mark shields (i.e., result mark labels P2) on the slice image SL2. The short-term reverse non-display expression h7 can represent the distribution of reverse mark shields (i.e., result mark labels P7) on the slice image SL7.

[0071] In some embodiments, forward storage key feature K in the working memory 146b is based on forward information data INF1. WF and forward memory feature V WF Update. Currently, the forward information data INF1 is slice image SL1, manual mark P M1 This includes slice image SL2 and result mark label P2. On the other hand, reverse memory key feature K in the working memory 146b based on reverse information data INF2. WB and reverse memory value feature V WB Update. Currently, the reverse information data INF2 is slice image SL8, manual mark P M8 This includes slice image SL7 and result mark label P7.

[0072] As shown in Figure 6, the query encoder 146d receives forward input queries Q related to the forward input medical image (i.e., slice image SL3). SL3 And reverse input queries Q regarding reverse input medical images (i.e., slice images SL6) SL6 It is used to generate a high-dimensional feature representation (i.e., forward input query Q) of the incoming slice images SL3 and SL6. SL3 and reverse input query Q SL6 It is used to convert to ).

[0073] As shown in Figure 6, in some embodiments, the forward memory key feature K WF Reverse direction memory key feature K WB Forward long-term memory key feature K LF and reverse long-term memory key feature K LB These are connected in series in common to form a memory key Kc. The bidirectional XMem model 146 executed by processor 140 is a forward input query Q SL3 and reverse input query Q SL6 The association matrix W is based on the similarity between the memory key Kc and the association matrix W. AM It is used to calculate [something].

[0074] As shown in Figure 6, the bidirectional XMem model 146 performs a memory read operation on the associated matrix W AM And based on the read value Vc, read feature F RO This will cause it to happen.

[0075] Decoder 146e receives a forward input query Q SL3 , reverse input query Q SL6 , readout feature F RO Based on the short-term forward non-display expression h2 and short-term reverse non-display expression h7 from sensory memory 146a, a forward mark shield (i.e., result mark label P3) for the forward input medical image (i.e., slice image SL3) and a reverse mark shield (i.e., result mark label P6) for the reverse input medical image (i.e., slice image SL6) are generated.

[0076] Decoder 146e is used to generate result partitioning shields from integrated features obtained by the query encoder 146d and the memory modules (sensory memory 146a, working memory 146b, and long-term memory 146c). The input to decoder 146e is a combination of the features of the current slice from the query encoder 146d and the high-dimensional feature representation of the relevant historical data retrieved from the memory modules. This combination ensures that the current observation and the past context together form the partitioning results (result mark labels P3 and P6).

[0077] The shield encoder 146f is used to generate a short-term forward invisibility expression h3 based on the forward mark shield (i.e., result mark label P3), and a short-term reverse invisibility expression h6 based on the reverse mark shield (i.e., result mark label P6). The short-term forward invisibility expression h3 and the short-term reverse invisibility expression h6 are updated in the sensory memory 146a for later splitting of another pair of slice images.

[0078] In this case, the result mark label P3 for slice image SL3 and the result mark label P6 for slice image SL6 may be generated by the bidirectional XMem model 146 by referencing the current input data and historical data. As shown in Figure 6, for later splitting, the result mark label P3 for slice image SL3 is added to the forward information data INF1, and the result mark label P6 for slice image SL6 is added to the reverse information data INF2.

[0079] Similarly, the resulting mark labels P4 and P5 for the other slice images SL4 and SL5 may be generated by the bidirectional XMem model 146.

[0080] In some embodiments, the query encoder 146d, decoder 146e, and shield encoder 146f shown in Figures 5 and 6 may be implemented by software instructions executed by the processor 140 shown in Figure 1. In some embodiments, the sensory memory 146a, working memory 146b, and long-term memory 146c shown in Figures 5 and 6 may be implemented by memory blocks defined in the memory 160 shown in Figure 1, or by individual memories.

[0081] As shown in the above embodiment, the medical image marking device 100 can process various sequential data, such as serialized images and medical films, simultaneously, and can effectively utilize various different marking data to perform a bidirectional inference mechanism.

[0082] In some embodiments, the medical image marking apparatus 100 may be implemented by a computer, a computing server, or a medical image server. The processor 140 may be implemented by a central processing unit, a graphics processing unit, a tensor processing unit, or a specific application integrated circuit.

[0083] The medical image marking method performed by the medical image marking device 100 may be one embodiment of the present disclosure. The medical image marking method includes the steps of acquiring a serialized medical image SIMG and making a first manual mark (e.g., manual mark P) on a first slice image of the serialized medical image SIMG. M1 ) and a second manual mark on the second slice image of the serialized medical image SIMG (e.g., manual mark P M8 The steps include receiving a ) and executing a bidirectional inference mechanism (e.g., the tracking model 142 and integration device 144 shown in Figure 2, or the bidirectional XMem model 146 shown in Figures 5 and 6) to obtain a first manual mark (e.g., manual mark P M1 ) and a second manual mark (e.g., manual mark P) M8 The process includes the step of generating result mark labels (e.g., result mark labels P2 to P7) on intermediate slice images (e.g., slice images SL2 to SL7) of the medical image SIMG serialized based on ). Details of these steps have been discussed in the above embodiment and will not be repeated.

[0084] While certain embodiments of this disclosure are disclosed in relation to the embodiments described above, these embodiments are not intended to limit this disclosure. Various substitutions and improvements can be made in this disclosure by those skilled in the art without departing from the principles and spirit of this disclosure. Accordingly, the scope of protection of this disclosure is determined by the appended claims. [Explanation of Symbols]

[0085] 100: Medical image marking device 120: Interface 140: Processor 142: Tracking Model 144: Integration device 146: Bidirectional XMem Model 146a: Sensory memory 146b: Working memory 146c: Long-term memory 146d: Query Encoder 146e: Decoder 146f: Shielded encoder 160: Memory F RO : Readout features h1, h2, h3: Short-term forward hidden expression h6, h7, h8: short-term backward hidden expression INF1: Forward information data INF2: Reverse direction information data Kc: Memory Key K LF : Forward Long-Term Memory Key Features K LB : Reverse Long-Term Memory Key Features K WF : Forward memory key features K WB Reverse direction memory key features P2, P3, P4, P5, P6, P7: Result Mark Labels P B2 , P B3 , P B4 , P B5 , P B6 , P B7 : Reverse direction prediction mark P F2 , P F3 , P F4 , P F5 , P F6 , P F7 : Forward prediction mark P M1 , P M8 :Manual marking Q SL2 Q SL3 : Forward input query Q SL6 Q SL7 : Reverse input query SIMG: Serialized Medical Images SL1, SL2, SL3, SL4: Slice images SL5, SL6, SL7, SL8: Slice images Vc: Read value V LB : Reverse Long-Term Memory Value Characteristics V LF : Forward Long-Term Memory Characteristics V WB : Reverse memory value features V WF : Forward memory value features W AM Related Matrix

Claims

1. A medical image marking device, Interface and Memory for storing multiple serialized medical images, A processor coupled to the interface and the memory, Equipped with, The processor receives, via the interface, a first manual mark on a first slice image in the plurality of serialized medical images and a second manual mark on a second slice image in the plurality of serialized medical images, and It is characterized by being used to execute a bidirectional inference mechanism to generate a plurality of result mark labels on a plurality of intermediate slice images of a plurality of serialized medical images, respectively, based on the first manual mark and the second manual mark, The bidirectional inference mechanism executed by the processor is: By executing a tracking model based on the first manual mark on the first slice image, a plurality of forward predictive marks are generated in a forward order on the plurality of intermediate slice images of the plurality of serialized medical images, By executing the tracking model based on the second manual marks on the second slice image, a plurality of reverse prediction marks are generated in reverse order on the plurality of intermediate slice images of the plurality of serialized medical images, The plurality of forward prediction marks and the plurality of backward prediction marks are integrated to generate the plurality of result mark labels on the plurality of intermediate slice images, Medical image marking device, including [specific component].

2. The medical image marking apparatus according to claim 1, further comprising a tracking model for detecting, associating, and tracking the first manual mark on the first slice image between each of the plurality of intermediate slice images in the forward order, and a tracking model for detecting, associating, and tracking the second manual mark on the second slice image between each of the plurality of intermediate slice images in the reverse order.

3. Depending on the K-th slice image in the plurality of serialized medical images, the result mark label on the K-th slice image is: Calculate the first distance between the first slice image and the Kth slice image and the second distance between the Kth slice image and the second slice image. If the first distance is shorter than the second distance, select the forward prediction mark on the K-th slice image as the result mark label on the K-th slice image, and The medical image marking apparatus according to claim 1, wherein if the second distance is shorter than the first distance, the reverse prediction mark on the K-th slice image is selected as the result mark label on the K-th slice image.

4. Depending on the K-th slice image in the plurality of serialized medical images, the result mark label on the K-th slice image is: Calculate the first distance between the first slice image and the Kth slice image and the second distance between the Kth slice image and the second slice image, and A medical image marking apparatus according to claim 1, which generates the result mark label on the K-th slice image according to the weighted sum of the forward prediction mark and the reverse prediction mark on the K-th slice image based on the first distance and the second distance.

5. A medical image marking device, Interface and Memory for storing multiple serialized medical images, A processor coupled to the interface and the memory, Equipped with, The processor receives, via the interface, a first manual mark on a first slice image in the plurality of serialized medical images and a second manual mark on a second slice image in the plurality of serialized medical images, and It is characterized by being used to execute a bidirectional inference mechanism to generate a plurality of result mark labels on a plurality of intermediate slice images of a plurality of serialized medical images, respectively, based on the first manual mark and the second manual mark, The bidirectional inference mechanism executed by the processor is: This includes running a bidirectional XMem model to generate the plurality of result mark labels on the plurality of intermediate slice images based on the first manual mark and the second manual mark, The aforementioned bidirectional XMem model is Sensory memory for storing short-term forward non-display expressions and short-term reverse non-display expressions, A working memory for storing forward memory key features, reverse memory key features, forward memory value features, and reverse memory value features, A long-term memory for storing forward long-term memory key features, reverse long-term memory key features, forward long-term memory value features, and reverse long-term memory value features, A medical image marking device equipped with [a specific feature].

6. The aforementioned bidirectional XMem model is A query encoder for generating forward input queries for forward input medical images and reverse input queries for reverse input medical images, A decoder for generating a forward mark shield for the forward input medical image and a reverse mark shield for the reverse input medical image based on the forward input query, the reverse input query, read features, the short-term forward hiding expression from the sensory memory, and the short-term reverse hiding expression, A shield encoder for generating the short-term forward non-display expression and the short-term reverse non-display expression based on the forward mark shield and the reverse mark shield, The medical image marking device according to claim 5, further comprising the above.

7. The aforementioned readout features are generated based on the associated matrix and the readout values. The association matrix is ​​calculated by the processor based on the forward input query, the reverse input query, the forward memory key feature, the reverse memory key feature, the forward long-term memory key feature, and the reverse long-term memory key feature. The medical image marking apparatus according to claim 6, wherein the read value is calculated by the processor based on the forward storage value feature, the reverse storage value feature, the forward long-term storage value feature, and the reverse long-term storage value feature.

8. A medical image marking method performed by a medical image marking device, The steps include obtaining multiple serialized medical images, The steps include receiving a first manual mark on a first slice image of the plurality of serialized medical images and a second manual mark on a second slice image of the plurality of serialized medical images via the interface of the medical image marking device, The processor of the medical image marking device performs a bidirectional inference mechanism to generate a plurality of result mark labels on a plurality of intermediate slice images of the plurality of serialized medical images based on the first manual mark and the second manual mark, It is characterized by including, The aforementioned bidirectional inference mechanism is By executing a tracking model based on the first manual mark on the first slice image, a plurality of forward predictive marks are generated in a forward order on the plurality of intermediate slice images of the plurality of serialized medical images, By executing the tracking model based on the second manual marks on the second slice image, a plurality of reverse prediction marks are generated in reverse order on the plurality of intermediate slice images of the plurality of serialized medical images, The plurality of forward prediction marks and the plurality of backward prediction marks are integrated to generate the plurality of result mark labels on the plurality of intermediate slice images, A medical image marking method, including [specific details omitted].

Images