Medical image processing device, ultrasound diagnostic device, medical image processing method, and medical image processing program

Abstract

To reflect the spatial characteristics of the three-dimensional structure of the object being detected. [Solution] The medical image processing apparatus according to the embodiment comprises an image acquisition unit, a spatial information acquisition unit, a feature map generation unit, and an execution unit. The image acquisition unit acquires a plurality of medical images. The spatial information acquisition unit acquires three-dimensional spatial information corresponding to the plurality of medical images and representing the features of the three-dimensional space. The feature map generation unit generates a feature map based on the plurality of medical images and the three-dimensional spatial information. The execution unit performs processing on the target medical image based on the feature map.

Description

[Technical Field]

[0001] Embodiments disclosed herein and in the drawings relate to medical image processing apparatus, ultrasound diagnostic apparatus, medical image processing method, and medical image processing program. [Background technology]

[0002] Traditionally, video-based lesion detection has been a localized examination method for the organ being examined, and multiple types of lesions may appear in the video footage. Common medical video examinations include colonoscopy, gastroscopy, internal ultrasound, and external ultrasound. Ultrasound lesion detection technology can help physicians find potentially lesional areas in ultrasound images in a timely manner, thereby increasing the disease detection rate in routine examinations.

[0003] In video detection tasks, it is difficult to accurately determine the type and location of a lesion from a single frame image. In medical video detection, features of each frame image in a time-series image are usually extracted first, and then features of the time-series image are obtained using a feature fusion method to more accurately determine the location and type of the lesion. Feature fusion techniques include, for example, feature aggregation-based methods such as optical flow feature mapping.

[0004] As an example, there is a target detection method in which the current frame and the preceding keyframe corresponding to the current frame are obtained in a video frame sequence, an optical flow feature map and optical flow map are determined between the preceding keyframe and the current frame, and if it is determined that the current frame is not a keyframe based on the optical flow map, the keyframe features corresponding to the preceding keyframe are obtained, an affine transformation is performed on the keyframe features based on the optical flow map to obtain image features corresponding to the current frame, the pixel level of each pixel point in the current frame is detected according to the image features, and a detection result corresponding to the target in the current frame is obtained. However, in this method, keyframe features are superimposed on the current frame using optical flow feature mapping, but the extracted and superimposed features remain 2D features, so the spatial features of the 3D tissue structure are lost.

[0005] Another example involves using panoramic image stitching technology to expand the effective detection range. This method saves images collected at previous time points, uses image stitching technology to obtain a wider area of ​​imagery, and expands the perceptual range of the detection model, thereby improving detection accuracy. However, while this method stitches together 2D images when using previous frames, the extracted image features remain 2D surface features, resulting in the loss of spatial features of 3D tissue structure. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] U.S. Patent No. 11816895 [Patent Document 2] U.S. Patent No. 11423318 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] One of the problems that the embodiments disclosed herein and in the drawings aim to solve is to reflect the spatial characteristics of the three-dimensional structure of the object to be detected. However, the problems that the embodiments disclosed herein and in the drawings aim to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0008] The medical image processing apparatus according to this embodiment comprises an image acquisition unit, a spatial information acquisition unit, a feature map generation unit, and an execution unit. The image acquisition unit acquires a plurality of medical images. The spatial information acquisition unit acquires three-dimensional spatial information corresponding to the plurality of medical images and representing the features of the three-dimensional space. The feature map generation unit generates a feature map based on the plurality of medical images and the three-dimensional spatial information. The execution unit performs processing on the target medical image based on the feature map. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a block diagram showing the functional configuration of an image processing apparatus according to the first embodiment. [Figure 2] Figure 2 is a schematic diagram illustrating the generation of a 3D feature map from a 3D image in the first embodiment. [Figure 3] Figure 3 is a schematic diagram illustrating the generation of a two-dimensional feature map by compressing a three-dimensional feature map in the first embodiment. [Figure 4] Figure 4 is a flowchart showing the image processing method performed by the image processing device according to the first embodiment. [Figure 5] Figure 5 is a block diagram showing the functional configuration of an image processing apparatus according to the second embodiment. [Figure 6] Figure 6 is a block diagram showing the functional configuration of the image sorting unit according to the second embodiment. [Figure 7] Figure 7 shows a flowchart that is executed when sorting images in real time in the second embodiment. [Figure 8] FIG. 8 is a flowchart showing a specific example in step S113 of FIG. 7. [Figure 9] FIG. 9 is a block diagram showing a functional configuration of an image processing apparatus according to a third embodiment. [Figure 10] FIG. 10 is a block diagram showing a functional configuration of a detection unit according to a third embodiment. [Figure 11] FIG. 11 is a block diagram showing a functional configuration of an ultrasonic diagnostic apparatus using an image processing apparatus according to a third embodiment. [Figure 12] FIG. 12 is a timing chart showing operation timings of respective components in an ultrasonic diagnostic apparatus using an image processing apparatus according to a third embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a detection result of lesion detection performed by an image processing apparatus 102 according to a third embodiment. [Figure 14] FIG. 14 is a diagram showing an example of a detection result of lesion detection performed by an image processing apparatus 102 according to a third embodiment. [Figure 15] FIG. 15 is a block diagram showing a functional configuration of an image processing apparatus according to a fourth embodiment.

Embodiments of the Invention

[0010] An image processing apparatus according to an embodiment includes an image acquisition unit that acquires a plurality of images; a 3D feature map generation unit that generates a 3D feature map corresponding to the 3D image by treating the plurality of images as one 3D image; a 2D feature map generation unit that generates a 2D feature map by compressing the 3D feature map; and an output unit that outputs the 2D feature map.

[0011] Thereby, it is possible to output a 2D feature map that reflects the spatial features of the 3D structure of the detection target.

[0012] In addition, in the image processing apparatus, it is preferable that the three-dimensional feature map generation unit generates a three-dimensional feature map corresponding to the three-dimensional image configured in the time series direction, and the two-dimensional feature map generation unit compresses the three-dimensional feature map in the time series direction to generate the two-dimensional feature map.

[0013] Thereby, while ensuring real-time detection, it is possible to output a two-dimensional feature map that reflects the spatial features of the three-dimensional structure of the detection target.

[0014] In addition, in the image processing apparatus, it is preferable that the two-dimensional feature map generation unit assigns a time series weight to the three-dimensional feature map in the time series direction and compresses the three-dimensional feature map with the time series weight assigned thereto into the two-dimensional feature map.

[0015] Thereby, while considering the contribution degrees of frames at a plurality of times, it is possible to output a two-dimensional feature map that reflects the spatial features of the three-dimensional structure of the detection target.

[0016] In addition, in the image processing apparatus, it is preferable to assign a higher time series weight to a region closer to the current frame in the time series direction with respect to the three-dimensional feature map.

[0017] Thereby, while considering that a region closer to the current frame among frames at a plurality of times has a higher contribution degree, it is possible to output a two-dimensional feature map that reflects the spatial features of the three-dimensional structure of the detection target.

[0018] In addition, in the image processing apparatus, it is preferable that the time series weight is a learnable time series weight trained by machine learning.

[0019] Thereby, it is possible to learn a reasonable time series weight in the training process by machine learning.

[0020] Furthermore, in the image processing apparatus, it is preferable that the number of images constituting the three-dimensional image does not exceed a predetermined number.

[0021] This allows for the construction of a 3D image using a finite number of images, thereby reducing computational complexity and improving the real-time performance of the resulting 2D feature map.

[0022] Furthermore, it is preferable that the image processing apparatus further includes an image selection unit that selects an image from a plurality of images in a time series, and that the 3D feature map generation unit generates a 3D feature map corresponding to the plurality of images selected by the image selection unit as a single 3D image.

[0023] This allows for the selection of multiple images that are better suited to reflecting the spatial features of the three-dimensional structure being detected.

[0024] Furthermore, in the image processing apparatus, it is preferable that the image selection unit calculates the image similarity between the current frame image and the previous frame image, selects the current frame image if the image similarity does not exceed a threshold, and the 3D feature map generation unit places the current frame image selected by the image selection unit into a plurality of images constituting the 3D image. This allows for the selection of multiple images whose image similarity does not exceed a threshold, making them more suitable for reflecting the spatial features of the 3D structure being detected.

[0025] Furthermore, the image processing apparatus preferably includes a spatial coordinate acquisition unit that acquires spatial coordinates corresponding to each of a plurality of images in a time series, the image selection unit calculates the coordinate similarity between the current frame image and the previous frame image based on the spatial coordinates, and selects the current frame image if the coordinate similarity does not exceed a threshold, and the 3D feature map generation unit preferably places the current frame image selected by the image selection unit into a plurality of images constituting the 3D image.

[0026] This allows for the selection of multiple images whose spatial coordinate similarity does not exceed a threshold, making them more suitable for reflecting the spatial features of the 3D structure being detected.

[0027] Furthermore, in the image processing apparatus, it is preferable that the image selection unit calculates the distance between the image center points and the previous frame image and the angle between images based on the spatial coordinates, and selects the current frame image when the distance between the image center points or the angle between images exceeds a threshold.

[0028] This allows for the selection of multiple images that are more suitable for reflecting the spatial features of the 3D structure of the object being detected, based on whether the distance or angle between images exceeds a threshold.

[0029] Furthermore, in the image processing apparatus, the image acquisition unit acquires a plurality of ultrasound medical images captured by an ultrasound probe, the 3D feature map generation unit generates a 3D feature map corresponding to the 3D image by combining the plurality of ultrasound medical images into a single 3D image, the image processing apparatus further comprises a detection unit that performs a detection task based on the 2D feature map, and the output unit further outputs the detection result of the detection task.

[0030] This allows a two-dimensional feature map that reflects the spatial characteristics of the three-dimensional structure of the object being detected in ultrasound examinations to be used in the detection task of ultrasound medical images, thereby improving examination accuracy.

[0031] Furthermore, in the image processing apparatus, it is preferable that the image acquisition unit sequentially acquires a plurality of images, the 3D feature map generation unit sequentially generates a 3D feature map based on the sequentially acquired plurality of images, the 2D feature map generation unit sequentially generates a 2D feature map by compressing the sequentially generated 3D feature map, and the output unit sequentially outputs the 2D feature map.

[0032] This allows for real-time image acquisition tasks, where images are acquired sequentially one by one, and corresponding 2D feature maps are output.

[0033] Furthermore, in the image processing apparatus, it is preferable that the image acquisition unit sequentially acquires a predetermined number of images.

[0034] This allows for the sequential output of 2D feature maps corresponding to a predetermined number of images.

[0035] Furthermore, it is preferable that the image processing apparatus further includes an image alignment unit that aligns a plurality of images in order of scanning direction, the 3D feature map generation unit generates a 3D feature map of the 3D image by treating the aligned plurality of images as a single 3D image, and the 2D feature map generation unit generates the 2D feature map by compressing the 3D feature map in the alignment direction.

[0036] This allows for the output of a two-dimensional feature map that reflects the spatial characteristics of the three-dimensional structure of the object being detected, depending on the scanning direction, such as the rotation of the probe.

[0037] An image processing method according to one embodiment includes an image acquisition step of acquiring multiple images; a 3D feature map generation step of generating a 3D feature map corresponding to the 3D image by treating the multiple images as a single 3D image; a 2D feature map generation step of generating a 2D feature map by compressing the 3D feature map; and an output step of outputting the 2D feature map.

[0038] This allows for the output of a two-dimensional feature map that reflects the spatial characteristics of the three-dimensional structure of the object being detected.

[0039] An ultrasound diagnostic apparatus according to one embodiment includes an ultrasound probe that can move to image a subject, a display, and an image processing unit, wherein the image processing unit includes an image acquisition unit that acquires a plurality of ultrasound medical images in a time series acquired by the ultrasound probe, a 3D feature map generation unit that generates a 3D feature map corresponding to the 3D image by combining the plurality of ultrasound medical images into a single 3D image, a 2D feature map generation unit that generates a 2D feature map by compressing the 3D feature map, a detection unit that performs a detection task based on the 2D feature map, and an output unit that outputs the detection result of the detection task, wherein the display displays at least one frame of the ultrasound medical image and the detection result of the detection task.

[0040] This allows a two-dimensional feature map that reflects the spatial characteristics of the three-dimensional structure of the object detected by ultrasound to be used in the detection task of ultrasound medical images, thereby improving the detection accuracy of the ultrasound diagnostic device.

[0041] One embodiment of a computer program product includes a program for causing a computer to execute an image processing method which includes an image acquisition step of acquiring multiple images, a 3D feature map generation step of generating a 3D feature map corresponding to the 3D image by combining the multiple images into a single 3D image, a 2D feature map generation step of generating a 2D feature map by compressing the 3D feature map, and an output step of outputting the 2D feature map.

[0042] This allows for the output of a two-dimensional feature map that reflects the spatial characteristics of the three-dimensional structure of the object being detected.

[0043] The specific embodiments disclosed in this application will be described below with reference to the drawings.

[0044] (First Embodiment) Figure 1 is a block diagram showing the functional configuration of an image processing apparatus 100 according to the first embodiment. As shown in Figure 1, the image processing apparatus 100 comprises an image acquisition unit 110, a three-dimensional feature map generation unit 120, a two-dimensional feature map generation unit 130, and an output unit 140. The image processing apparatus 100 is a device that acquires a two-dimensional feature map from an image queue composed of multiple images, and will be described in detail below. Here, the image processing apparatus 100 is an example of a medical image processing apparatus.

[0045] The image acquisition unit 110 acquires multiple medical images (hereinafter referred to as "images"). Here, the image acquisition unit 110 is an example of an image acquisition unit.

[0046] Specifically, the image acquisition unit 110 acquires multiple medical images in a time series.

[0047] For example, the image acquisition unit 110 acquires multiple images from an image acquisition device. For example, the image acquisition unit 110 acquires multiple image frames within a certain period of time from an ultrasound diagnostic device via a wired or wireless communication connection. The ultrasound diagnostic device is, for example, a B-mode ultrasound diagnostic device, an ultrasound Doppler diagnostic device, etc. The image acquisition unit 110 may also acquire medical images from other medical image acquisition devices such as an endoscopic diagnostic device. The image acquisition unit 110 may also acquire images from image acquisition devices other than medical image acquisition devices. Furthermore, the image acquisition unit 110 may acquire multiple images that have been taken in advance, rather than acquiring images in real time. The image acquisition unit 110 may also be the image acquisition device itself that has an external communication function.

[0048] The imaging position, angle, etc., of the multiple images acquired by the image acquisition unit 110 change in three-dimensional space as the probe of the image acquisition device moves over a living body such as a person, so that the three-dimensional spatial structure of the detected object, which is a tissue, organ, etc. in a living body such as a person, is reflected in three-dimensional space.

[0049] When an image acquisition device acquires medical images in real time at a constant sampling frequency, the image acquisition unit 110 may, for example, acquire images of several frames within a certain time period as multiple images, meaning the image acquisition frequency of the image acquisition unit 110 may be equal to the sampling frequency of the image acquisition device. Alternatively, considering the computational power of real-time calculations, the image acquisition frequency of the image acquisition unit 110 may be lower than the sampling frequency of the image acquisition device, for example, acquiring images every few frames.

[0050] The 3D feature map generation unit 120 and the 2D feature map generation unit 130 generate feature maps based on multiple medical images acquired by the image acquisition unit 110. Here, the 3D feature map generation unit 120 and the 2D feature map generation unit 130 are examples of feature map generation units.

[0051] Specifically, the 3D feature map generation unit 120 generates a 3D feature map based on multiple medical images acquired by the image acquisition unit 110.

[0052] More specifically, the 3D feature map generation unit 120 constructs a 3D image by arranging multiple medical images acquired by the image acquisition unit 110 in chronological order, and generates a 3D feature map based on this 3D image.

[0053] In this case, for example, the 3D feature map generation unit 120 constructs a 3D image using a predetermined number of images from among the multiple medical images acquired by the image acquisition unit 110.

[0054] Furthermore, for example, the 3D feature map generation unit 120 generates a 3D feature map by applying 3D convolution to a 3D image using a convolutional neural network model.

[0055] For example, the 3D feature map generation unit 120 treats multiple images acquired by the image acquisition unit 110 as a single 3D image (image body, volume) and generates a 3D feature map corresponding to that 3D image. Figure 2 is a schematic diagram showing the generation of a 3D feature map from a 3D image in the first embodiment.

[0056] First, as shown in Figure 2, the 3D feature map generation unit 120 sequentially arranges the multiple images acquired by the image acquisition unit 110 into a 3D image that serves as a single image queue. For example, if the image acquisition unit 110 acquires images in real time according to the acquisition timing of the image collection device, the 3D feature map generation unit 120 arranges the multiple images in chronological order into a single image queue. Note that the order in which the multiple images are arranged is not limited to this; for example, if the image acquisition unit 110 acquires multiple images that have been taken in advance rather than acquiring images in real time, the multiple images may be arranged according to information such as the shooting angle.

[0057] The 3D feature map generation unit 120 ensures that the number of images constituting the 3D image does not exceed a predetermined threshold. For example, the number of frames of the images in the time series constituting the 3D image does not exceed a predetermined number of frames. This reduces the computational complexity and improves real-time performance by constructing a 3D image from a finite number of images.

[0058] Furthermore, as shown in Figure 2, if we define the resolution per image as W0 × H0, the number of images constituting the 3D image as L0, and the number of RGB image channels as 3, then the size of the 3D image will be L0 × W0 × H0 × 3.

[0059] The 3D feature map generation unit 120 generates a 3D feature map by performing a 3D convolution on a 3D image, for example, using a convolutional neural network (CNN) model. This 3D image corresponds to the input layer of the convolutional neural network and is input to the convolutional neural network model. The 3D convolution operation is performed on the input using a 3D convolution operator as the convolution kernel, and the 3D feature map is output. This convolutional neural network model may have, for example, one or more convolutional layers. The 3D feature map is input to the next convolutional layer as the output of the convolutional layer to generate a higher-generation 3D feature map. Each convolutional layer performs operations with one or more 3D convolution operators, and the multiple 3D convolution operators may have different sizes. This convolutional neural network model may also further include a pooling layer for dimensionality reduction of the output 3D feature map.

[0060] In convolutional neural network models, performing convolution across spatial dimensions using a 3D convolution operator preserves the structural relationships in the detected region's space, thereby allowing for a better interpretation of the spatial correlation between multiple images in 3D space.

[0061] The details of the 3D convolution performed in the convolutional neural network model are not limited; for example, parameters such as the size of the 3D convolution operator and the step size of the convolution operation can be set as appropriate depending on the situation and computing power. It is preferable to reduce the computational load by replacing the 2D convolution operator in conventional 2D real-time detection networks with a 3D convolution operator, for example, by using a cube corresponding to the size of the 2D convolution operator as the convolution kernel.

[0062] Note that this is not the only method for generating 3D feature maps from 3D images; for example, other neural network models besides CNNs or other image feature extraction algorithms may also be used.

[0063] As shown in Figure 2, the 3D feature map generated by the 3D feature map generation unit 120 is F 3D It can be expressed as =f(l,w,h,c), and its size is L1×W1×H1×C1. Here, W1, H1, and L1 are the width, height, and length of the 3D feature map, respectively, and C1 is the number of channels in the 3D feature map.

[0064] The 2D feature map generation unit 130 generates a 2D feature map by compressing the 3D feature map generated by the 3D feature map generation unit 120.

[0065] Specifically, the 2D feature map generation unit 130 generates a 2D feature map by compressing the 3D feature map generated by the 3D feature map generation unit 120 in the time-series direction.

[0066] For example, the 2D feature map generation unit 130 compresses the 3D feature map in the time series direction by averaging the 3D feature map generated by the 3D feature map generation unit 120 in the time series direction.

[0067] Figure 3 is a schematic diagram illustrating the generation of a two-dimensional feature map by compressing a three-dimensional feature map in the first embodiment.

[0068] As shown in Figure 3, the 2D feature map generation unit 130 compresses the 3D feature map in the length direction (L1), and the size of the compressed 2D feature map becomes W1 × H1 × C1. For example, if the 3D feature map generation unit 120 generates a 3D feature map by arranging multiple images in a time series into a single image queue, the length direction of the 3D feature map becomes the time series direction, and the 2D feature map generation unit 130 compresses the 3D feature map in the time series direction.

[0069] For example, the 2D feature map generation unit 130 generates a 3D feature map (F) as shown in equation (1) below. 3D By averaging the 3D feature map in the length (L1) direction, the 2D feature map (F2D ) generates.

[0070]

number

[0071] Furthermore, the method used by the 2D feature map generation unit 130 to compress the 3D feature map is not limited to averaging; other compression methods may also be used.

[0072] The output unit 140 outputs a two-dimensional feature map generated by the two-dimensional feature map generation unit 130. Specifically, for example, the two-dimensional feature map is output to an external detection device, which then performs a detection task based on the two-dimensional feature map. The detection task includes, but is not limited to, target detection, image segmentation, and image classification. For example, the detection device performs a medical image detection task in real time based on the two-dimensional feature map output in real time from the output unit 140.

[0073] The two-dimensional feature map output from the output unit 140 is a compressed version of the three-dimensional feature map, and therefore includes three-dimensional features between multiple images, in other words, three-dimensional structural relationships in the space of the target region detected by capturing multiple images. As a result, the image processing device 100 can output a two-dimensional feature map that reflects the spatial features of the three-dimensional structure of the object to be detected, and by using this two-dimensional feature map in the detection task, detection accuracy, for example, the detection accuracy of lesions in medical image detection, can be improved.

[0074] The image processing method performed by the image processing device 100 will now be described with reference to Figure 4. Figure 4 is a flowchart showing the image processing method performed by the image processing device 100 according to the first embodiment.

[0075] First, the image acquisition unit 110 acquires multiple images from the image acquisition device (S110). Then, the 3D feature map generation unit 120 combines the multiple images acquired by the image acquisition unit 110 into a single 3D image and generates a 3D feature map corresponding to that 3D image (S120). Next, the 2D feature map generation unit 130 generates a 2D feature map by compressing the 3D feature map generated by the 3D feature map generation unit 120 (S130). Finally, the output unit 140 outputs the 2D feature map generated by the 2D feature map generation unit 130 (S140). The image processing method described above makes it possible to output a 2D feature map that reflects the spatial features of the 3D structure of the object to be detected. If this 2D feature map is used in a detection task, the detection accuracy, for example, the detection accuracy of lesions in medical image detection, can be improved.

[0076] (Second embodiment) Figure 5 is a block diagram showing the functional configuration of the image processing apparatus 101 according to the second embodiment. As shown in Figure 5, the image processing apparatus 101 includes an image acquisition unit 111, a three-dimensional feature map generation unit 121, a two-dimensional feature map generation unit 131, an output unit 141, and an image sorting unit 151. In the second embodiment, we specifically consider the case in which the image processing apparatus 101 generates a two-dimensional feature map in real time as the probe of the medical image acquisition device moves on the surface of the subject in real time. The differences between the second embodiment and the first embodiment will be described below. Here, the image processing apparatus 101 is an example of a medical image processing apparatus.

[0077] The image acquisition unit 111 receives medical images collected in real time by the probe of the medical image acquisition device; in other words, medical images are continuously input chronologically during a single acquisition task of the medical image acquisition device.

[0078] The image sorting unit 151 corresponds to multiple medical images acquired by the image acquisition unit 111 and acquires three-dimensional spatial information representing the characteristics of the three-dimensional space. Here, the image sorting unit 151 is an example of a spatial information acquisition unit.

[0079] Specifically, the image sorting unit 151 acquires the similarity between images contained in multiple medical images acquired by the image acquisition unit 111 as three-dimensional spatial information.

[0080] At this time, for example, the image sorting unit 151 obtains a value as similarity that represents the error between images included in multiple medical images obtained by the image acquisition unit 111.

[0081] Alternatively, for example, the image selection unit 151 obtains a value representing the distance between images included in a plurality of medical images, based on the spatial coordinates corresponding to each of the plurality of medical images acquired by the image acquisition unit 111, as a similarity score.

[0082] For example, the image selection unit 151 selects images from a plurality of images acquired by the image acquisition unit 111 and inputs the selected images as a single image queue to the 3D feature map generation unit 121. Figure 6 is a block diagram showing the functional configuration of the image selection unit 151 according to the second embodiment. As shown in Figure 6, the image selection unit 151 includes a current frame management module 1511, an image queue management module 1512, a similarity module 1513, an output module 1514, and a spatial coordinate acquisition module 1515.

[0083] The following describes in detail the processing flow when an image is selected by the image selection unit 151 from multiple images acquired by the image acquisition unit 111 and input to the 3D feature map generation unit 121, with reference to Figure 7. Figure 7 shows a flowchart executed when images are selected in real time in the second embodiment.

[0084] First, in one acquisition task of the medical image acquisition device, the image acquisition unit 111 checks whether or not it has received an image frame from the medical image acquisition device (S111), and waits until it receives an image frame (S111:NO).

[0085] When the image acquisition unit 111 receives an image frame from the medical image acquisition device (S111:YES), it transmits that image frame to the current frame management module 1511 of the image selection unit 151. The current frame management module 1511 updates the current frame with the image frame (S112) and transmits the updated current frame to the similarity module 1513 of the image selection unit 151.

[0086] The similarity module 1513 of the image selection unit 151 determines whether or not to add the current frame to the image queue (S113). For example, the similarity module 1513 unconditionally adds the first received current frame to the queue, and then, for subsequent updated current frames, determines whether or not to add them to the image queue according to specific selection criteria. The specific selection criteria will be described later.

[0087] If the similarity module 1513 determines that it will not add the image frame to the image queue (S113:NO), the image selection unit 151 waits until the current frame management module 1511 updates the current frame next time.

[0088] If the similarity module 1513 determines that the current frame should be placed in the image queue (S113: YES), it instructs the image queue management module 1512 to place the current frame in the image queue. The image queue management module 1512 then determines whether the queue length of the updated image queue is greater than a threshold (S114).

[0089] If the image queue management module 1512 determines that the queue length of the updated image queue is not greater than the threshold (S114: NO), it outputs the updated image queue to the 3D feature map generation unit 121 via the output module 1514, and causes the 3D feature map generation unit 121 to generate a 3D feature map corresponding to this 3D image, treating the image queue as a single 3D image (S120).

[0090] If the image queue management module 1512 determines that the queue length of the updated image queue is greater than the threshold (S114: YES), the image queue management module 1512 deletes the first frame in the image queue (S115), and then outputs the updated image queue to the 3D feature map generation unit 121 via the output module 1514. Here, the first frame is the earliest (oldest) image frame in the time series direction of the image queue. The threshold for the queue length is preferably 5 or more and 15 or less, and more preferably 10. This makes it possible to reduce the computational load and improve real-time performance by constructing a 3D image with an image queue that does not exceed a predetermined threshold.

[0091] The sorting process performed by the similarity module 1513 of the image sorting unit 151 will be described in detail below with reference to Figure 8. Figure 8 is a flowchart showing a specific example in step S113 of Figure 7.

[0092] When the current frame is updated in step S112, the similarity module 1513 first obtains the last frame in the current image queue from the image queue management module 1512 (S1131). Here, the last frame is the latest (newest) image frame in the time series direction of the image queue.

[0093] Subsequently, the similarity module 1513 calculates the similarity between the current frame and the last frame (S1132). The similarity may be at least one of image similarity and spatial coordinate similarity.

[0094] When calculating image similarity, the similarity module 1513 uses, for example, the Mean Absolution Difference (MAD) algorithm to calculate the similarity as the average MAD of the error between each pixel between the current frame and the last frame, according to equation (2) below.

[0095]

number

[0096] Here, I c This is the pixel data of the current frame, and I L0 This is the pixel data of the last frame.

[0097] The method for calculating image similarity is not limited to this; for example, image similarity may be evaluated using hash values.

[0098] To calculate spatial coordinate similarity, the spatial coordinate acquisition module 1515 must first acquire the spatial coordinates corresponding to each of the multiple images in a time series from the medical image acquisition device. The medical image acquisition device, for example, has a positioning device such as a magnetic field coordinate transmitter built into its probe, and transmits the spatial coordinate information when the medical image is captured to the spatial coordinate acquisition module 1515 by adding it to the image header information. The similarity module 1513, for example, calculates the image center distance (ICD) using the spatial coordinate information, and calculates the similarity as the distance ICD between the center point coordinates of the current frame and the last frame using the following equation (3).

[0099]

number

[0100] Here, C c This is the coordinate of the center point of the current frame, C LO This is the coordinate of the center point of the last frame.

[0101] The method for calculating spatial coordinate similarity is not limited to this; for example, the angle between the plane normal vectors of the current frame and the last frame may be calculated to evaluate similarity. In other words, the angle between images obtained from a medical image acquisition device may be used to evaluate similarity. Alternatively, both the distance between center point coordinates and the angle between images may be calculated.

[0102] Furthermore, if the similarity is not calculated using spatial coordinates, the image selection unit 151 does not need to include the spatial coordinate acquisition module 1515.

[0103] Subsequently, the similarity module 1513 determines whether the similarity between the current frame and the last frame is less than a threshold (S1133). Specifically, the similarity module 1513 determines that the similarity between the current frame and the last frame is less than a threshold if at least one of the following conditions is met: the calculated image similarity is less than a threshold, or the spatial coordinate similarity is less than a threshold. For example, when the similarity module 1513 calculates similarity using image similarity, it determines that the similarity between the current frame and the last frame is less than a threshold if the average error MAD between the current frame and the last frame exceeds a threshold. For example, when the similarity module 1513 calculates similarity using spatial coordinate similarity, it may determine that the similarity between the current frame and the last frame is less than a threshold if at least one of the distance between the center point coordinates or the angle between the images exceeds a threshold.

[0104] If the similarity module 1513 determines that the similarity between the current frame and the last frame is less than a threshold (S1133:YES), it instructs the image queue management module 1512 to place the current frame at the end of the image queue, making it the updated last frame (S1134). Subsequently, if the current frame is updated again in step S112, the similarity module 1513 retrieves the updated last frame from the image queue management module 1512. If the image queue management module 1512 updates the last frame (S113:YES), it proceeds to the determination in step S114.

[0105] If the similarity module 1513 determines that the similarity between the current frame and the last frame is not below a threshold (S1133:NO), it returns to step S111 without adding the current frame to the image queue (S113:NO) and waits until the next image frame is acquired.

[0106] Furthermore, the similarity module 1513 may determine not only the similarity between the current frame and the last frame, but also the similarity between the current frame and other frames or multiple other frames in the image queue.

[0107] When the probe of the medical image acquisition device moves slowly or is stationary, the similarity between the generated images is high, and therefore the image queue composed of these frames cannot reflect the actual spatial structure of the object being detected. Consequently, the selection judgment by the similarity module 1513 described above selects only images with a similarity below the threshold, so that multiple images whose similarity does not exceed the threshold and are more suitable for reflecting the spatial features of the three-dimensional structure of the object being detected can be selected.

[0108] In the second embodiment, the 3D feature map generation unit 121 and the 2D feature map generation unit 131 generate feature maps based on a plurality of medical images acquired by the image acquisition unit 111 and 3D spatial information acquired by the image selection unit 151. Here, the 3D feature map generation unit 121 and the 2D feature map generation unit 131 are examples of feature map generation units.

[0109] The 3D feature map generation unit 121 generates a 3D feature map based on multiple medical images acquired by the image acquisition unit 111 and 3D spatial information acquired by the image selection unit 151.

[0110] Specifically, the 3D feature map generation unit 120 constructs a 3D image by arranging multiple medical images acquired by the image acquisition unit 110 in chronological order, and generates a 3D feature map based on this 3D image.

[0111] At this time, the 3D feature map generation unit 121 constructs a 3D image using medical images with a similarity below a threshold, selected by the image selection unit 151 from among multiple medical images acquired by the image acquisition unit 111.

[0112] For example, the 3D feature map generation unit 121 constructs a 3D image by using medical images whose similarity is below a threshold as medical images whose error value between images obtained by the image selection unit 151 exceeds a threshold.

[0113] Alternatively, for example, the 3D feature map generation unit 121 constructs a 3D image by using medical images whose similarity is less than the threshold as medical images whose distance value between images obtained by the image selection unit 151 exceeds a threshold.

[0114] The 3D feature map generation unit 121 constructs the selected image queue into a single 3D image and generates a 3D feature map based on 3D convolution. Since this process is the same as that of the 3D feature map generation unit 120 in the first embodiment, a detailed explanation will be omitted.

[0115] The 2D feature map generation unit 131 generates a 2D feature map by compressing the 3D feature map generated by the 3D feature map generation unit 121.

[0116] Specifically, the 2D feature map generation unit 131 assigns weights to the 3D feature map generated by the 3D feature map generation unit 121 for each position in the time series direction, and compresses the 3D feature map according to these weights to generate a 2D feature map.

[0117] In this case, for example, the 2D feature map generation unit 131 assigns larger weights to the 3D feature map generated by the 3D feature map generation unit 121 for positions closer to the current frame in the time series direction.

[0118] Furthermore, for example, the 2D feature map generation unit 131 uses weights obtained by training a machine learning model, which uses weights set for each position in the time series direction as learning parameters.

[0119] For example, as shown in the following formula (4), in the time series direction, the two-dimensional feature map generation unit 131 assigns a time series weight weight(l) to the three-dimensional feature map (F 3D ) generated by the three-dimensional feature map generation unit 121, compresses the weighted three-dimensional feature map in the time series direction, and generates a two-dimensional feature map (F 2D ).

[0120]

Number

[0121] In the second embodiment, since it is expected to use a two-dimensional feature map reflecting the spatial features of the three-dimensional structure of the detection target for real-time detection of medical images in the current frame, when compressing the three-dimensional feature map in the time series direction, a time series weight is assigned in consideration of the contribution degree of each image in the image queue for the detection task.

[0122] For example, since the frames at times closer to the current frame are expected to have a greater contribution degree to the detection task, higher time series weights are assigned to the regions closer to the current frame in the time series direction. That is, the weight function weight(l) in the above formula (4) is considered to be a monotonically increasing function, whereby, in the three-dimensional feature map, the regions closer to the current frame have higher time series weights. For example, a weight function as shown in the following formula (5) can be set.

[0123]

Number

[0124] The parameters δ and γ here can be set according to actual experience.

[0125] Furthermore, it is conceivable to train weight parameters (weight(1), weight(2), ..., weight(L1)) that can be learned by a machine learning model. In other words, for the weight function weight(l), a reasonable time-series weight can be learned during the training process using a machine learning model such as a neural network. For example, the parameters δ and γ in equation (5) above can be, for example, the learning parameters in a neural network.

[0126] The processing performed by the output unit 141 is the same as the processing performed by the output unit 140 in the first embodiment, which outputs a two-dimensional feature map, so a description will be omitted.

[0127] The above describes an embodiment in which the image selection unit 151 selects an image from multiple images acquired by the image acquisition unit 111 when the probe of a medical image acquisition device moves in real time across the surface of the patient. However, when a medical image acquisition device, such as an ultrasound diagnostic device, performs a single real-time image acquisition task, the image processing device 101 may sequentially acquire each frame image and sequentially output a two-dimensional feature map for each frame image. In other words, the image processing device 101 does not necessarily have to include the image selection unit 151.

[0128] Specifically, in the image processing device 101, the image acquisition unit 111 outputs a plurality of sequentially acquired images, including the image frame, to the 3D feature map generation unit 121 each time it receives an image frame from the medical image acquisition device. The image acquisition unit 111 also outputs a predetermined number of sequentially acquired images to the 3D feature map generation unit 121.

[0129] The 3D feature map generation unit 121 generates a 3D feature map for each image frame acquired one by one by sequentially arranging the acquired images into a single 3D image sequence based on the multiple images acquired sequentially by the image acquisition unit 111.

[0130] Similarly, the 2D feature map generation unit 131 sequentially generates 2D feature maps by compressing the 3D feature maps sequentially generated by the 3D feature map generation unit 121, and the output unit 141 sequentially outputs the 2D feature maps.

[0131] As a result, the image acquisition unit 111 acquires image frames one by one sequentially, and the output unit 141 sequentially outputs the corresponding two-dimensional feature maps. This improves the real-time and continuity of the generation of two-dimensional feature maps, enabling the system to handle real-time image acquisition tasks.

[0132] According to the image processing apparatus 101 of the second embodiment, as the probe of the medical image acquisition device moves over the surface of the subject, a two-dimensional feature map reflecting the spatial features of the three-dimensional structure of the object to be detected can be generated in real time. If this two-dimensional feature map is used for the detection task of the medical image of the current frame, the detection accuracy can be improved.

[0133] (Third embodiment) Figure 9 is a block diagram showing the functional configuration of the image processing apparatus 102 according to the second embodiment. As shown in Figure 9, the image processing apparatus 102 includes an image acquisition unit 112, a three-dimensional feature map generation unit 122, a two-dimensional feature map generation unit 132, an output unit 142, an image sorting unit 152, and a detection unit 162. In the third embodiment, in addition to the second embodiment, we specifically consider a situation in which the image processing apparatus 102 further has a function to perform a detection task on medical images. In particular, we consider the case in which the image processing apparatus 102 is used for ultrasound diagnosis. The differences between the third embodiment and the second embodiment will be explained below. Here, the image processing apparatus 102 is an example of a medical image processing apparatus.

[0134] The processing performed by the image acquisition unit 112, the 3D feature map generation unit 122, the 2D feature map generation unit 132, and the image sorting unit 152 in the third embodiment is similar to the processing performed by the image acquisition unit 111, the 3D feature map generation unit 121, the 2D feature map generation unit 131, and the image sorting unit 151 in the second embodiment. Since the 2D feature map is generated in real time as the ultrasound probe moves on the subject's surface in real time, a detailed explanation is omitted.

[0135] The detection unit 162 performs processing on the target medical image based on the feature maps generated by the 3D feature map generation unit 122 and the 2D feature map generation unit 132. Here, the detection unit 162 is an example of an execution unit.

[0136] Specifically, the detection unit 162 performs processing on the target medical image based on the two-dimensional feature map generated by the two-dimensional feature map generation unit 132.

[0137] For example, the detection unit 162 performs a target extraction task as processing for the target medical image, which involves detecting the target object in the target medical image.

[0138] Furthermore, for example, the detection unit 162 performs an image segmentation task as processing for the target medical image, which involves segmenting the target medical image.

[0139] Furthermore, for example, the detection unit 162 performs an image classification task to determine the type of the target medical image as part of the processing of the target medical image.

[0140] Figure 10 is a block diagram showing the functional configuration of the detection unit 162 according to the third embodiment. As shown in Figure 10, the detection unit 162 includes a target extraction module 1621, an image segmentation module 1622, and an image classification module 1623.

[0141] The target extraction module 1621 detects whether a target lesion or organ region exists in the current frame (i.e., the last frame of the uncompressed image queue corresponding to the 2D feature map) based on the 2D feature map input from the 2D feature map generation unit 132. If a target exists, it encloses the location of the target in a prompt box and labels the type of target. In other words, the target extraction module 1621 performs the target extraction task for ultrasound diagnosis.

[0142] For example, the target extraction module 1621 takes a 2D feature map as input and performs the target extraction task using a neural network model trained on several types of lesions. During the training process of this neural network model, training is performed based, for example, on a loss function between candidate boxes of the target to be extracted and physician annotation boxes.

[0143] The image segmentation module 1622 performs image segmentation on the current frame (i.e., the last frame of the uncompressed image queue corresponding to the 2D feature map) based on the 2D feature map input from the 2D feature map generation unit 132. For example, it labels the pixel regions of the target lesion or organ as the foreground and the other pixel regions as the background. In other words, the target extraction module 1621 performs the image segmentation task for ultrasound diagnosis.

[0144] For example, the image segmentation module 1622 takes a 2D feature map as input and performs the image segmentation task using a neural network model trained on several types of lesions and organs. During the training process of this neural network model, training is performed based on loss functions such as cross-entropy or Dice Loss.

[0145] The image classification module 1623 determines the image type of the current frame (i.e., the last frame of the uncompressed image queue corresponding to the 2D feature map) based on the 2D feature map input from the 2D feature map generation unit 132. The image type is, for example, the type of disease or the time phase. In other words, the target extraction module 1621 performs the image classification task for ultrasound diagnosis.

[0146] For example, the image classification module 1623 takes a two-dimensional feature map as input and performs an image classification task using a neural network model trained on several disease types and time phases. This neural network model corresponds to, for example, a fully connected layer in a convolutional neural network.

[0147] Because this neural network model takes a 2D feature map as input, it reduces computational complexity and improves the feasibility of real-time detection compared to performing detection tasks with a 3D feature map as input, while still reflecting the spatial features of the 3D structure of the target being detected. Furthermore, it allows the use of neural network models from conventional 2D real-time detection networks.

[0148] The detection unit 162 may selectively include one or more of the above modules depending on the detection needs. The output unit 142 outputs the detection results from the detection unit 162.

[0149] The detection result from the detection unit 162 is output to the external display of the image processing device 102 via the output unit 142. The image processing device 102 may also have a display unit (not shown) that displays the ultrasound image captured from the ultrasound probe and labels the current frame of the ultrasound image using the detection result from the detection unit 162. For example, a prompt box indicating the type and location of the object to be detected by the target extraction module 1621 is superimposed and displayed on the ultrasound medical image of the current frame. Furthermore, the output of the detection result is not limited to the display device, but may also be output to an audio guidance device or the like.

[0150] The ultrasound diagnostic apparatus 1000 using the image processing device 102 will be described below. Figure 11 is a block diagram showing the functional configuration of the ultrasound diagnostic apparatus 1000 using the image processing device 102 according to the third embodiment. As shown in Figure 11, the ultrasound diagnostic apparatus 1000 comprises an ultrasound probe 200, an image processing device 102, and a display 300. In the ultrasound diagnostic apparatus 1000, the image processing device 102 functions as an image processing unit.

[0151] The ultrasound probe 200 is wirelessly or wiredly connected to the image processing device 102 and the display 300, and transmits image signals captured in real time to the image processing device 102 and the display 300 as the ultrasound probe 200 moves over the surface of the patient. The display 300 receives the image signals from the ultrasound probe 200 and displays the images in real time. The image processing device 102 receives the image signals from the ultrasound probe 200, generates a two-dimensional feature map as described above, performs a detection task related to ultrasound diagnosis, outputs the detection results to the display 300, and the display 300 uses these detection results to label the ultrasound images.

[0152] Figure 12 is a timing chart showing the operating timing of each component in the ultrasound diagnostic apparatus 1000 using the image processing apparatus 102 according to the third embodiment. As shown in Figure 12, when the ultrasound probe 200 captures one frame of ultrasound image, the image of the current frame is transmitted to the image processing apparatus 102 and the display 300 (S1-1), and the display 300 displays the image.

[0153] When the image processing device 102 receives the ultrasound image of the current frame, it determines whether or not to add the current frame to the image queue. Specifically, the image sorting unit 152 of the image processing device 102 determines whether or not to add the current frame to the image queue, and the determination criterion is, for example, the similarity determination described in the second embodiment. If it is determined that the current frame should be added to the image queue, the image processing device 102 generates a corresponding two-dimensional feature map with the current frame as the last frame in the image queue. Then, the detection unit 162 of the image processing device 102 performs a detection task on the current frame based on the two-dimensional feature map, and the output unit 142 transmits the detection result to the display 300 (S2-1).

[0154] When the display 300 receives this detection result, it displays at least one frame of the ultrasound medical image and the detection result of the detection task. For example, the display 300 labels the detection result in the current frame. Specifically, it updates the previous label. This labeling includes, but is not limited to, a prompt box for labeling the lesion area, labeling pixels of the lesion area in the foreground, or labeling the type and phase of the disease.

[0155] Subsequently, after the ultrasound probe 200 acquires another frame of ultrasound image, the data flow described above is repeated. For example, as shown in Figure 12, after the ultrasound probe 200 transmits the current frame to the image processing device 102 and the display 300 (S1-2), if the image processing device 102 determines that it should not add it to the image queue, the generation of the 2D feature map and the corresponding detection are not performed, and by not transmitting data to the display 300, it sends an instruction to the display 300 to either retain the detection result labeled in the previous frame or delete the detection result labeled in the previous frame (S2-2).

[0156] The data flow described above is repeatedly executed between each component of the ultrasound diagnostic device 1000 during a single examination of a subject by the ultrasound diagnostic device 1000, and the labeling is updated again, for example, in S1-3 to S2-3. As a result, the detection results are labeled in real time in the current frame, enabling, for example, real-time tracking to a target lesion or target organ.

[0157] The ultrasound diagnostic device 1000 is used, for example, for the detection of primary liver cancer (HCC), metastatic liver cancer (Meta), hepatic hemangioma (Hema), and hepatic cysts (Cyst). Compared to detection based solely on two-dimensional features, it is measured that detection accuracy is improved by detecting the above-mentioned lesions using a two-dimensional feature map that reflects the spatial features of the three-dimensional structure of the target. Furthermore, because the above-mentioned lesions are detected using this two-dimensional feature map, the detection speed can meet the demands of real-time detection tasks. For example, when using a commercially available graphics processor, the processing speed can meet the demands of real-time detection tasks.

[0158] Figures 13 and 14 show examples of lesion detection results performed by the image processing device 102 according to the third embodiment.

[0159] In Figures 13 and 14, the image on the right shows the detection results of lesion detection output by the output unit 142 described above, while the image on the left shows the detection results when lesion detection is performed using a two-dimensional feature map that does not reflect the spatial features of the three-dimensional structure of the target, as a comparative example.

[0160] For example, as shown on the right side of Figure 13, the output unit 142 superimposes prompt boxes related to the type and location of the object detected by the target extraction module 1621 of the detection unit 162 onto the ultrasound medical image of the current frame, and displays them on the external display of the image processing device 102 (for example, the display 300 of the ultrasound diagnostic device 1000).

[0161] For example, as shown on the right side of Figure 13, if the detected lesion is a hemangioma (Hema), the output unit 142 superimposes a rectangular prompt box on the ultrasound medical image, labeled "Hema" indicating the type of hemangioma and the numerical value "0.72" indicating its location, based on the processing results of the target extraction task performed by the target extraction module 1621 of the detection unit 162, and displays it on an external display.

[0162] Here, for example, if the lesion to be detected is a hemangioma, as shown on the left side of Figure 13, in the comparative example, the spatial features of the three-dimensional tissue structure are lost, which may result in the hemangioma being missed. In contrast, as shown on the right side of Figure 13, the image processing device 102 according to the third embodiment uses a feature map that reflects the spatial features of the three-dimensional structure of the target to be detected to perform lesion detection, thereby enabling the correct detection of hemangiomas.

[0163] Furthermore, for example, if the lesion to be detected is a cyst, as shown on the left side of Figure 14, in the comparative example, the spatial features of the three-dimensional tissue structure are lost, which may result in the gallbladder being mistakenly detected as a cyst. In contrast, as shown on the right side of Figure 14, the image processing device 102 according to the third embodiment uses a feature map that reflects the spatial features of the three-dimensional structure of the target to be detected to perform lesion detection, thus preventing false detections.

[0164] In the third embodiment described above, the detection unit 162 performs processing on the target medical image based on the two-dimensional feature map generated by the two-dimensional feature map generation unit 132, but the embodiments are not limited to this.

[0165] For example, the detection unit 162 may perform processing on the target medical image based on the 3D feature map generated by the 3D feature map generation unit 122.

[0166] In this case, the target extraction module 1621 performs the ultrasound diagnostic target extraction task based on the 3D feature map input from the 3D feature map generation unit 122. For example, the target extraction module 1621 takes the 3D feature map as input and performs the target extraction task using a neural network model trained on several types of lesions. During the training process of this neural network model, training is performed, for example, based on a loss function between candidate boxes of the target to be extracted and physician annotation boxes.

[0167] Furthermore, the image segmentation module 1622 performs the ultrasound image segmentation task based on the 3D feature map input from the 3D feature map generation unit 122. For example, the image segmentation module 1622 takes the 3D feature map as input and performs the image segmentation task using a neural network model trained on several types of lesions and organs. During the training process of this neural network model, training is performed based on loss functions such as cross-entropy or Dice Loss.

[0168] Furthermore, the image classification module 1623 performs an ultrasound diagnostic image classification task based on the 3D feature map input from the 3D feature map generation unit 122. For example, the image classification module 1623 takes the 3D feature map as input and performs the image classification task using a neural network model trained for several disease types and time phases. This neural network model corresponds, for example, to the fully connected layer in a convolutional neural network.

[0169] Furthermore, if the detection unit 162 performs processing on the target medical image based on the 3D feature map generated by the 3D feature map generation unit 122, the image processing device 102 does not need to include the 2D feature map generation unit 132.

[0170] (Fourth embodiment) Figure 15 is a block diagram showing the functional configuration of the image processing apparatus 103 according to the fourth embodiment. As shown in Figure 15, the image processing apparatus 103 includes an image acquisition unit 113, a three-dimensional feature map generation unit 123, a two-dimensional feature map generation unit 133, an output unit 143, and an image alignment unit 173. The difference between the fourth embodiment and the embodiments described above is that while the embodiments described above considered configuring an image queue in the time-series direction and compressing the three-dimensional feature map in the time-series direction, the fourth embodiment considers rearranging images according to features other than time-series. The differences of the fourth embodiment will be described below. Here, the image processing apparatus 103 is an example of a medical image processing apparatus.

[0171] The image alignment unit 173 aligns multiple images acquired from the image acquisition unit 113 and transmits the image queue, which consists of the aligned multiple images, to the 3D feature map generation unit 123.

[0172] Specifically, the image alignment unit 173 aligns multiple images in the order of the scanning direction, which is, for example, the normal vector of the imaging plane. The image alignment unit 173 performs feature point matching on multiple images and calculates the normal vector of the imaging plane for each of the multiple images. The image acquisition unit 113 also acquires the normal vector of the imaging plane corresponding to each of the multiple images from, for example, a magnetic field coordinate oscillator or gyro sensor built into the probe of the image acquisition device. The image alignment unit 173 aligns the multiple images in the order in which the projection of the normal vector of the imaging plane onto a predetermined plane rotates clockwise, that is, in the order of rotation during probe scanning.

[0173] The 3D feature map generation unit 123 treats multiple aligned images as a single 3D image and generates a 3D feature map corresponding to that 3D image. The 2D feature map generation unit 133 compresses the 3D feature map in the alignment direction, which is aligned in the order of the scanning direction, and generates a 2D feature map.

[0174] This allows for the output of a two-dimensional feature map that reflects the spatial characteristics of the three-dimensional structure, which is aligned in the order of the scanning direction of the object being detected.

[0175] The image alignment unit 173 is not limited to aligning multiple images in the order of the scanning direction; for example, it may also align multiple images in a coordinate order in a predetermined direction.

[0176] Furthermore, in the fourth embodiment, the image processing apparatus 103 can also be used for real-time detection tasks of ultrasound medical images. That is, the image alignment unit 173 can select multiple image frames from a set including the current frame and several previous frames, and align them not in chronological order, but for example, in the order of the scanning direction.

[0177] In the image processing apparatus 103 of the fourth embodiment, by further arranging the image queue in the order of the scanning direction, a two-dimensional feature map that more faithfully reflects the spatial characteristics of the three-dimensional structure of the object to be detected can be output. If this two-dimensional feature map is used in the detection task, the detection accuracy can be further improved.

[0178] In the embodiments described above, the multiple functional units constituting the image processing apparatus can be implemented as software on a device equipped with an independent computer or other processor and memory, or they can be implemented in a distributed manner across multiple devices, with the processor executing each functional unit of the image processing apparatus stored in memory. Alternatively, each function of the image processing apparatus may be implemented in hardware as a circuit capable of executing it. The circuit realizing the image processing apparatus can transmit and receive data and collect data via a network such as the Internet. Furthermore, the functions of each part of the image processing apparatus may be realized as a computer program product by having a computer processor execute a program pre-stored on a recording medium.

[0179] Furthermore, in the embodiments described above, the image processing apparatus is not limited to being implemented by a single processor, but may be configured by combining multiple independent processors, with each processor executing a program to realize each processing function. Also, each function of the image processing apparatus may be implemented by appropriately distributing or integrating it across one or more processing circuits. Furthermore, each function of the image processing apparatus may be implemented by a mixture of hardware such as circuits and software. In addition, the programs corresponding to each function of the image processing apparatus may be stored in a single memory circuit, or the programs corresponding to each function may be distributed and stored across multiple memory circuits, and the image processing apparatus may read and execute each program from each memory circuit.

[0180] Furthermore, the processor used in the image processing device refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). Here, instead of storing the computer program in a memory circuit, the computer program may be directly incorporated into the processor circuit. In this case, the processor realizes its function by reading and executing the program incorporated into the circuit. Moreover, each processor in this embodiment is not limited to being configured as a single circuit; multiple independent circuits may be combined to form a single processor and realize its function.

[0181] Furthermore, the computer program product that realizes the functions of each part of the image processing device includes a computer program executed by the processor, and this computer program is provided pre-installed in ROM (Read Only Memory) or memory circuits. This computer program may also be provided as a file in an installable or executable format on these devices, recorded on a computer-readable, non-transient storage medium such as a CD (Compact Disk)-ROM, FD (Flexible Disk), CD-R (Recordable), or DVD (Digital Versatile Disk). Alternatively, this computer program may be stored on a computer connected to a network such as the Internet and provided or distributed by downloading it via the network. For example, this computer program consists of modules containing the functions described above. In actual hardware, the CPU reads the computer program from a storage medium such as ROM and executes it, loading each module onto the main memory and generating it in the main memory.

[0182] Furthermore, in the embodiments described above, each component of each illustrated device is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific form of distribution or integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed or integrated in any unit according to various loads and usage conditions. Moreover, each function performed by each device can be realized, all or any part thereof, by a CPU and a program that is analyzed and executed by the CPU, or by hardware using wired logic.

[0183] The various types of data discussed in this specification are typically digital data.

[0184] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be implemented in a variety of other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0185] 100, 101, 102, 103 Image processing device 110,111,112,113 Image acquisition unit 120, 121, 122, 123 3D Feature Map Generation Unit 130, 131, 132, 133 2D Feature Map Generation Unit 140, 141, 142, 143 Output section 151,152 Image sorting section 1511 Current Frame Management Module 1512 Image Queue Management Module 1513 Similarity Module 1514 Output Module 1515 Spatial Coordinate Acquisition Module 162 Detection unit 1621 Target Extraction Module 1622 Image Segmentation Module 1623 Image Classification Module 173 Image alignment section 1000 Ultrasound diagnostic equipment 200 Ultrasound Probes 300 displays

Claims

1. An image acquisition unit that acquires multiple medical images, A spatial information acquisition unit that acquires three-dimensional spatial information representing the characteristics of three-dimensional space, corresponding to the aforementioned multiple medical images, A feature map generation unit generates a feature map based on the aforementioned multiple medical images and the aforementioned three-dimensional spatial information, Based on the aforementioned feature map, an execution unit performs processing on the target medical image. A medical image processing device equipped with [a specific feature].

2. The feature map generation unit generates a three-dimensional feature map based on the plurality of medical images and the three-dimensional spatial information, and generates a two-dimensional feature map by compressing the three-dimensional feature map. The execution unit performs processing on the target medical image based on the two-dimensional feature map. The medical image processing apparatus according to claim 1.

3. The image acquisition unit acquires multiple medical images in a time series, The feature map generation unit generates the two-dimensional feature map by compressing the three-dimensional feature map in the time-series direction. The medical image processing apparatus according to claim 2.

4. The feature map generation unit constructs a three-dimensional image by arranging the multiple medical images in chronological order, and generates the three-dimensional feature map based on the three-dimensional image. The medical image processing apparatus according to claim 3.

5. The feature map generation unit constructs the three-dimensional image using a predetermined number of images from the plurality of medical images. The medical image processing apparatus according to claim 4.

6. The spatial information acquisition unit acquires the similarity between images included in the plurality of medical images as three-dimensional spatial information, The feature map generation unit constructs the three-dimensional image using the medical images among the plurality of medical images whose similarity is below a threshold. The medical image processing apparatus according to claim 4.

7. The spatial information acquisition unit acquires a value representing the error between images included in the plurality of medical images as the similarity, The feature map generation unit constructs the three-dimensional image by using medical images whose similarity is less than the threshold as medical images whose value representing the error between the images exceeds a threshold. The medical image processing apparatus according to claim 6.

8. The spatial information acquisition unit acquires a value representing the distance between images included in the plurality of medical images, based on the spatial coordinates corresponding to each of the plurality of medical images, as the similarity. The feature map generation unit constructs the three-dimensional image by using medical images whose similarity is less than the threshold as medical images whose distance value between the images exceeds the threshold. The medical image processing apparatus according to claim 6.

9. The feature map generation unit generates the three-dimensional feature map by applying three-dimensional convolution to the three-dimensional image using a convolutional neural network model. The medical image processing apparatus according to claim 4.

10. The feature map generation unit compresses the three-dimensional feature map in the time series direction by averaging the three-dimensional feature map in the time series direction. The medical image processing apparatus according to claim 3.

11. The feature map generation unit assigns weights to the three-dimensional feature map for each position in the time series direction, and compresses the three-dimensional feature map according to these weights to generate the two-dimensional feature map. The medical image processing apparatus according to claim 3.

12. The feature map generation unit assigns a larger weight to the 3D feature map for positions closer to the current frame in the time series direction. The medical image processing apparatus according to claim 11.

13. The feature map generation unit uses weights obtained by training a machine learning model, in which weights set for each position in the time series direction are used as learning parameters. The medical image processing apparatus according to claim 11.

14. The execution unit performs a target extraction process to detect a target object in the target medical image as processing for the target medical image. A medical image processing apparatus according to any one of claims 1 to 13.

15. The execution unit performs an image segmentation process on the target medical image as processing for the target medical image, which involves image segmentation of the target medical image. A medical image processing apparatus according to any one of claims 1 to 13.

16. The execution unit performs an image classification process to determine the type of the target medical image as processing for the target medical image. A medical image processing apparatus according to any one of claims 1 to 13.

17. It comprises an ultrasound probe capable of imaging a subject, and a processing circuit, The aforementioned processing circuit is An image acquisition unit that acquires multiple ultrasound medical images in a time series sequentially captured by the ultrasound probe, A spatial information acquisition unit that acquires three-dimensional spatial information representing the characteristics of three-dimensional space, corresponding to the aforementioned multiple ultrasound medical images, A feature map generation unit generates a feature map based on the plurality of ultrasound medical images and the three-dimensional spatial information, An execution unit that performs processing on the target ultrasound medical image based on the aforementioned feature map. An ultrasound diagnostic device equipped with the following features.

18. Steps to acquire multiple medical images, The steps include acquiring three-dimensional spatial information that represents the characteristics of three-dimensional space, corresponding to the aforementioned multiple medical images, The steps include generating a feature map based on the aforementioned multiple medical images and the three-dimensional spatial information, The steps include: performing processing on the target medical image based on the aforementioned feature map; A medical image processing method, including [the specified term].

19. Procedure for acquiring multiple medical images, A procedure for acquiring three-dimensional spatial information that represents the characteristics of three-dimensional space, corresponding to the aforementioned multiple medical images, A procedure for generating a feature map based on the aforementioned multiple medical images and the aforementioned three-dimensional spatial information, A procedure for performing processing on the target medical image based on the aforementioned feature map. A medical image processing program that causes a computer to perform certain actions.

Images