Image generation device, image generation method, and program

Abstract

To provide an image generation device, an image generation method, and a program capable of generating a high-quality medical image including a specific identification object by using a space compound method.SOLUTION: The device comprises: an image acquisition unit 14 which acquires a plurality of ultrasonic images generated based on a plurality of reception signals corresponding to respective reflected ultrasonic waves of ultrasonic waves transmitted in a plurality of different transmission directions; a structure identification unit 17 which inputs the acquired ultrasonic images to an identification model for identifying an identification object included in an inputted ultrasonic image and thereby acquires a certainty degree outputted from the identification model; and an image synthesis unit 18 which generates a synthetic image based on the ultrasonic images and the certainty degree.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to an image generation apparatus, an image generation method, and a program for generating medical images.

Background Art

[0002] Conventionally, there has been known an ultrasonic diagnostic apparatus that has an ultrasonic probe provided with an array of a large number of vibrators, transmits and receives ultrasonic waves to and from a subject such as a living body, generates ultrasonic image data based on signals obtained from the received ultrasonic waves, and displays an ultrasonic image based thereon on an image display device. Ultrasonic image diagnosis by such an apparatus can obtain in real time the state of the heartbeat of the heart, the movement of the fetus, etc. by a simple operation of merely applying the ultrasonic probe to the body surface of the subject, and since it is non-invasive and highly safe, it can be repeatedly performed.

[0003] However, in the images obtained by such an ultrasonic diagnostic apparatus, in addition to information regarding tissues within the subject, there are various noises and speckles generated by interference phenomena of received signals obtained from ultrasonic waves received by the ultrasonic probe, and these often become obstacles when accurately grasping the position and shape of the boundaries of tissues within the subject.

[0004] In recent years, as a processing method for reducing such noises and speckles, for example, an ultrasonic diagnostic apparatus using a spatial compounding method has become widespread. The spatial compounding method is a method in which ultrasonic waves are transmitted and received in a plurality of different directions at the same time with respect to the same part of the subject, and average superimposition of a plurality of acquired ultrasonic image data is performed. Thereby, noises and speckles are reduced by the square root of N in the composite image data obtained by synthesizing these, for example, when N pieces of ultrasonic image data are obtained.

[0005] Also, according to this spatial compounding method, the extraction performance of anisotropic sites can be improved. An anisotropic site is a site where the received signal intensity such as scattering and reflection when ultrasonic waves hit it varies depending on the angle. Specifically, for example, it is a site in soft tissue that is fibrous like tendons and ligaments in skeletal muscle within a subject, and although the reflection intensity is not as strong as that of the bone surface, it exhibits specular reflection characteristics.

[0006] An ultrasonic diagnostic apparatus using such a spatial compounding method is disclosed, for example, in Patent Document 1. Patent Document 1 discloses a technique for generating a higher-quality ultrasonic image by synthesizing the average value, maximum value, minimum value, median value, etc. of each pixel value of ultrasonic images obtained from reflection signals in a plurality of directions by a control signal selected according to the type of diagnostic examination for depicting a target with higher image quality.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0008] In the technique disclosed in Patent Document 1, for example, when targeting a nerve, the received signal from the nerve can be amplified, so the nerve can be depicted with high image quality. However, in the technique disclosed in Patent Document 1, elements other than the nerve, such as noise and received signals from other structures that do not need to be depicted, are also amplified, and as a result, the image quality of the entire ultrasonic image may deteriorate.

[0009] An object of the present disclosure is to provide an image generation apparatus, an image generation method, and a program capable of generating a medical image including a specific object to be identified with high image quality using a spatial compounding method.

Means for Solving the Problems

[0010] The image generation device of the present disclosure includes an image acquisition unit that acquires a plurality of ultrasonic images generated based on a plurality of received signals corresponding to respective reflected ultrasonic waves of transmitted ultrasonic waves transmitted in a plurality of different transmission directions, and an identifier that identifies an object to be identified shown in the input ultrasonic image. By inputting the acquired plurality of ultrasonic images to the identifier, based on the identification result output from the identifier, a confidence image is generated that shows the likelihood that the object is the object to be identified for each pixel of the ultrasonic image. An identification result acquisition unit, an image synthesis unit that generates a composite image based on the plurality of ultrasonic images and the confidence image, and an operation input unit that receives an operation. When there are a plurality of objects to be identified, the operation input unit receives an operation for setting a generation method of the confidence image for each object to be identified, and the identification result acquisition unit generates the confidence image for each object to be identified using the set generation method. either the first method or the second method When there are a plurality of objects to be identified, the operation input unit receives an operation for setting a generation method of the confidence image for each object to be identified, and the identification result acquisition unit generates the confidence image for each object to be identified using the set generation method. wherein the first method is a method of generating a plurality of first confidence images based on each of the plurality of ultrasonic images and synthesizing the plurality of first confidence images to generate the confidence image, and the second method is a method of generating the confidence image based on a maximum value image generated by extracting the maximum value of the pixel values for each pixel of the plurality of ultrasonic images or an average value image generated by calculating the average value of the pixel values for each pixel of the plurality of ultrasonic images 。

[0011] The image generation method of the present disclosure is an image generation method executed by a computer included in an image generation device. The computer acquires a plurality of ultrasonic images generated based on a plurality of received signals corresponding to respective reflected ultrasonic waves of transmitted ultrasonic waves transmitted in a plurality of different transmission directions, and inputs the acquired plurality of ultrasonic images to an identifier that identifies an object to be identified shown in the input ultrasonic image. Based on the identification result output from the identifier, a confidence image is generated that shows the likelihood that the object is the object to be identified for each pixel of the ultrasonic image. A composite image is generated based on the plurality of ultrasonic images and the confidence image. The generation of the confidence image includes, when there are a plurality of objects to be identified, a step of receiving an operation for setting a generation method of the confidence image for each object to be identified, and a step of generating the confidence image for each object to be identified using the set generation method. either the first method or the second method When there are a plurality of objects to be identified, the operation input unit receives an operation for setting a generation method of the confidence image for each object to be identified, and the identification result acquisition unit generates the confidence image for each object to be identified using the set generation method. wherein the first method is a method of generating a plurality of first confidence images based on each of the plurality of ultrasonic images and synthesizing the plurality of first confidence images to generate the confidence image, and the second method is a method of generating the confidence image based on a maximum value image generated by extracting the maximum value of the pixel values for each pixel of the plurality of ultrasonic images or an average value image generated by calculating the average value of the pixel values for each pixel of the plurality of ultrasonic images 。

[0012] The program of the present disclosure is a program executed by a computer, and includes a procedure for acquiring a plurality of ultrasonic images generated based on a plurality of received signals corresponding to respective reflected ultrasonic waves of transmitted ultrasonic waves transmitted in a plurality of different transmission directions, and inputting the acquired plurality of ultrasonic images to an identifier that identifies an object to be identified shown in the input ultrasonic image, and generating a confidence image showing, for each pixel of the ultrasonic image, a degree of likelihood that the object is the object to be identified based on an identification result output from the identifier, and a procedure for generating a composite image based on the plurality of ultrasonic images and the confidence image, and the procedure for generating the confidence image includes, when there are a plurality of objects to be identified, for each object to be identified, a procedure for receiving an operation for setting a method for generating the confidence image either the first method or the second method and a procedure for generating the confidence image using the set generation method for each object to be identified wherein the first method is a method of generating a plurality of first confidence images based on each of the plurality of ultrasonic images and synthesizing the plurality of first confidence images to generate the confidence image, and the second method is a method of generating the confidence image based on a maximum value image generated by extracting the maximum value of the pixel values for each pixel of the plurality of ultrasonic images or an average value image generated by calculating the average value of the pixel values for each pixel of the plurality of ultrasonic images .

Advantages of the Invention

[0013] According to the present invention, a medical image including a specific object to be identified can be generated with high image quality using a spatial compounding method

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 7C

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0015] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples. In the following description, those having the same functions and configurations are denoted by the same reference numerals, and the description thereof is omitted.

[0016] <Configuration> [Ultrasonic Diagnostic Device 100] FIG. 1 is a diagram showing an example of the configuration of the ultrasonic diagnostic device 100. As shown in FIG. 1, the ultrasonic diagnostic device 100 includes an image generation device 1 and an ultrasonic probe 2. The ultrasonic probe 2 transmits ultrasonic waves (transmission ultrasonic waves) to the subject and receives reflected waves (reflected ultrasonic waves: echoes) of the ultrasonic waves reflected in the subject. In the following description, a living body such as a human body is adopted as an example of the subject.

[0017] The image generation device 1 is connected to the ultrasonic probe 2 via a cable 3, and transmits a drive signal of an electrical signal to the ultrasonic probe 2 to cause the ultrasonic probe 2 to transmit transmission ultrasonic waves to the subject. Then, based on the reception signal, which is an electrical signal generated by the ultrasonic probe 2 in response to the reflected ultrasonic waves from the subject received by the ultrasonic probe 2, the internal state of the subject is imaged as an ultrasonic image.

[0018] The ultrasonic probe 2 has a vibrator 2a (see Fig. 2) composed of a plurality of piezoelectric elements, and the vibrators 2a are arranged in a one-dimensional array in, for example, the azimuth direction (scanning direction). The number of vibrators 2a can be arbitrarily set.

[0019] [Image generation device 1] Fig. 2 is a block diagram showing a configuration example of the image generation device 1. As shown in Fig. 2, the image generation device 1 includes, for example, an operation input unit 11, a transmission unit 12, a reception unit 13, an image acquisition unit 14, an image processing unit 15, a DSC (Digital Scan Converter) 16, a structure identification unit 17, an image synthesis unit 18, a display unit 19, and a control unit 110.

[0020] The operation input unit 11 is, for example, an operation device for performing commands for instructing the start of diagnosis and data such as information about the subject, and specifically, for example, various switches, buttons, trackballs, mice, keyboards, etc. The operation input unit 11 outputs an operation signal based on the input operation to the control unit 110.

[0021] The transmission unit 12 is a circuit that supplies a drive signal, which is an electrical signal, to the ultrasonic probe 2 via a cable 3 according to the control of the control unit 110 to generate transmitted ultrasonic waves in the ultrasonic probe 2. The transmission unit 12 includes, for example, a clock generation circuit, a delay circuit, and a pulse generation circuit (not shown). The clock generation circuit is a circuit that generates a clock signal for determining the transmission timing and transmission frequency of the drive signal. The delay circuit sets a delay time for each individual path corresponding to each vibrator 2a, and delays the transmission of the drive signal by the set delay time to perform focusing of the transmission beam (transmission beam forming) and setting of the angle of the transmission beam (steering) composed of the transmitted ultrasonic waves. The pulse generation circuit is a circuit that generates a pulse signal as a drive signal at a predetermined period.

[0022] The transmitting unit 12 configured as described above drives a continuous part (for example, several tens) of a plurality (for example, one hundred to two hundred and several tens) of vibrators 2a arranged in the ultrasonic probe 2 to generate transmitted ultrasonic waves. Then, each time the transmitting unit 12 generates transmitted ultrasonic waves, it performs scanning by shifting the vibrators 2a to be driven in the azimuth direction. Further, the transmitting unit 12 can receive a plurality of reflected signals with different angles by performing scanning while appropriately changing the angle of the transmitted beam. In the following description, the angle of the transmitted beam that is appropriately changed is described as the steering angle.

[0023] The receiving unit 13 is a circuit that receives a received signal, which is an electrical signal, from the ultrasonic probe 2 via the cable 3 according to the control of the control unit 110. The receiving unit 13 includes, for example, an amplifier, an A / D conversion circuit, and a phased addition circuit. The amplifier is a circuit for amplifying the received signal with a preset amplification factor for each individual path corresponding to each vibrator 2a. The A / D conversion circuit is a circuit for performing analog / digital conversion (A / D conversion) on the amplified received signal. The phased addition circuit is a circuit for giving a delay time to the A / D converted received signal for each individual path corresponding to each vibrator 2a to adjust the phase, and adding (phased addition) these to generate sound line data. That is, the phased addition circuit performs received beam forming on the received signal for each vibrator 2a to generate sound line data.

[0024] The image acquisition unit 14 performs envelope detection processing, logarithmic compression, etc. on the voice line data input from the reception unit 13 according to the control of the control unit 110, adjusts the dynamic range and gain, and performs luminance conversion to generate a B-mode ultrasonic image. The B-mode ultrasonic image represents the strength of the received signal by luminance. In the present embodiment, the image acquisition unit 14 may be configured to generate an A-mode image (amplitude image), an M-mode image (motion image), and an ultrasonic image by the Doppler method in addition to the B-mode image. Hereinafter, the case where the B-mode ultrasonic image is the target of processing will be described, and the B-mode ultrasonic image will be simply referred to as an ultrasonic image. However, in the image generation apparatus 1 of the present disclosure, an image other than the B-mode ultrasonic image may also be the target of processing.

[0025] Further, when the ultrasonic probe 2 scans in a plurality of directions with the angle shifted, the image acquisition unit 14 generates a plurality of ultrasonic images based on a plurality of reflected signals having different angles. A part or all of the scanning regions of the plurality of ultrasonic image data generated in this way overlap each other. These plurality of ultrasonic image data are synthesized in the image synthesis unit 18.

[0026] The image processing unit 15 performs various image processing on the plurality of ultrasonic image data generated by the image acquisition unit 14.

[0027] The DSC 16 performs scanning frequency conversion, etc. on the plurality of ultrasonic images output by the image processing unit 15 according to the control of the control unit 110, and converts them into image signals in a format that can be displayed on the display unit 19.

[0028] The structure identification unit 17 identifies a specific structure (target) inside the ultrasonic image input from the DSC 16, the control unit 110, or the image synthesis unit 18. The structure identification unit 17 is an example of the identification result acquisition unit of the present disclosure. The target is a structure of an object that should be clearly visible to the user of the ultrasonic diagnostic apparatus 100 during diagnosis using the ultrasonic diagnostic apparatus 100, and is an example of the identification object of the present disclosure.

[0029] Examples of the target include multiple types of structures such as nerves, fascia, blood vessels, and puncture needles. A puncture needle is a needle for collecting tissue by piercing the living body or injecting a chemical solution into the living body. The target may be appropriately set from among multiple types of structures that can be targets by a user's operation via the operation input unit 11, or any one of the structures may be determined as the target in advance. The number of targets identified by the structure identification unit 17 may be one or more. Specifically, only a nerve may be set as the target, or a nerve and a puncture needle may be set as the targets.

[0030] The structure identification unit 17 has an identification model, which is a learning model that has been machine-learned in advance to identify the target. The identification model is an example of the identifier and the learned identifier of the present disclosure. The identification model is constructed by supervised machine learning that trains the relationship between the feature amount (for example, luminance array) of the ultrasonic image and the information regarding the confidence level of the target using a known machine learning algorithm (so-called deep learning) such as a neural network as teacher data. When there are multiple structures that can be targets, the identification model may be generated for each target, or one identifier may identify multiple structures. The identification model included in the structure identification unit 17 is an example of the learned identifier of the present disclosure.

[0031] The confidence level is an index indicating the likelihood that a certain region in the ultrasonic image is the target, and is an example of the identification result of the present disclosure. The target and the regions around it have a high confidence level, while the regions outside the target and its surroundings (non-target regions) have a low confidence level. The confidence level is generated for each pixel of the ultrasonic image, for example.

[0032] Based on the output of the identification model, the structure identification unit 17 generates a confidence image corresponding to the input ultrasonic image. The confidence image is obtained by plotting the confidence level at each pixel of the ultrasonic image and showing the distribution of the confidence level over the entire ultrasonic image. Details of the structure identification unit 17 will be described later.

[0033] The image synthesis unit 18 synthesizes a plurality of ultrasonic images generated by the image acquisition unit 14 to generate a synthesized image. In particular, the image synthesis unit 18 generates a spatial compound image by synthesizing overlapping portions of a plurality of ultrasonic images generated based on a plurality of reflection signals received by transmission beams transmitted at a plurality of steering angles. In the following description, a plurality of ultrasonic images generated based on a plurality of reflection signals received by transmission beams transmitted at a plurality of steering angles are referred to as a plurality of ultrasonic images with different steering angles.

[0034] When synthesizing a plurality of ultrasonic images with different steering angles with each other, the image synthesis unit 18 weights each ultrasonic image based on the confidence image input from the structure identification unit 17. Thereby, the image synthesis unit 18 can output a spatial compound image in which the target is emphasized. Details of the synthesis process of the plurality of ultrasonic images by the image synthesis unit 18 will be described later.

[0035] The display unit 19 is a display device such as an LED (Light-Emitting Diode), LCD (Liquid Crystal Display), CRT (Cathode-Ray Tube) display, organic EL (Electronic Luminescence) display, inorganic EL display, and plasma display. The display unit 19 displays the synthesized image output from the image synthesis unit 18 according to the control of the control unit 110. Further, the display unit 19 may display the final confidence image (details will be described later) output from the structure identification unit 17.

[0036] The control unit 110 has, for example, a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory), reads out various processing programs such as system programs stored in the ROM and expands them in the RAM, and centrally controls the operations of each part of the ultrasonic diagnostic apparatus 100 according to the expanded programs.

[0037] The ROM is composed of a non-volatile memory such as a semiconductor, etc., and stores a system program corresponding to the ultrasonic diagnostic apparatus 100, various processing programs executable on the system program, and various data such as a gamma table. These programs are stored in the form of computer-readable program codes, and the CPU sequentially executes operations according to the program codes. The RAM forms a work area for temporarily storing various programs executed by the CPU and data related to these programs.

[0038] [Structure Identification Unit 17] Hereinafter, the structure identification unit 17 will be described in detail. FIG. 3 is a diagram showing a configuration example of the structure identification unit 17. As shown in FIG. 3, the structure identification unit 17 includes a teacher data generation unit 171, an identification model training unit 172, an identification model execution unit 173, a confidence image generation unit 174, and a storage unit 175. The storage unit 175 stores an identification model 40, reference data 50, teacher data 60, and training history data 70.

[0039] The teacher data generation unit 171 generates teacher data 60 for training the identification model 40 based on a first image for generating teacher data and the previously prepared reference data 50. The first image is extracted, for example, from an ultrasonic image for training. The teacher data 60 is a data set in which the feature amount (for example, luminance array) of a second image extracted from the first image and information regarding the confidence of the target (for example, the confidence of the target corresponding to the pixel block at the center of the second image) are associated. Note that the pixel block is each divided region when the image is divided into a plurality of regions, and may be composed of a pixel group including a plurality of pixels or may be composed of one pixel.

[0040] The identification model training unit 172 trains the identification model 40 by machine learning using the teacher data 60 generated by the teacher data generation unit 171. Specifically, when the identification model training unit 172 inputs an example (feature amount of the second image) of the teacher data 60 into the identification model 40, it modifies the identification model 40 so that the answer (information regarding the confidence level of the target) of the teacher data 60 is output.

[0041] The identification model execution unit 173 executes the trained identification model 40 to generate data for identifying a target in a diagnostic ultrasonic image. Note that the diagnostic ultrasonic image (hereinafter referred to as the diagnostic image) is an image that is generated by the image acquisition unit 14 and input to the structure identification unit 17 via the image processing unit 15 and the DSC 16 when a user performs a diagnosis using the ultrasonic diagnostic apparatus 100, rather than the training ultrasonic image.

[0042] For example, the identification model execution unit 173 extracts an identification image from the diagnostic image, and executes the identification model 40 using the identification image as an input, thereby obtaining, as an output, information regarding the confidence level of the target in the identification image.

[0043] Note that the training ultrasonic image and the identification image are, for example, at least any one of a composite image generated in the past rather than the composite image displayed on the display unit 19 at that time, a plurality of diagnostic images, an average value image generated by calculating the average of the pixel values for each pixel of the plurality of diagnostic images, or a maximum value image generated based on the maximum value of the pixel values for each pixel in a plurality of past diagnostic images. The training ultrasonic image is stored in the storage unit 175 or the like.

[0044] Based on the output from the identification model execution unit 173, the confidence image generation unit 174 generates a confidence image corresponding to the whole or a part of the diagnostic image (for example, the area surrounded by the ROI frame). At this time, the confidence image generation unit 174 may remove the noise included in the confidence image based on the temporal change of the information regarding the confidence obtained from the temporally consecutive ultrasonic images. Specifically, in the time axis direction, the noise included in the confidence image can be removed by applying a moving average filter process or a median filter process. Also, an area where the change (steepness) of the information regarding the confidence exceeds a preset threshold may be detected as a noise area, and the noise removal process may be performed only for this noise area.

[0045] The storage unit 175 is configured by, for example, a non-volatile semiconductor memory (so-called flash memory), a hard disk drive (HDD), or the like. The storage unit 175 may be a disk drive that reads and writes information by driving an optical disk such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray Disc (「Blu-ray」 is a registered trademark)), or a magneto-optical disk such as an MO (Magneto-Optical disk).

[0046] As described above, the storage unit 175 stores the identification model 40, the reference data 50, the teacher data 60, and the training history data 70. The teacher data 60 used for training the identification model 40 may be appropriately overwritten when new teacher data 60 is generated by the teacher data generation unit 171. The training history data 70 includes information such as the number of teacher data 60 used for training and the training date and time.

[0047] FIG. 4A and FIG. 4B are diagrams showing an example of the reference data 50. As shown in FIG. 4A and FIG. 4B, the reference data 50 includes first reference data 51 referred to when a target is included in the first image (ultrasonic image for generating teacher data), and second reference data 52 referred to when the target is not included in the first image.

[0048] The first reference data 51 shown in FIG. 4A is referred to when a target is drawn at the center of the first image. For example, the confidence level of the target is set according to a circular Gaussian distribution in the range of 0.0 to 1.0.

[0049] The second reference data 52 shows the confidence distribution when there is no target in the first image (when the first image is composed of non-targets). In the second reference data 52 shown in FIG. 4B, the confidence level of the target corresponding to all regions is set to 0. Note that a plurality of the first reference data 51 and the second reference data 52 may be prepared as necessary. For example, as the first reference data 51, data may be prepared that is referred to when a longitudinal section of a nerve is drawn in the first image. Further, when there are a plurality of targets, as the second reference data 52, second reference data for other targets may be prepared.

[0050] <Operation> Hereinafter, an operation example of the image generation apparatus 1 will be described. First, the generation process of the discrimination model by the structure discrimination unit 17 will be described in detail.

[0051] [Discrimination Model Generation Process] FIG. 5 is a flowchart showing an example of a discrimination model generation process for training the discrimination model 40. This process is performed, for example, when a training mode is selected by mode selection in the operation input unit 11.

[0052] In step S101, the teacher data generation unit 171 receives a designation of a training ultrasonic image according to the control of the control unit 110. The training ultrasonic image is, for example, acquired in advance for training and stored in the storage unit 175, and is read out based on a user input operation using the operation input unit 11. Also, for example, an ultrasonic image acquired at the time of past diagnosis may be applied to the training ultrasonic image.

[0053] In step S102, the teacher data generation unit 171 receives a label designation according to the control of the control unit 110. The label has a first label designated when the training target is the target and a second label designated when the training target is non-target, and one of the labels is selected based on the user's input operation via the operation input unit 11. When the first label is designated, teacher data 60 is generated using the first reference data 51, and when the second label is designated, teacher data 60 is generated using the second reference data 52.

[0054] In step S103, the teacher data generation unit 171 sets a first image (ultrasonic image for generating teacher data) in the ultrasonic image for training according to the control of the control unit 110. FIG. 6 is a diagram showing the relationship between the ultrasonic image 80 for training and the first image 81. In the example shown in FIG. 6, the first image 81 includes a first image 811 set when the target is the training target and a first image 812 set when the non-target is the training target.

[0055] The first image 81 is set, for example, based on an operation related to target designation by the user using the operation input unit 11. For example, when the user selects a region where the target (or non-target) is depicted on the ultrasonic image 80, a region of a predetermined size centered on the region is set as the first image 81. Also, for example, the size of the region to be set as the first image 81 (the white rectangular frame in FIG. 6) may be specified based on the user's operation.

[0056] Hereinafter, the case where the first label is designated in step S102 and the first image 811 including the target is set will be specifically described.

[0057] In step S104, the teacher data generation unit 171 extracts the second image 82 from the first image 811 according to the control of the control unit 110. FIGS. 7A, 7B, and 7C are diagrams for explaining the method of generating teacher data. FIG. 7A is a diagram for explaining the second image 82. The second image 82 is an image included in the first image 811 and serves as the input (example) of the teacher data. For example, the second image 82 composed of an N×N pixel block is extracted from the first image 81 composed of an M×M pixel block (N<M).

[0058] In step S105, the teacher data generation unit 171 obtains the feature amount of the second image 82 according to the control of the control unit 110. The feature amount of the second image 82 is, for example, a luminance array composed of luminance values for each pixel (or pixel block) of the second image 82.

[0059] In step S106, the teacher data generation unit 171 associates the feature amount of the second image 82 with the target confidence according to the control of the control unit 110 based on the reference data 50.

[0060] Specifically, as shown in FIGS. 7A and 7B, the teacher data generation unit 171 compares the second image 82 with the first reference data 51, specifies the target confidence V corresponding to the central pixel block of the second image 82 from the first reference data 51, and associates it with the feature amount of the second image 82. FIG. 7B is a diagram showing the state of specifying the target confidence V from the first reference data 51. At this time, the first reference data 51 is appropriately adjusted according to the size of the first image 81.

[0061] By the process of step S106, a set of teacher data 60 is generated in which the feature amount of the second image 82 is associated with the confidence level V of the target corresponding to the pixel block at the center of the second image 82. By performing the processes of steps S104 to S106 while sliding the extraction area of the second image 82 in the first image 81, a plurality of sets of teacher data 60 are generated. For example, when the reference data 50 is composed of K×K pixel blocks, the confidence level of the target can be assigned to the feature amounts of (K×K) second images 82. That is, by designating the first image 81 only once, (K×K) pieces of teacher data can be generated.

[0062] Through the above processes executed by the teacher data generation unit 171, the teacher data 60 is generated.

[0063] In step S107, the discrimination model training unit 172 trains the discrimination model 40 by machine learning using the generated teacher data 60 according to the control of the control unit 110. Specifically, as shown in FIG. 7C, when the discrimination model training unit 172 inputs an example (feature amount of the second image 82) of the teacher data 60 to the discrimination model 40, the discrimination model training unit 172 modifies the discrimination model 40 so that the answer of the teacher data 60 (confidence level V of the target corresponding to the pixel block at the center of the second image 82) is output. FIG. 7C is a diagram showing how the discrimination model 40 is modified based on the second image 82. Based on the training result, the discrimination model 40 and the training history data 70 stored in the storage unit 175 are updated.

[0064] When the discrimination model 40 is trained, it is preferable that the training history of the discrimination model 40 (for example, the number of teacher data used for training) is displayed on the display unit 19. Thereby, the user can grasp the training degree of the discrimination model 40 and can know how much more training is required in the future in order to obtain sufficient accuracy in target discrimination using the discrimination model 40.

[0065] In the above description, the structure identification unit 17 has a teacher data generation unit 171, an identification model training unit 172, an identification model execution unit 173, a confidence image generation unit 174, and a storage unit 175, and the mode in which the identification model is trained by these configurations has been described. However, the present disclosure is not limited to this.

[0066] The above-described identification model generation process does not necessarily have to be performed by the image generation device 1 of the present disclosure. For example, the structure identification unit 17 does not necessarily have to have an identification model training unit, an identification model execution unit, and a confidence image generation unit. In this case, the structure identification unit 17 acquires an identification model generated by a device external to the image generation device 1 from the external device. Note that, as for the method by which the external device generates the identification model, a method similar to the method described above may be adopted. In this case, the structure identification unit 17 stores the identification model acquired from outside the image generation device 1 in the storage unit 175, and reads out and uses the identification model stored in the storage unit 175 in the generation process of the composite image described below.

[0067] [Generation process of composite image] Using the identification model generated as described above, or the identification model generated by a device external to the image generation device 1, the image generation device 1 generates a composite image. Hereinafter, the generation process of the composite image will be described in detail.

[0068] FIG. 8 is a flowchart showing an operation example when the image generation device 1 generates a composite image.

[0069] In step S201, the control unit 110 acquires a plurality of diagnostic images with different steering angles from each other. More specifically, the control unit 110 controls the transmission unit 12 to transmit ultrasonic waves by giving a predetermined steering angle from the ultrasonic probe 2, and controls the reception unit 13 to acquire a reception signal corresponding to the reflected ultrasonic waves (ultrasonic echoes) received by the ultrasonic probe 2. Then, the control unit 110 controls the image acquisition unit 14 to generate a B-mode ultrasonic image based on the reception signal. By performing this process a plurality of times while changing the steering angle, the control unit 110 acquires a plurality of diagnostic images with different steering angles from each other.

[0070] In step S202, the structure identification unit 17 generates a final confidence image based on a plurality of diagnostic images according to the control of the control unit 110. The final confidence image is a confidence image used for calculating the weighting value in step S203. As methods for generating the final confidence image based on a plurality of diagnostic images, at least the following two methods can be adopted.

[0071] The first method is a method of generating a plurality of confidence images based on each of the plurality of diagnostic images, and then synthesizing the plurality of confidence images to generate a final confidence image. The second method is a method of generating a maximum value image or an average value image of the plurality of diagnostic images, and directly generating a final confidence image based on the maximum value image or the average value image. The maximum value image is an image obtained by extracting the maximum pixel value for each same pixel of the plurality of diagnostic images and using each pixel value as one image. The average value image is an image obtained by calculating the average value of the pixel values of the plurality of diagnostic images for each pixel and using each pixel value as one image. In the second method, since a plurality of confidence images are not generated, the final confidence image can be acquired with a small number of processing times.

[0072] In the first method, the details of the process of generating a plurality of confidence images based on each of the plurality of diagnostic images are as follows, for example. The identification model execution unit 173 of the structure identification unit 17 extracts an identification image from each diagnostic image and inputs the feature amount (for example, luminance array) of the identification image into the identification model 40. Then, the identification model execution unit 173 obtains, as an output from the identification model 40, the confidence corresponding to the pixel block at the center of the identification image. Further, the confidence image generation unit 174 generates a confidence image by obtaining the confidence for the entire diagnostic image.

[0073] FIG. 9 is a schematic diagram showing how the confidence is obtained by the structure identification unit 17. In FIG. 9, an identification image 92 set inside a diagnostic image 91 is shown, and how the confidence V(x, y) corresponding to the luminance array B(x, y) of the pixel block at the center of the identification image 92 is obtained is schematically shown. The control unit 110 controls the confidence image generation unit 174 of the structure identification unit 17 to repeat this process, and obtains the confidence for the entire diagnostic image 91, thereby generating a confidence image 93 corresponding to the entire diagnostic image 91. Note that the confidence image may be generated to correspond to a part of the diagnostic image.

[0074] Then, the confidence image generation unit 174 generates a final confidence image based on the plurality of confidence images. As a method of generating a final confidence image based on the plurality of confidence images, there is a method of extracting or calculating the maximum value, average value, or minimum value for each pixel of the plurality of confidence images and generating a final confidence image having the extracted value as the pixel value. Note that which of the maximum value, average value, or minimum value for each pixel of the plurality of confidence images is used can be selected by a user operation using the operation input unit 11.

[0075] When generating the final confidence image in the first method, the characteristics of the final confidence image differ depending on whether the maximum value, average value, or minimum value of each pixel of the plurality of confidence images is used. When the maximum value is used, the sensitivity, that is, the degree to which a target area can be detected, is relatively high, and the false positive rate, that is, the degree to which an area that is not a target is detected as a target, also becomes relatively high. When the minimum value is used, the sensitivity and false positive rate are relatively low, and when the average value is used, the sensitivity and false positive rate are lower than when the maximum value is used and higher than when the minimum value is used.

[0076] In the second method, the confidence image generation unit 174 controls the identification model execution unit 173 of the structure identification unit 17 to extract an identification image from the maximum value image or the average value image generated based on a plurality of diagnostic images, inputs the feature amount (for example, luminance array) of the identification image into the identification model 40, and obtains the confidence as the output. Further, the confidence image generation unit 174 generates the final confidence image by obtaining the confidence for the entire maximum value image or average value image.

[0077] Note that in the second method, which of the maximum value image and the average value image is used can be selected, for example, by a user operation using the operation input unit 11.

[0078] The characteristics of the final confidence image differ depending on which of the maximum value image and the average value image is used in the second method. When the maximum value image is used, the sensitivity is relatively high and the false positive rate is relatively high. When the average value is used, the sensitivity and false positive rate are lower than when the maximum value image is used.

[0079] Note that when generating the final confidence image in step S202, which of the first method and the second method described above is adopted can be selected, for example, by a user operation using the operation input unit 11.

[0080] In step S202, when adopting the first method, it is preferable to select whether to use the maximum value, average value, or minimum value for each pixel of the plurality of confidence images, and when adopting the second method, it is preferable to select whether to use the maximum value image or the average value image according to the purpose of diagnosis.

[0081] A specific example will be given for explanation. For example, when the user pierces the subject with a puncture needle while looking at the composite image displayed by the image generation device 1 and injects an anesthetic solution near the nerve region, it is preferable that the false positive rate in the region other than the nerve in the composite image is low. In this case, it is preferable to use the average value in the first method, and it is preferable to use the average value image in the second method.

[0082] Also, for example, when the user pierces the subject with a puncture needle while looking at the composite image displayed by the image generation device 1 and injects a drug solution into the blood vessel region, it is preferable that the sensitivity is increased so that the nerve region that should not be punctured can be reliably detected. In this case, it is preferable to use the maximum value in the first method, and it is preferable to use the maximum value image in the second method.

[0083] When there are multiple targets, when the final confidence image is generated in step S202, a different discrimination model is used for each target to generate the final confidence image for each target. Here, it may be possible to set whether to use the first method or the second method when generating the final confidence image for each target. Also, when the first method is used, it may be possible to set whether to use the maximum value, average value, or minimum value for each pixel of the plurality of confidence images for each target. Furthermore, when the second method is used, it may be possible to set whether to use the maximum value image or the average value image for each target. Thereby, the way of emphasizing in the composite image can be made different for each target, and a composite image according to the purpose of diagnosis can be obtained.

[0084] In the description of step S202 above, it was assumed that the structure identification unit 17 generates a final confidence image based on a plurality of diagnostic images according to the control of the control unit 110. The plurality of diagnostic images used by the structure identification unit 17 to generate the final confidence image may be, for example, images before the signal format conversion is performed by the DSC16, or images after the conversion is performed by the DSC16. When using the image before the format conversion by the DSC16, the structure identification unit 17 may output the generated confidence image to the DSC16, and use the image after the format conversion by the DSC16 to generate the final confidence image.

[0085] In step S203, the image synthesis unit 18 calculates a weighting value for synthesizing a plurality of diagnostic images based on the final confidence image according to the control of the control unit 110. The weighting value is a value from 0 to 1 and is set for each pixel of the diagnostic image. The weighting value is set to the confidence level at each pixel of the final confidence image corresponding to each pixel of the diagnostic image. In the present embodiment, it is assumed that the value of each pixel of the final confidence image is the weighting value at that pixel. However, for example, the weighting value may be calculated based on the value of each pixel of the final confidence image using a predetermined calculation method.

[0086] In step S204, the image synthesis unit 18 generates a synthesized image based on a plurality of diagnostic images with different steering angles from each other and the weighting value according to the control of the control unit 110. The synthesized image is obtained by α-blending a maximum value image obtained by extracting the maximum value for each pixel of the plurality of diagnostic images as each pixel value and an average value image obtained by calculating the average value for each pixel of the plurality of diagnostic images as each pixel value using the weighting value.

[0087] The generation of the synthesized image will be described in detail with reference to FIG. 10. FIG. 10 is a diagram schematically showing a state of synthesizing a plurality of diagnostic images with different steering angles from each other. In FIG. 10, five diagnostic images A, B, C, D, and E with different steering angles from each other are shown.

[0088] In this case, the image synthesis unit 18 calculates the maximum value Max(A, B, C, D, E) among the pixel values of the diagnostic images A, B, C, D, and E for the same pixel, and the average value Mean(A, B, C, D, E) of the pixel values of the diagnostic images A, B, C, D, and E for the same pixel, respectively. Then, the image synthesis unit 18 performs α-blending processing using the weighting value α corresponding to the pixel value of the pixel in the final confidence image. Therefore, each pixel value Data of the synthesized image obtained by the image synthesis unit 18 is represented by the following formula (1).

[0089]

Number

[0090] As another synthesis method, the image synthesis unit 18 may calculate each pixel value Data of the synthesized image by weighting each of the pixel values of the diagnostic images A, B, C, D, and E for the same pixel and then adding them all together as shown in the following formula (2).

[0091]

Number

[0092] When using formula (2), it is not necessary to generate the final confidence image. Based on the plurality of confidence images generated based on each of the plurality of diagnostic images, the respective weighting values α1, α2, α3, α4, and α5 may be determined.

[0093] By performing such processing for all pixels, a synthesized image is generated. By generating a synthesized image by the spatial compounding method in this way, noise, particularly speckle noise, can be reduced, and a structure having anisotropy with respect to transmitted ultrasonic waves can be accurately detected. Furthermore, by using the weighting value based on the confidence image for generating the synthesized image, the structure set as the target can be accurately detected, and the situation where noise other than the target or the structure is erroneously emphasized can be avoided.

[0094] Note that the method for calculating the weighting value is not limited to the above-described calculation method (the method using the final confidence image), and it may be calculated by a different method. When the weighting value is calculated by a different method, for example, the larger value may be adopted among the weighting value calculated by the above-described calculation method and the weighting value calculated by a different method.

[0095] Note that the generation process of the composite image described above is performed for each frame, and the composite image is updated and displayed for each frame on the display unit 19, so that the user can perform a diagnosis based on the moving image of the ultrasonic image that smoothly shows the movement of the structure.

[0096] <Effect> As described above, the configuration and operation of the image generation device 1 of the present disclosure have been described. In the image generation device 1 of the present disclosure, since the composite image is generated using the weighting value based on the confidence image, pixels with high confidence in the target can be emphasized, and the visibility of the target when the composite image is displayed on the display unit 19 is improved. Furthermore, it is possible to avoid a situation where noise other than the target or the structure is erroneously emphasized.

[0097] In the image generation device 1 of the present disclosure, the confidence image used in generating the composite image is generated based on an image different from the composite image currently displayed on the display unit 19, such as a past composite image, a past diagnostic image, or a maximum value image or an average value image generated based on the past diagnostic image. Therefore, compared with the case where the confidence image is generated using the composite image currently displayed on the display unit 19 and a plurality of current diagnostic images are synthesized based on the generated confidence image, a new composite image can be generated in a short time. As a result, even when it takes time to detect the target, the user can quickly generate a composite image in which the target is easy to visually recognize.

Industrial Applicability

[0098] The present invention is suitable for an image generation device that synthesizes a plurality of ultrasonic images.

Explanation of Signs

[0099] 100 Ultrasonic diagnostic device 1 Image generation device 2 Ultrasonic probes 2a Vibrators 3 Cables 11 Operation input unit 12 Transmission unit 13 Reception unit 14 Image acquisition unit 15 Image processing unit 16 DSC 17 Structure identification unit 171 Teacher data generation unit 172 Identification model training unit 173 Identification model execution unit 174 Confidence image generation unit 175 Memory unit 18 Image synthesis unit 19 Display unit 110 Control unit

Claims

1. An image acquisition unit that acquires a plurality of ultrasonic images generated based on a plurality of received signals corresponding to reflected ultrasonic waves of transmitted ultrasonic waves transmitted in a plurality of different transmission directions; An identification result acquisition unit that generates a confidence image indicating the likelihood that each pixel of the ultrasonic image is the object to be identified, based on the identification result output from the identifier that identifies the object to be identified in the input ultrasonic image, by inputting the plurality of acquired ultrasonic images to the identifier; An image synthesis unit that generates a synthesized image based on the plurality of ultrasonic images and the confidence image; An operation input unit that receives an operation; Comprising; When there are a plurality of objects to be identified, the operation input unit receives an operation for setting, for each object to be identified, either a first method or a second method as a method for generating the confidence image; The identification result acquisition unit generates the confidence image for each object to be identified using the set generation method; The first method is a method of generating a plurality of first confidence images based on each of the plurality of ultrasonic images and synthesizing the plurality of first confidence images to generate the confidence image. The second method is a method of generating the confidence image based on a maximum value image generated by extracting the maximum value of the pixel values for each pixel of the plurality of ultrasonic images, or an average value image generated by calculating the average value of the pixel values for each pixel of the plurality of ultrasonic images; An image generation device.

2. In the first method, the identification result acquisition unit generates the confidence image using any one of the maximum value, average value, or minimum value for each pixel of the plurality of first confidence images; The image generation device according to Claim 1.

3. The identifier is a trained identifier trained by machine learning; The identification result acquisition unit inputs the ultrasonic image to the trained identifier; The image generation device according to Claim 1.

4. The trained identifier is a neural network; The image generation device according to Claim 3.

5. Further comprising a display unit that displays the synthesized image; The identification result acquisition unit inputs an image different from the synthesized image displayed on the display unit to the identifier; The image generation device according to any one of Claims 1 to 4.

6. The identification result acquisition unit inputs, to the identifier, a composite image generated in the past from the composite image currently displayed on the display unit, the plurality of ultrasonic images generated in the past, and an image generated by calculating the average or maximum value of the pixel values for each pixel of the plurality of ultrasonic images generated in the past. The image generation apparatus according to claim 5. **Claim 7** The image composition unit calculates a weighting value based on the confidence image, and generates the composite image based on the plurality of ultrasonic images and the weighting value. The image generation apparatus according to any one of claims 1 to 6. **Claim 8** The image composition unit calculates the weighting value for each pixel based on the confidence image. The image generation apparatus according to claim 7. **Claim 9** The image composition unit calculates the weighting value based on a final confidence image generated by calculating the average or maximum value of the pixel values for each pixel of the plurality of confidence images. The image generation apparatus according to claim 7 or 8. **Claim 10** The object to be identified is a nerve, fascia, blood vessel, or puncture needle. The image generation apparatus according to any one of claims 1 to 9. **Claim 11** The image composition unit changes the method for calculating the weighting value based on the type of the object to be identified. The image generation apparatus according to any one of claims 7 to 9. **Claim 12** An image generation method executed by a computer included in an image generation apparatus, wherein the computer acquires a plurality of ultrasonic images generated based on a plurality of reception signals corresponding to reflected ultrasonic waves of transmitted ultrasonic waves transmitted in a plurality of different transmission directions, generates a confidence image indicating, for each pixel of the ultrasonic image, the likelihood that the object is the object to be identified, based on the identification result output from the identifier by inputting the plurality of acquired ultrasonic images to an identifier that identifies an object to be identified shown in the input ultrasonic image, generates a composite image based on the plurality of ultrasonic images and the confidence image. An image generation method, wherein the generation of the confidence image when there are a plurality of objects to be identified, includes a step of receiving an operation of setting, for each object to be identified, either a first method or a second method as a method for generating the confidence image, and a step of generating the confidence image for each object to be identified using the set generation method. is included. The first method is a method of generating a plurality of first confidence images based on each of the plurality of ultrasonic images, and synthesizing the plurality of first confidence images to generate the confidence image. The second method is a method of generating the confidence image based on a maximum value image generated by extracting the maximum value of the pixel values for each pixel of the plurality of ultrasonic images, or an average value image generated by calculating the average value of the pixel values for each pixel of the plurality of ultrasonic images. Image generation method.

13. A program executed by a computer, a procedure of acquiring a plurality of ultrasonic images generated based on a plurality of received signals corresponding to each of the reflected ultrasonic waves of the transmitted ultrasonic waves transmitted in a plurality of different transmission directions; a procedure of generating a confidence image indicating the likelihood that each pixel of the ultrasonic image is the object to be identified based on the identification result output from the identifier by inputting the plurality of acquired ultrasonic images to the identifier that identifies the object to be identified shown in the input ultrasonic image; a procedure of generating a composite image based on the plurality of ultrasonic images and the confidence image; a program for causing the computer to execute, the procedure for generating the confidence image is when there are a plurality of objects to be identified, for each object to be identified, a procedure of receiving an operation of setting either the first method or the second method as a method for generating the confidence image; for each object to be identified, a procedure of generating the confidence image using the set generation method; including the first method is a method of generating a plurality of first confidence images based on each of the plurality of ultrasonic images, and synthesizing the plurality of first confidence images to generate the confidence image. The second method is a method of generating the confidence image based on a maximum value image generated by extracting the maximum value of the pixel values for each pixel of the plurality of ultrasonic images, or an average value image generated by calculating the average value of the pixel values for each pixel of the plurality of ultrasonic images. Program.

Images