Information processing device, ultrasound diagnostic device, and method

Abstract

To reduce a user's burden for setting parameters of a rendering image.SOLUTION: An information processing device includes an acquisition part, a region estimation part, a parameter setting part, and a rendering part. The acquisition part acquires a two-dimensional image related to three-dimensional data including an observation object. The region estimation part estimates a region of the observation object on the basis of the two-dimensional image. The parameter setting part sets a rendering parameter on the basis of the estimated region and the three-dimensional data. The rendering part generates a rendering image of the three-dimensional data using the rendering parameter.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The embodiments disclosed in the present specification and drawings relate to an information processing device, an ultrasound diagnostic device, and a method. [Background technology]

[0002] A technology is known in which ultrasound diagnostic equipment acquires ultrasound echo reflection intensity as three-dimensional data and generates a rendered image. Recently, rendering images have also been generated based on global illumination, a rendering method that takes light sources and shadows into account in calculations. When rendering using this method, in the case of computed tomography (CT) or magnetic resonance imaging (MRI), the data acquired by each equipment is associated with clinical tissue locations, so colors such as ambient light color, reflection color, and attenuation color are assigned to the data before rendering.

[0003] On the other hand, in ultrasound diagnostic devices, echo reflection intensity does not correspond to clinical tissue locations, so parameters cannot be automatically set for echo reflection intensity as with CT or MRI data. In particular, in modes that simultaneously display the surface structure (shell structure) and internal structure of a living body, it is necessary to set both a representative brightness value (representative brightness value) that contributes to the formation of the contour, as well as high and low brightness thresholds, which are brightness values ​​around the representative brightness value. These parameters must be manually adjusted so that the contour of the object to be observed is displayed, which is a burden for the user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-181168 Summary of the Invention [Problem to be solved by the invention]

[0005] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to reduce the burden on users in setting parameters for rendering images. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of the configurations shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0006] An information processing device according to an embodiment includes an acquisition unit, a part estimation unit, a parameter setting unit, and a rendering unit. The acquisition unit acquires a two-dimensional image related to three-dimensional data including an observation object. The part estimation unit estimates a part of the observation object based on the two-dimensional image. The parameter setting unit sets rendering parameters based on the estimated part and the three-dimensional data. The rendering unit generates a rendering image of the three-dimensional data using the rendering parameters. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an ultrasonic diagnostic apparatus according to the first embodiment. [Figure 2] FIG. 2 is an explanatory diagram illustrating a typical method for photographing a fetus. [Figure 3] FIG. 3 is a flowchart showing an example of the operation of the processing circuit of the ultrasonic diagnostic apparatus according to the first embodiment. [Figure 4] FIG. 4 shows an example of a two-dimensional image used in part estimation in the first embodiment. [Figure 5] FIG. 5 is a block diagram illustrating an outline of the part estimation process in the first embodiment. [Figure 6] FIG. 6 is an example of a part estimation image in the first embodiment. [Figure 7]FIG. 7 shows a first specific example of a transfer function that associates brightness values ​​with opacity in the first embodiment. [Figure 8] FIG. 8 shows a second specific example of a transfer function that associates brightness values ​​with opacity in the first embodiment. [Figure 9] FIG. 9 is a block diagram illustrating an example of the configuration of an information processing device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of an ultrasound diagnostic apparatus will be described in detail with reference to the drawings.

[0009] (First embodiment) Fig. 1 is a block diagram showing an example of the configuration of an ultrasound diagnostic apparatus according to the first embodiment. The ultrasound diagnostic apparatus 1 in Fig. 1 includes an apparatus main body 100 and an ultrasound probe 101. The apparatus main body 100 is connected to an input device 102 and an output device 103. The apparatus main body 100 is also connected to an external device 104 via a network NW. The external device 104 is, for example, a server equipped with PACS (Picture Archiving and Communication Systems).

[0010] The ultrasonic probe 101 performs an ultrasonic scan of a scan region in a living body P, which is a subject, under the control of, for example, the device main body 100. The ultrasonic probe 101 has, for example, a plurality of piezoelectric transducers, a matching layer provided between the plurality of piezoelectric transducers and a case, and a backing material that prevents ultrasonic waves from propagating backward in the radiation direction from the plurality of piezoelectric transducers. The ultrasonic probe 101 is, for example, a two-dimensional array probe in which a plurality of ultrasonic transducers are arranged along a first element array direction (elevation direction) and a second element array direction (azimuth direction). The ultrasonic probe 101 is detachably connected to the device main body 100. The ultrasonic probe 101 may be provided with buttons that are pressed for offset processing, operations to freeze an ultrasound image (freeze operation), and the like.

[0011] The multiple piezoelectric transducers generate ultrasonic waves based on a drive signal supplied from an ultrasonic transmission circuit 110 (described later) included in the device main body 100. This causes ultrasonic waves to be transmitted from the ultrasonic probe 101 to the living body P. When ultrasonic waves are transmitted from the ultrasonic probe 101 to the living body P, the transmitted ultrasonic waves are reflected successively by discontinuous surfaces of acoustic impedance in the body tissue of the living body P and received as reflected wave signals by the multiple piezoelectric transducers. The amplitude of the received reflected wave signals depends on the difference in acoustic impedance at the discontinuous surfaces from which the ultrasonic waves are reflected. Furthermore, when a transmitted ultrasonic pulse is reflected by a moving blood flow or the surface of a heart wall, etc., the reflected wave signal undergoes a frequency shift due to the Doppler effect, depending on the velocity component of the moving object in the direction of ultrasonic transmission. The ultrasonic probe 101 receives the reflected wave signal from the living body P and converts it into an electrical signal.

[0012] 1 illustrates an example of the connection relationship between one ultrasonic probe 101 and the device main body 100. However, it is possible to connect multiple ultrasonic probes to the device main body 100. Which of the multiple connected ultrasonic probes is to be used for ultrasonic scanning can be arbitrarily selected, for example, by using a software button on a touch panel, which will be described later.

[0013] The device main body 100 is a device that generates an ultrasound image based on a reflected wave signal received by an ultrasound probe 101. The device main body 100 has an ultrasound transmission circuit 110, an ultrasound reception circuit 120, an internal storage circuit 130, an image memory 140, an input interface 150, an output interface 160, a communication interface 170, and a processing circuit 180.

[0014] The ultrasonic transmission circuit 110 is a processor that supplies a drive signal to the ultrasonic probe 101. The ultrasonic transmission circuit 110 is realized by, for example, a trigger generation circuit, a delay circuit, and a pulser circuit. The trigger generation circuit repeatedly generates rate pulses for forming transmitted ultrasonic waves at a predetermined rate frequency. The delay circuit provides each rate pulse generated by the trigger generation circuit with a delay time for each of the multiple piezoelectric transducers required to focus the ultrasonic waves generated from the ultrasonic probe into a beam and determine the transmission directivity. The pulser circuit applies drive signals (drive pulses) to the multiple ultrasonic transducers provided in the ultrasonic probe 101 at a timing based on the rate pulse. By changing the delay time provided to each rate pulse using the delay circuit, the transmission direction from the surfaces of the multiple piezoelectric transducers can be freely adjusted.

[0015] Furthermore, the ultrasound transmission circuit 110 can arbitrarily change the output intensity of the ultrasound waves using the drive signal. In the ultrasound diagnostic device, increasing the output intensity can reduce the influence of ultrasound attenuation within the living body P. By reducing the influence of ultrasound attenuation, the ultrasound diagnostic device can acquire a reflected wave signal with a high S / N ratio during reception.

[0016] Generally, when ultrasound propagates through a living body P, the strength of the ultrasound vibration (also called acoustic power), which corresponds to the output intensity, attenuates. The attenuation of acoustic power occurs due to absorption, scattering, reflection, and the like. The degree of reduction in acoustic power depends on the frequency of the ultrasound and the distance in the direction of ultrasound radiation. For example, the degree of attenuation increases as the frequency of the ultrasound increases. Furthermore, the longer the distance in the direction of ultrasound radiation, the greater the degree of attenuation.

[0017] The ultrasonic receiving circuit 120 is a processor that performs various processes on the reflected wave signals received by the ultrasonic probe 101 to generate received signals. The ultrasonic receiving circuit 120 generates received signals based on the reflected wave signals of ultrasonic waves acquired by the ultrasonic probe 101. Specifically, the ultrasonic receiving circuit 120 is realized by, for example, a preamplifier, an A / D converter, a demodulator, and a beamformer. The preamplifier amplifies the reflected wave signals received by the ultrasonic probe 101 for each channel and performs gain correction processing. The A / D converter converts the gain-corrected reflected wave signals into digital signals. The demodulator demodulates the digital signals. The beamformer, for example, applies a delay time required to determine the reception directivity to the demodulated digital signals and adds up multiple digital signals with the applied delay time. The addition processing of the beamformer generates received signals in which the reflection components from the direction corresponding to the reception directivity are emphasized. Note that hereinafter, the "ultrasonic reflected wave signals" and "received signals" are collectively referred to as "echo signals." Therefore, the "strength of the received signal" may be rephrased as "the reflection strength of the echo signal (echo reflection strength)."

[0018] The internal storage circuitry 130 includes a processor-readable storage medium, such as a magnetic storage medium, an optical storage medium, or a semiconductor memory. The internal storage circuitry 130 stores a program for transmitting and receiving ultrasound waves, a program for a region estimation process (described later), a program for a rendering image generation process, and various data. The programs and various data may be pre-stored in the internal storage circuitry 130. Alternatively, the programs and various data may be stored in a non-transitory storage medium, distributed, read from the non-transitory storage medium, and installed in the internal storage circuitry 130. The internal storage circuitry 130 also stores B-mode image data, contrast image data, image data related to blood flow images, and three-dimensional data generated by the processing circuitry 180 in accordance with operations input via the input interface 150. The internal storage circuitry 130 can also transfer the stored image data and three-dimensional data to an external device 104 or the like via the communication interface 170.

[0019] The internal storage circuit 130 may be a drive device that reads and writes various information from and to a portable storage medium such as a CD drive, a DVD drive, or a flash memory. The internal storage circuit 130 can also write stored data to the portable storage medium and store the data in the external device 104 via the portable storage medium.

[0020] The image memory 140 has a processor-readable storage medium, such as a magnetic storage medium, an optical storage medium, or a semiconductor memory. The image memory 140 stores image data corresponding to a plurality of frames immediately before a freeze operation input via the input interface 150. The image data stored in the image memory 140 is, for example, continuously displayed (cine display). Note that the image memory 140 is not limited to storing image data, and may also store three-dimensional data.

[0021] The internal storage circuit 130 and the image memory 140 do not necessarily have to be realized by independent storage devices. The internal storage circuit 130 and the image memory 140 may be realized by a single storage device. Furthermore, the internal storage circuit 130 and the image memory 140 may each be realized by multiple storage devices.

[0022] The input interface 150 accepts various instructions from an operator via the input device 102. Examples of the input device 102 include a mouse, a keyboard, a panel switch, a slider switch, a trackball, a rotary encoder, an operation panel, and a touch panel. The input interface 150 is connected to the processing circuitry 180 via a bus, for example, and converts operation instructions input by the operator into electrical signals and outputs the electrical signals to the processing circuitry 180. Note that the input interface 150 is not limited to those connected to physical operation components such as a mouse and a keyboard. For example, an example of an input interface also includes a circuit that receives electrical signals corresponding to operation instructions input from an external input device provided separately from the ultrasound diagnostic apparatus 1 and outputs the electrical signals to the processing circuitry 180.

[0023] The output interface 160 is an interface for outputting, for example, an electrical signal from the processing circuit 180 to the output device 103. The output device 103 is any display such as a liquid crystal display, an organic EL display, an LED display, a plasma display, or a CRT display. The output device 103 may be a touch panel display that also serves as the input device 102. In addition to the display, the output device 103 may further include a speaker that outputs audio. The output interface 160 is connected to the processing circuit 180 via, for example, a bus, and outputs the electrical signal from the processing circuit 180 to the output device 103.

[0024] The communication interface 170 is connected to the external device 104 via, for example, a network NW, and performs data communication with the external device 104 .

[0025] The processing circuitry 180 is, for example, a processor that functions as the core of the ultrasound diagnostic apparatus 1. The processing circuitry 180 executes a program stored in the internal storage circuitry 130 to realize functions corresponding to the program. The processing circuitry 180 has, for example, a B-mode processing function 181, a Doppler processing function 182, an image generation function 183, a three-dimensional data generation function 184 that functions as a three-dimensional data generation unit, an acquisition function 185A that functions as an acquisition unit, a region estimation function 185B that functions as a region estimation unit, a parameter setting function 185C that functions as a parameter setting unit, a rendering function 185D that functions as a rendering unit, a display control function 186 that functions as a display control unit, and a system control function 187. Note that the Doppler processing function 182 has little relevance to this embodiment and may therefore be omitted from the functions of the processing circuitry 180.

[0026] The B-mode processing function 181 is a function that generates B-mode data based on the received signal (echo signal) received from the ultrasonic receiving circuit 120. Using the B-mode processing function 181, the processing circuit 180 performs, for example, envelope detection processing and logarithmic compression processing on the received signal received from the ultrasonic receiving circuit 120, and generates data (B-mode data) in which the signal strength (echo reflection strength) of the received signal is expressed as a brightness value (luminance value). The generated B-mode data is stored in a RAW data memory (not shown) as B-mode RAW data on a two-dimensional ultrasonic scan line (raster).

[0027] Furthermore, the processing circuitry 180 can perform harmonic imaging using the B-mode processing function 181. Harmonic imaging is an imaging method that utilizes not only fundamental wave components contained in reflected ultrasonic wave signals but also harmonic components (harmonic components). Harmonic imaging includes, for example, tissue harmonic imaging (THI), which does not use a contrast agent, and contrast harmonic imaging (CHI), which uses a contrast agent.

[0028] THI can extract harmonic components using an imaging method called the Amplitude Modulation (AM) method, the Phase Modulation (PM) method, or the AMPM method, which is a combination of the AM and PM methods.

[0029] In the AM, PM, and AMPM methods, ultrasonic waves with different amplitudes and phases are transmitted multiple times along the same scan line. This allows the ultrasonic receiving circuit 120 to generate multiple pieces of reflected wave data for each scan line and output the generated reflected wave data. The processing circuit 180 extracts harmonic components by performing addition and subtraction processing of the multiple pieces of reflected wave data for each scan line using the B-mode processing function 181 in accordance with the modulation method. The processing circuit 180 then performs envelope detection processing and the like on the reflected wave data of the harmonic components to generate B-mode data.

[0030] Furthermore, in CHI, for example, harmonic components are extracted using a frequency filter. The processing circuitry 180 can separate reflected wave data (harmonic components) whose reflection source is the contrast agent from reflected wave data (fundamental wave components) whose reflection source is tissue within the living body P using a B-mode processing function 181. As a result, the processing circuitry 180 can select harmonic components from the contrast agent using a filter and generate B-mode data for generating contrast image data.

[0031] The B-mode data for generating contrast image data is data that represents the echo reflection intensity from the contrast agent as a reflection source, expressed as a brightness value. The processing circuitry 180 can also extract the fundamental wave component from the reflected wave data of the living body P to generate B-mode data for generating tissue image data.

[0032] The Doppler processing function 182 is a function that generates data (Doppler information) that extracts motion information based on the Doppler effect of a moving object within a ROI (Region Of Interest) set in a scan area by performing frequency analysis on the received signal received from the ultrasound receiving circuit 120. The generated Doppler information is stored in a RAW data memory (not shown) as Doppler RAW data (also referred to as Doppler data) on a two-dimensional ultrasound scan line.

[0033] Specifically, the processing circuitry 180 uses the Doppler processing function 182 to estimate, for example, the average velocity, average variance, average power, etc., as motion information of a moving object at each of a plurality of sample points, and generates Doppler data indicating the estimated motion information. The moving object is, for example, blood flow, tissue such as a heart wall, or a contrast agent. The processing circuitry 180 according to this embodiment uses the Doppler processing function 182 to estimate, for each of a plurality of sample points, the average velocity of blood flow, the variance of blood flow velocity, the power value of blood flow signals, etc., as motion information of blood flow (blood flow information), and generates Doppler data indicating the estimated blood flow information.

[0034] The image generation function 183 is a function that generates B-mode image data based on data generated by the B-mode processing function 181. For example, the processing circuitry 180 converts (scan converts) a scan line signal sequence of an ultrasound scan into a scan line signal sequence of a video format typified by a television or the like using the image generation function 183, and generates image data for display (display image data). Specifically, the processing circuitry 180 performs RAW-to-pixel conversion on the B-mode RAW data stored in the RAW data memory, for example, by performing coordinate conversion according to the ultrasound scanning form of the ultrasound probe 101, thereby generating two-dimensional B-mode image data (also referred to as ultrasound image data) composed of pixels. In other words, the processing circuitry 180 generates a plurality of ultrasound images (medical images) corresponding to a plurality of consecutive frames by transmitting and receiving ultrasound using the image generation function 183.

[0035] Furthermore, the processing circuitry 180 generates Doppler image data in which blood flow information is visualized, for example, by performing RAW-to-pixel conversion on the Doppler RAW data stored in the RAW data memory. The Doppler image data is mean velocity image data, variance image data, power image data, or image data combining these. The processing circuitry 180 generates, as the Doppler image data, color Doppler image data in which blood flow information is displayed in color, and Doppler image data in which one piece of blood flow information is displayed in a grayscale waveform.

[0036] The three-dimensional data generation function 184 is a function that generates three-dimensional B-mode data (three-dimensional data) based on the reception signal received from the ultrasound reception circuit 120. The processing circuit 180 generates three-dimensional data using the three-dimensional data generation function 184 by assigning brightness values ​​to voxels arranged in three-dimensional space using the B-mode data generated by the B-mode processing function 181. This three-dimensional data may be called volume data. Note that, since the brightness values ​​correspond to the echo reflection intensity, it may be interpreted that the echo reflection intensity is assigned to the voxels of the volume data. Therefore, hereinafter, the "brightness value of the volume data" may be used in a similar sense to the "echo reflection intensity."

[0037] Acquisition function 185A is a function that acquires data related to the body part estimation process and data related to the rendering image generation process, which will be described later. Data related to the body part estimation process includes, for example, a two-dimensional ultrasound image or a simple rendering image that is rendered by applying X-ray projection (full additive projection) to three-dimensional data in a predetermined ray direction. Hereinafter, a two-dimensional ultrasound image will be simply referred to as an ultrasound image. Furthermore, when there is no need to distinguish between an ultrasound image and a simple rendering image, they will be referred to as a two-dimensional image (2D image). Data related to the rendering image generation process includes, for example, three-dimensional data. Specifically, processing circuit 180 acquires two-dimensional images and three-dimensional data using acquisition function 185A.

[0038] The part estimation function 185B is a function that estimates the part of the observation object based on a two-dimensional image. The observation object in this embodiment is a fetus. Parts of the fetus are, for example, the brain, head, spine, and whole body. Specifically, the part estimation function 185B causes the processing circuit 180 to estimate the part of the observation object contained in the two-dimensional image by applying a trained model to the two-dimensional image, and output the estimation result. The estimation result, for example, associates part information with a position on the two-dimensional image. Note that if the two-dimensional image is an ultrasound image, the part of the observation object may be estimated using multiple ultrasound images taken at different cross sections.

[0039] The trained model is, for example, a machine learning model that has been trained based on a two-dimensional image containing a region of an observation target that has been prepared in advance. Note that different trained models may be prepared for when region estimation is performed using a simplified rendering image and when region estimation is performed using an ultrasound image.

[0040] The machine learning model according to this embodiment is typically a deep neural network (DNN), which is a multi-layer network model that mimics the neural circuits of a biological brain. A DNN includes a composite function with parameters that is defined by a combination of multiple adjustable functions and parameters.

[0041] Note that, using part estimation function 185B, processing circuitry 180 may estimate the part of the observation target included in the two-dimensional image using image processing that searches for a pattern structure in the two-dimensional image, and output the estimation result.

[0042] The parameter setting function 185C is a function for setting parameters based on the estimation results. The set parameter is, for example, a transfer function related to opacity used in a mode in which the surface structure and internal structure of a living body are simultaneously displayed (hereinafter referred to as internal transparency display mode). This transfer function is calculated based on a representative brightness value of the part to be displayed and low and high thresholds, which are brightness values ​​before and after the representative brightness value by a predetermined range.

[0043] Specifically, the parameter setting function 185C causes the processing circuitry 180 to acquire a representative luminance value of the estimated region in the three-dimensional data, determine thresholds (e.g., low and high thresholds) around the representative luminance value, and set parameters based on the representative luminance value and the thresholds. The thresholds around the representative luminance value may be determined, for example, based on a table that associates biological regions with threshold widths. This table is also stored, for example, in the internal storage circuitry 130.

[0044] If there are multiple estimated parts, processing circuitry 180 may obtain a representative brightness value for each of the multiple estimated parts, and may set parameters based on the multiple representative brightness values ​​and their respective thresholds.

[0045] The rendering function 185D is a function for generating a rendering image. Rendering images include, for example, volume rendering images and global illumination images. In this embodiment, a rendering image that does not take a light source into consideration is defined as a volume rendering image, and a rendering image that takes a light source into consideration is defined as a global illumination image. Note that when the internal transparency display mode is executed, a global illumination image is displayed.

[0046] A volume rendering image is obtained by performing volume rendering on volume data. In volume rendering, the brightness and color of each voxel are set according to the brightness value assigned to the voxel of the volume data (the brightness value of the volume data). Then, volume rendering displays a projection image of the voxels projected from an arbitrary viewpoint.

[0047] On the other hand, a global illumination image is rendered using a photon map, for example, using a ray tracing method. In this embodiment, the global illumination image is generated as the final rendered image that is actually displayed.

[0048] The rendering function 185D causes the processing circuit 180 to generate a rendering image based on the set parameters. Specifically, the processing circuit 180 generates a rendering image of the three-dimensional data using the rendering parameters. The rendering parameters include, for example, a transfer function related to opacity (also called an opacity curve), base color parameters, a light characteristic map, and information on the viewpoint position. The processing by the rendering function 185D may be referred to as global illumination rendering processing.

[0049] Note that the processing circuit 180 may use the rendering function 185D to generate a simple rendering image to be used in the body part estimation process. The amount of calculation required for the process of generating a simple rendering image is less than the amount of calculation required for the process of generating a global illumination image. Furthermore, the process of generating a simple rendering image is not limited to full additive projection, and any rendering method that projects the entire data may be used.

[0050] The display control function 186 is a function that displays images based on various ultrasound image data generated by the image generation function 183 on a display serving as the output device 103. Specifically, for example, the display control function 186 causes the processing circuitry 180 to control the display of images based on image data including B-mode image data, Doppler image data, or both generated by the image generation function 183 on the display.

[0051] More specifically, the processing circuitry 180 uses the display control function 186 to convert (scan convert) a scan line signal sequence of an ultrasound scan into a scan line signal sequence of a video format typified by a television or the like, and generates image data for display. The processing circuitry 180 may also perform various processes on the image data for display, such as dynamic range, brightness, contrast, and gamma curve correction, and RGB conversion. The processing circuitry 180 may also add supplementary information, such as text information of various parameters, scales, and body marks, to the image data for display. The processing circuitry 180 may also generate a user interface (GUI: Graphical User Interface) for an operator to input various instructions via an input device, and display the GUI on a display.

[0052] Furthermore, the processing circuitry 180 may display the rendering image generated by the rendering function 185D using the display control function 186. The processing circuitry 180 may also display a GUI related to the settings of the rendering image together with the rendering image. This allows the user to change the parameters displayed on the GUI in real time to change the rendering image to a desired display. The parameters that the user can change include, for example, an opacity parameter, a base color parameter, and a rendering parameter.

[0053] The system control function 187 is a function that controls the overall operation of the ultrasound diagnostic apparatus 1. For example, the system control function 187 causes the processing circuitry 180 to control the ultrasound transmission circuitry 110 and the ultrasound reception circuitry 120 based on parameters related to the transmission and reception of ultrasound.

[0054] The configuration of the ultrasound diagnostic apparatus according to the first embodiment has been described above. Next, imaging of a fetus will be described as an example of the region estimation process and the rendering image generation process according to the first embodiment. Note that the subject to be imaged is not limited to a fetus.

[0055] Fig. 2 is an explanatory diagram illustrating a typical method of imaging a fetus. Fig. 2 schematically shows the state of the fetus UB in the living body P as the mother. In ultrasound imaging of the fetus, the user presses the ultrasound probe 101 against the body surface of the living body P so that the fetus UB as the object of observation is within the scan area. Then, the ultrasound diagnostic device 1 performs an ultrasound scan to generate three-dimensional data regarding the fetus UB.

[0056] 2 indicates the direction from the object of observation toward the radiation surface of the ultrasound probe 101. Generally, when a three-dimensional ultrasound scan of a fetus is performed using an ultrasound diagnostic device, it is assumed that the image will be viewed from the direction in which the probe is placed. Therefore, when generating a simplified rendering image to be used in the body part estimation process, the ultrasound diagnostic device 1 sets the ray direction to direction d and then generates the simplified rendering image based on the three-dimensional data.

[0057] Fig. 3 is a flowchart showing an example of the operation of the processing circuit of the ultrasound diagnostic apparatus according to the first embodiment. The flowchart of Fig. 3 is started, for example, when a user executes an internal transparency display mode. The flowchart of Fig. 3 explains the generation of a rendering image for three-dimensional data generated from one data set. In the following, as a specific example, it is assumed that a rendering image of the fetus UB in Fig. 2 is generated.

[0058] (Step ST110) When the internal transparency display mode is executed, processing circuitry 180 executes acquisition function 185A. When acquisition function 185A is executed, processing circuitry 180 acquires a two-dimensional image and three-dimensional data. The following describes a case where an ultrasound image is acquired as a two-dimensional image.

[0059] Fig. 4 is an example of a two-dimensional image used in region estimation in the first embodiment. The ultrasound image 400 in Fig. 4 includes a cross-sectional image 410 of the fetus UB in Fig. 2. The ultrasound image 400 was scanned at the same position as the ultrasound probe 101 at which the three-dimensional data was acquired.

[0060] (Step ST120) After acquiring the two-dimensional image, processing circuitry 180 executes region estimation function 185B. When region estimation function 185B is executed, processing circuitry 180 estimates a region based on the two-dimensional image.

[0061] FIG. 5 is a block diagram illustrating an overview of the part estimation process in the first embodiment. Processing circuitry 180 executes part estimation process 500 on the acquired two-dimensional image to generate part information as an estimation result. Part estimation process 500 may be inference using DNN or image processing using pattern recognition. The part information, for example, associates part information with a position on the two-dimensional image. The two-dimensional image associated with part information may be called a part estimation image.

[0062] Fig. 6 is an example of a part estimation image in the first embodiment. Part estimation image 600 in Fig. 6 is generated, for example, by executing part estimation process 500 on ultrasound image 400. Part estimation image 600 shows a whole body outline 610 of fetus UB and a brain region 620.

[0063] (Step ST130) After estimating the region, processing circuitry 180 executes parameter setting function 185C. By executing parameter setting function 185C, processing circuitry 180 acquires a representative brightness value of the estimated region in the three-dimensional data. Specifically, processing circuitry 180 acquires multiple brightness values ​​in the three-dimensional data corresponding to the contour or region of the region estimated in the two-dimensional image. Processing circuitry 180 sets the average or median of the acquired multiple brightness values ​​as the representative brightness value.

[0064] (Step ST140) After acquiring the representative luminance value, the processing circuit 180 determines a threshold value corresponding to the representative luminance value. Specifically, the processing circuit 180 determines luminance values ​​before and after the representative luminance value based on a table that associates body parts with threshold widths.

[0065] (Step ST150) After determining the threshold value, the processing circuit 180 sets parameters based on the representative luminance value and the threshold value. Specifically, the processing circuit 180 generates a transfer function related to opacity based on the representative luminance value and the preceding and following threshold values, and sets rendering parameters based on the transfer function. Parameters included in the rendering parameters other than the transfer function may be determined in advance. Below, transfer functions when there is one representative luminance value and when there are multiple representative luminance values ​​will be described with reference to FIGS. 7 and 8.

[0066] FIG. 7 shows a first specific example of a transfer function associating brightness values ​​with opacity in the first embodiment. The transfer function 700 in FIG. 7 is expressed as a parabola connecting the vertex at the opacity corresponding to the representative brightness value R and the low brightness threshold th1 and the high brightness threshold th2, which are thresholds before and after the representative brightness value R. The opacity corresponding to the representative brightness value R is determined in advance, for example, depending on the region. The opacity of the low brightness threshold th1 and the high brightness threshold th2 is zero. In other words, the transfer function 700 is a parameter that visualizes brightness values ​​in three-dimensional data that fall within the range from the low brightness threshold th1 to the high brightness threshold th2 in a rendering image.

[0067] FIG. 8 shows a second example of a transfer function associating brightness values ​​with opacity in the first embodiment. The transfer function 800 in FIG. 8 is represented by a first parabola connecting the first vertex with a low brightness threshold th11 and a high brightness threshold th2, which are thresholds before and after the representative brightness value R1, and a second parabola connecting the second vertex with a low brightness threshold th21 and a high brightness threshold th22, which are thresholds before and after the representative brightness value R2, and a second parabola connecting the second vertex with an opacity corresponding to a representative brightness value R2, which is greater than the representative brightness value R1. The opacities corresponding to the representative brightness values ​​R1 and R2 are predetermined for each region, for example. The opacities of the low brightness threshold th11, the high brightness threshold th12, the low brightness threshold th21, and the high brightness threshold th22 are zero. That is, the transfer function 800 is a parameter for visualizing brightness values ​​on three-dimensional data that fall within a first range from a low brightness threshold th11 to a high brightness threshold th12 and a second range from a low brightness threshold th21 to a high brightness threshold th22. Note that the user may select either the representative brightness value R1 or the representative brightness value R2, and use the transfer function based on the selected representative brightness value in subsequent processing.

[0068] (Step ST160) After setting the parameters, processing circuitry 180 executes rendering function 185D. When executing rendering function 185D, processing circuitry 180 generates a rendered image based on the set parameters. Specifically, processing circuitry 180 generates the rendered image by executing a global illumination rendering process using the rendering parameters.

[0069] (Step ST170) After generating the rendering image, the processing circuitry 180 executes the display control function 186. When the display control function 186 is executed, the processing circuitry 180 causes the rendering image to be displayed on a display serving as the output device 103. After the processing of step ST170, the flowchart in FIG. 3 ends.

[0070] In the above flowchart, when a simple rendering image is used for the two-dimensional image used in the part estimation process, only three-dimensional data may be acquired in step ST110, and a step of generating a simple rendering image from the three-dimensional data may be added after step ST110. Thus, processing circuitry 180 generates the simple rendering image before generating the rendering image that will ultimately be displayed. Furthermore, both an ultrasound image and a simple rendering image may be used as the two-dimensional image in the part estimation process.

[0071] The process of FIG. 3 may be repeatedly executed until the user ends the inner transparent display mode or changes to another mode.

[0072] As described above, the ultrasound diagnostic apparatus according to the first embodiment acquires a two-dimensional image related to three-dimensional data including an observation target, estimates the area of ​​the observation target based on the two-dimensional image, sets rendering parameters based on the estimated area and the three-dimensional data, and generates a rendering image of the three-dimensional data using the rendering parameters.

[0073] Therefore, the ultrasound diagnostic apparatus according to the first embodiment can reduce the burden on the user in setting parameters for a rendering image by automatically setting rendering parameters for a region to be observed.

[0074] (Second embodiment) In the first embodiment, an ultrasound diagnostic device having multiple functions related to rendering image generation processing including body part estimation processing has been described, whereas in the second embodiment, an information processing device having these multiple functions will be described.

[0075] 9 is a block diagram showing an example of the configuration of an information processing device according to the second embodiment. The information processing device 900 in FIG. 9 is connected to an input device 901 and an output device 902. The information processing device 900 is also connected to a medical imaging device 903 via a network NW. The medical imaging device 903 corresponds to, for example, an ultrasound diagnostic device. The input device 901 and the output device 902 are substantially the same as the input device 102 and the output device 103 in FIG. 1.

[0076] The information processing device 900 is a device that executes rendering image generation processing including part estimation processing to generate a rendering image. The information processing device 900 has a memory circuitry 910, an input interface 920, an output interface 930, a communication interface 940, and a processing circuitry 950.

[0077] The storage circuitry 910 includes a processor-readable storage medium, such as a magnetic storage medium, an optical storage medium, or a semiconductor memory. The storage circuitry 910 stores a program related to the region estimation process, a program related to the rendering image generation process, and various data. The programs and various data may be stored in the storage circuitry 910 in advance. Alternatively, the programs and various data may be stored in a non-transitory storage medium and distributed, and then read from the non-transitory storage medium and installed in the storage circuitry 910. The storage circuitry 910 also stores B-mode image data, contrast image data, image data related to blood flow images, and three-dimensional data generated by the medical imaging device 903 in accordance with operations input via the input interface 920.

[0078] The storage circuitry 910 may be a drive device that reads and writes various information from and to a portable storage medium such as a CD drive, a DVD drive, or a flash memory. The storage circuitry 910 can also write stored data to the portable storage medium and store the data in an external device via the portable storage medium.

[0079] The input interface 920 accepts various instructions from an operator via the input device 901. Examples of the input device 901 include a mouse, a keyboard, a panel switch, a slider switch, a trackball, a rotary encoder, an operation panel, and a touch panel. The input interface 920 is connected to the processing circuit 950 via a bus, for example, converts operation instructions input by the operator into electrical signals, and outputs the electrical signals to the processing circuit 950. Note that the input interface 920 is not limited to those connected to physical operation components such as a mouse and a keyboard. For example, a circuit that receives electrical signals corresponding to operation instructions input from an external input device provided separately from the information processing device 900 and outputs the electrical signals to the processing circuit 950 is also included as an example of an input interface.

[0080] The output interface 930 is an interface for outputting, for example, an electrical signal from the processing circuit 950 to the output device 902. The output device 902 is any display such as a liquid crystal display, an organic EL display, an LED display, a plasma display, or a CRT display. The output device 902 may be a touch panel display that also serves as the input device 901. In addition to the display, the output device 902 may further include a speaker that outputs audio. The output interface 930 is connected to the processing circuit 950 via, for example, a bus, and outputs the electrical signal from the processing circuit 950 to the output device 902.

[0081] The communication interface 940 is connected to the medical imaging apparatus 903 via, for example, a network NW, and performs data communication with the medical imaging apparatus 903 .

[0082] The processing circuitry 950 is, for example, a processor that functions as the core of the information processing device 900. The processing circuitry 950 executes a program stored in the storage circuitry 910 to realize a function corresponding to the program. The processing circuitry 950 has, for example, an acquisition function 951A that functions as an acquisition unit, a part estimation function 951B that functions as a part estimation unit, a parameter setting function 951C that functions as a parameter setting unit, a rendering function 951D that functions as a rendering unit, and a display control function 952 that functions as a display control unit.

[0083] The acquisition function 951A is a function for acquiring data related to the rendering image generation process including the part estimation process. Specifically, the acquisition function 951A causes the processing circuitry 950 to acquire parameters input by the user, parameters set as defaults in the information processing device 900, and three-dimensional data from the medical imaging device 903.

[0084] The part estimation function 951B, the parameter setting function 951C, the rendering function 951D, and the display control function 952 each have functions that are approximately the same as, for example, the part estimation function 185B, the parameter setting function 185C, the rendering function 185D, and the display control function 186 in the first embodiment, respectively.

[0085] The information processing device 900 may generate three-dimensional data based on data relating to a living body received from the medical imaging device 903 (for example, a received signal in an ultrasound diagnostic device).

[0086] Therefore, the information processing device according to the second embodiment can be expected to have the same effects as those of the first embodiment.

[0087] (Other embodiments) In the above embodiments, a rendering image is displayed based on one estimation result obtained in the part estimation process. In other embodiments, a rendering image is displayed based on multiple estimation results.

[0088] For example, in the case of a fetus scan, the multiple estimation results include a first estimation result for the whole body only and a second estimation result for the head and whole body. The first estimation result generates, for example, transfer function 700 shown in Figure 7, and the second estimation result generates, for example, transfer function 800 shown in Figure 8.

[0089] The ultrasound diagnostic device or the information processing device generates a first rendering image using first rendering parameters set based on the transfer function 700, generates a second rendering image using second rendering parameters set based on the transfer function 800, and displays these rendering images on a display, thereby allowing a user to select a desired image from the first rendering image and the second rendering image displayed on the display.

[0090] According to at least one of the embodiments described above, it is possible to reduce the burden on the user in setting parameters for a rendering image.

[0091] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0092] 1. Ultrasound diagnostic equipment 100 Device body 101 Ultrasound probe 102 Input Device 103 Output Device 104 External device 110 Ultrasonic transmission circuit 120 Ultrasonic receiving circuit 130 Internal memory circuit 140 image memory 150 Input Interface 160 output interface 170 Communication Interface 180 Processing Circuit 181 B-mode processing function 182 Doppler processing function 183 Image generation function 184 3D data generation function 185A Acquisition Function 185B Body part estimation function 185C Parameter setting function 185D rendering function 186 Display Control Function 187 System Control Functions 400 ultrasound images 410 Cross-sectional image 500 Body part estimation processing 600 body part estimation images 610 Contour 620 areas 700,800 transfer functions 900 Information Processing Equipment 901 Input Device 902 Output Device 903 Medical Imaging Equipment 910 Memory circuit 920 input interface 930 output interface 940 Communication Interface 950 Processing Circuit 951A Acquisition Function 951B Body Part Estimation Function 951C parameter setting function 951D Rendering Function 952 Display Control Function d direction NW Network P living organism R, R1, R2 representative brightness value th1,th11,th21 low brightness threshold th2,th12,th22 High brightness threshold UB fetus

Claims

1. an acquisition unit that acquires a two-dimensional image related to three-dimensional data including an observation target; a part estimation unit that estimates a part of the observation object based on the two-dimensional image; a parameter setting unit that sets rendering parameters based on the estimated region and the three-dimensional data; a rendering unit that generates a rendering image of the three-dimensional data using the rendering parameters; Equipped with The parameter setting unit determining thresholds before and after a representative luminance value based on the luminance value of the three-dimensional data corresponding to a body part based on a table in which a pre-stored body part is associated with a threshold width; The information processing device sets the rendering parameters based on the representative luminance value and the preceding and following threshold values.

2. The two-dimensional image is an ultrasound image scanned at the same position as the position of the ultrasound probe that acquired the three-dimensional data. The information processing device according to claim 1 .

3. the two-dimensional image is a simplified rendering image rendered using a rendering method that projects the entire three-dimensional data; The information processing device according to claim 1 .

4. the rendering unit generates the simplified rendering image before generating the rendering image; The information processing device according to claim 3 .

5. the rendering unit generates the simplified rendering image by performing a total additive projection of the three-dimensional data. The information processing device according to claim 4 .

6. the rendering unit generates the simplified rendering image using a direction toward an emission surface of an ultrasound probe that has acquired the three-dimensional data as a ray direction.

6. The information processing device according to claim 4.

7. the parameter setting unit generates a transfer function related to opacity based on the representative luminance value and the front and rear thresholds, and sets the rendering parameters based on the transfer function. The information processing device according to claim 1 .

8. the rendering unit generates the rendering image by executing a global illumination rendering process. The information processing device according to any one of claims 1 to 7.

9. The observation subject is a fetus. The information processing device according to any one of claims 1 to 8.

10. The site is any one of the brain, head, spine, and whole body. The information processing device according to any one of claims 1 to 9.

11. a three-dimensional data generating unit that generates the three-dimensional data based on the echo reflection intensity acquired by the ultrasonic probe; The information processing device according to claim 1 , further comprising:

12. an ultrasound probe for acquiring echo reflection intensity from a living body; a three-dimensional data generating unit that generates three-dimensional data based on the echo reflection intensity; The information processing device according to any one of claims 1 to 10. An ultrasound diagnostic device comprising:

13. acquiring a two-dimensional image associated with three-dimensional data including an object of observation; estimating a region of the observation target based on the two-dimensional image; setting rendering parameters based on the estimated region and the three-dimensional data; generating a rendering image of the three-dimensional data using the rendering parameters; Equipped with setting the rendering parameters determining thresholds before and after a representative luminance value based on the luminance value of the three-dimensional data corresponding to a body part based on a table in which a pre-stored body part is associated with a threshold width; The method further comprises setting the rendering parameters based on the representative luminance value and the front and rear thresholds.

Images