Ultrasonic tomographic image processing apparatus and ultrasonic tomographic image processing program

Abstract

To improve specification accuracy when specifying a tissue structure contained in an ultrasonic tomographic image using a first learning model for predicting a cross section type of the ultrasonic tomographic image and a second learning model for predicting a tissue structure contained in an ultrasonic tomographic image of a specific cross section type.SOLUTION: A cross section type specification unit 38 inputs an object image to a first learning model 32, and, on the basis of a prediction result of the first learning model 32 for the object image TI, specifies a cross section type of the object image TI. A tissue structure specification unit 40 inputs the object image TI to a first second learning model 34 associated with a specific cross section, and specifies a first tissue structure contained in the object image TI. A display control unit 22 causes a display 24 to display first tissue structure information on the first tissue structure. The tissue structure specification unit 40 inputs the object image TI to another second learning model 34 other than the first second learning model 34 according to an instruction from a user, and specifies a second tissue structure contained in the object image TI.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This specification discloses improvements to an ultrasonic tomography image processing apparatus and an ultrasonic tomography image processing program.

Background Art

[0002] Conventionally, there is a technique for identifying tissue structures (such as organs and blood vessels) included in an ultrasonic tomography image (B-mode image) by analyzing the ultrasonic tomography image formed by an ultrasonic diagnostic apparatus. Conventionally, there are also ultrasonic diagnostic apparatuses having such functions. By identifying tissue structures in an ultrasonic tomography image, for example, measurements related to the identified tissue structures can be performed using the ultrasonic tomography image.

[0003] For example, Patent Document 1 discloses an ultrasonic diagnostic apparatus that automatically discriminates tissue structures as measurement targets included in an ultrasonic tomography image by performing image recognition processing such as processing using a learning model or pattern matching processing with template data on the formed ultrasonic tomography image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Incidentally, consider identifying tissue structures included in an ultrasonic tomographic image using a first learning model that predicts and outputs a cross-sectional type of the ultrasonic tomographic image, and a plurality of second learning models that predict and output tissue structures included in the ultrasonic tomographic image. The plurality of second learning models are each associated with each cross-sectional type. In this case, first, the ultrasonic tomographic image to be processed is input to the first learning model, whereby the cross-sectional type of the ultrasonic tomographic image is identified. Next, the ultrasonic tomographic image is input to the second learning model associated with the identified cross-sectional type, whereby the tissue structures included in the ultrasonic tomographic image are identified.

[0006] When identifying tissue structures by the above processing, it is desirable to improve the identification accuracy.

[0007] An object of the ultrasonic tomographic image processing apparatus according to the present embodiment is to improve the identification accuracy when identifying tissue structures included in an ultrasonic tomographic image using a first learning model that predicts a cross-sectional type of the ultrasonic tomographic image and a second learning model that predicts tissue structures included in an ultrasonic tomographic image of a specific cross-sectional type.

Means for Solving the Problem

[0008] The ultrasonic tomography image processing apparatus disclosed in this specification is an ultrasonic tomography image processing apparatus accessible to a first learning model trained to predict and output the cross-sectional type of an input ultrasonic tomography image, and a plurality of second learning models respectively associated with each cross-sectional type of the ultrasonic tomography image, wherein each of the second learning models is trained to predict and output the tissue structure included in the ultrasonic tomography image of the corresponding cross-sectional type. The ultrasonic tomography image processing apparatus includes a cross-sectional type specifying unit that specifies a specific cross-section, which is the cross-sectional type of the target image, by inputting the target image, which is the ultrasonic tomography image to be processed, into the first learning model; a tissue structure specifying unit that specifies a first tissue structure included in the target image by inputting the target image into a first second learning model, which is the second learning model associated with the specific cross-section; and a display control unit that causes the display unit to display first tissue structure information, which is information about the first tissue structure. The tissue structure specifying unit inputs the target image into the second learning models other than the first second learning model, including the second learning models associated with cross-sectional types other than the specific cross-section, in response to an instruction from the user who has confirmed the first tissue structure information and indicating a position on the target image, and specifies a second tissue structure included in the vicinity of the position indicated by the user on the target image based on the prediction result of the second learning model.

[0009] The display control unit causes the display unit to display cross-section information indicating the specific cross-section. When the specific cross-section and the cross-sectional type associated with the second learning model that contributed to the specification of the second tissue structure are different from each other, the display control unit may cause the display unit to display, instead of the cross-section information indicating the specific cross-section, modified cross-section information indicating the cross-sectional type corresponding to the second learning model that contributed to the specification of the second tissue structure.

[0010] The plurality of second learning models are grouped corresponding to each part of the subject, and the tissue structure specifying unit inputs the target image into the second learning models belonging to the same group as the first second learning model among the plurality of second learning models other than the first second learning model according to an instruction from a user who has confirmed the first tissue structure information, and further specifies a second tissue structure included in the target image based on the prediction result of the second learning model.

[0011] The tissue structure specifying unit inputs the target image into each of the plurality of second learning models other than the first second learning model according to an instruction from a user who has confirmed the first tissue structure information, calculates the order of prediction accuracy for each of the labels of the plurality of tissue structures predicted by each of the plurality of second learning models, the display control unit causes the display unit to display the labels of the plurality of tissue structures predicted by each of the plurality of second learning models in a display mode expressing the order of the prediction accuracy, and the tissue structure specifying unit specifies, as the second tissue structure, the tissue structure related to the label selected by the user among the plurality of tissue structures predicted by each of the plurality of second learning models.

[0012] An image adjustment unit that adjusts the image quality of the ultrasonic tomographic image based on image quality adjustment information in which a cross-section type of the ultrasonic tomographic image and an image quality adjustment parameter for adjusting the image quality of the ultrasonic tomographic image of the cross-section type are associated with each other, and when the specific cross-section and the cross-section type associated with the second learning model that contributed to the specification of the second tissue structure are different from each other, the image adjustment unit that adjusts the image quality of the target image using the image quality adjustment parameter associated with the cross-section type corresponding to the second learning model that contributed to the specification of the second tissue structure, may be further provided.

[0013] In addition, the ultrasonic tomography image program disclosed in this specification includes a first learning model trained to predict and output the cross-sectional type of the input ultrasonic tomography image, and a plurality of second learning models respectively associated with each cross-sectional type of the ultrasonic tomography image, wherein each of the second learning models is trained to predict and output the tissue structure included in the ultrasonic tomography image of the corresponding cross-sectional type. A computer accessible to the plurality of second learning models inputs a target image, which is an ultrasonic tomography image to be processed, into the first learning model to identify a specific cross-section, which is the cross-sectional type of the target image. A tissue structure identification unit that inputs the target image into the first second learning model, which is the second learning model associated with the specific cross-section, to identify a first tissue structure included in the target image, and a display control unit that causes the display unit to display first tissue structure information, which is information about the first tissue structure. The tissue structure identification unit inputs the target image into the second learning models other than the first second learning model, including the second learning models associated with cross-sectional types other than the specific cross-section, in response to an instruction from a user who has confirmed the first tissue structure information and indicates a position on the target image, and based on the prediction result of the second learning model, identifies a second tissue structure included in the vicinity of the position indicated by the user on the target image.

Advantages of the Invention

[0014] According to the medical image processing apparatus disclosed in this specification, when identifying the tissue structure included in the ultrasonic tomography image by using a first learning model that predicts the cross-sectional type of the ultrasonic tomography image and a second learning model that predicts the tissue structure included in the ultrasonic tomography image of a specific cross-sectional type, the identification accuracy can be improved.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Mode for Carrying Out the Invention

[0016] FIG. 1 is a schematic configuration diagram of an ultrasonic diagnostic apparatus 10 as an ultrasonic tomography image processing apparatus according to the present embodiment. The ultrasonic diagnostic apparatus 10 is a medical device installed in a medical institution such as a hospital.

[0017] The ultrasonic diagnostic apparatus 10 is an apparatus that scans an ultrasonic beam on a subject and generates an ultrasonic image, which is a medical image, based on the received signal obtained thereby. In particular, in the present embodiment, the ultrasonic diagnostic apparatus 10 forms an ultrasonic tomographic image (B-mode image) in which the amplitude intensity of the reflected wave from the scanning surface is converted into luminance based on the received signal. Note that the ultrasonic diagnostic apparatus 10 can also form other ultrasonic images such as a Doppler image representing the movement speed of tissues in the subject formed based on the difference in frequency (Doppler shift) between the transmitted wave and the received wave.

[0018] Although details will be described later, the ultrasonic diagnostic apparatus 10 performs a process of identifying tissue structures in the subject by analyzing the formed ultrasonic tomographic image. In the present embodiment, the ultrasonic diagnostic apparatus 10 performs processes such as measurement on the identified tissue object. That is, in the present embodiment, the tissue object to be identified is the measurement object.

[0019] The transmission / reception unit 14, signal processing unit 16, image forming unit 18, image quality adjustment unit 20, display control unit 22, cross-section type identification unit 38, and tissue structure identification unit 40 included in the ultrasonic diagnostic apparatus 10 are configured by a processor. The processor includes at least one of a general-purpose processing device (such as a CPU (Central Processing Unit)) and a dedicated processing device (such as a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or programmable logic device). The processor may be configured by the cooperation of a plurality of physically separated processing devices rather than by a single processing device. Further, each of the above units may be realized by the cooperation of hardware such as a processor and software.

[0020] The ultrasonic probe 12 is a device that transmits and receives ultrasonic waves to and from a subject (particularly a tissue object to be measured). The ultrasonic probe 12 has a vibration element array composed of a plurality of vibration elements that transmit and receive ultrasonic waves to and from the subject.

[0021] The transmission / reception unit 14 transmits a transmission signal to the ultrasonic probe 12 (specifically, each vibration element of the vibration element array) under the control from the control unit 28 (described later). Thereby, ultrasonic waves are transmitted from each vibration element toward the subject.

[0022] Further, the transmission / reception unit 14 receives a reception signal from each vibration element that has received a reflected wave from the subject. The transmission / reception unit 14 has an adder and a plurality of delay units corresponding to each vibration element, and performs an in-phase addition process of aligning the phases of the reception signals from each vibration element and adding them by the adder and the plurality of delay units. Thereby, a reception beam signal in which information indicating the signal intensity of the reflected wave from the subject is arranged in the depth direction of the subject is formed.

[0023] The signal processing unit 16 performs various signal processes including filter processing such as applying a band-pass filter and detection processing on the received beam signal from the transmission / reception unit 14.

[0024] The image forming unit 18 forms an ultrasonic tomographic image (B-mode image) based on the received beam signal that has been signal-processed in the signal processing unit 16. First, the image forming unit 18 converts the received beam signal into data in the coordinate space of the ultrasonic image. Then, the image forming unit 18 forms an ultrasonic tomographic image based on the coordinate conversion signal.

[0025] The image quality adjustment unit 20 executes a process of adjusting the image quality (for example, luminance, etc.) of the ultrasonic tomographic image formed by the image forming unit 18. The image quality adjustment unit 20 executes an image quality adjustment process according to the processing result (that is, the cross-section type of the ultrasonic tomographic image) of the cross-section type specifying unit 38 described later. Details of the processing content of the image quality adjustment unit 20 will be described later.

[0026] The display control unit 22 performs control to display the ultrasonic tomographic image formed by the image forming unit 18 and various other information on the display 24. The display 24 as the display unit is a display device composed of, for example, a liquid crystal display or an organic EL (Electro Luminescence).

[0027] The input interface 26 is composed of, for example, buttons, a trackball, a touch panel, etc. The input interface 26 is used to input a user's command into the ultrasonic diagnostic apparatus 10. In the present embodiment, the display 24 serves as a touch panel, and the display 24 also exhibits the function of the input interface 26.

[0028] The control unit 28 is configured to include at least one of a general-purpose processor (such as a CPU (Central Processing Unit)) and a dedicated processor (such as a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a programmable logic device). The control unit 28 may be configured by the cooperation of a plurality of processing devices existing at physically separated positions, rather than by a single processing device. The control unit 28 controls each part of the ultrasonic diagnostic apparatus 10 according to an ultrasonic tomography image processing program stored in a memory 30 described later.

[0029] The memory 30 is configured to include an HDD (Hard Disk Drive), an SSD (Solid State Drive), an eMMC (embedded Multi Media Card), or a ROM (Read Only Memory). The memory 30 is connected to each part shown in FIG. 1 so as to be accessible from each part including the processor of the ultrasonic diagnostic apparatus 10, specifically, the control unit 28, and the cross-sectional type specifying unit 38 and the tissue structure specifying unit 40 described later. An ultrasonic tomography image processing program for operating each part of the ultrasonic diagnostic apparatus 10 is stored in the memory 30. Note that the ultrasonic tomography image processing program can also be stored in a computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory or a CD-ROM. The ultrasonic diagnostic apparatus 10 can read and execute the ultrasonic tomography image processing program from such a storage medium.

[0030] Also, as shown in FIG. 1, a first learning model 32 and a plurality of second learning models 34 are stored in the memory 30.

[0031] The first learning model 32 is composed of a model such as a CNN (Convolutional Neural Network), but it may be any model as long as it can exhibit the functions described below. The first learning model 32 uses the ultrasonic tomographic image as input data, predicts the cross-sectional type of the input ultrasonic tomographic image, and outputs the prediction result.

[0032] In this embodiment, the first learning model 32 is learned by a learning device which is a device other than the ultrasonic diagnostic apparatus 10, and the learned second learning model 34 is stored in the memory 30. The processor of the learning device uses a combination of an ultrasonic tomographic image and information (teacher data) indicating the cross-sectional type of the ultrasonic tomographic image as learning data to learn the first learning model 32. Specifically, the processor of the learning device inputs the ultrasonic tomographic image which is the learning data into the first learning model 32. The first learning model 32 predicts the cross-sectional type of the input ultrasonic tomographic image and outputs the prediction result. The processor of the learning device adjusts the parameters of the first learning model 32 so that the difference between the cross-sectional type predicted by the first learning model 32 and the cross-sectional type which is the teacher data becomes small. By repeating such learning processing, the learned first learning model 32 can predict and output the cross-sectional type of the ultrasonic tomographic image. However, even if it is a sufficiently learned first learning model 32, it cannot completely and accurately predict the cross-sectional type of the ultrasonic tomographic image, and there may be cases where an inaccurate prediction result is output.

[0033] The memory 30 may store one first learning model 32, or may store a plurality of first learning models 32 respectively corresponding to a plurality of parts of the subject (such as the abdomen and lower limbs). In this case, each first learning model 32 is a model specialized in predicting the cross-sectional type of the ultrasonic tomographic image of the corresponding part. For example, the first learning model 32 corresponding to the abdomen is a model specialized in predicting the cross-sectional type of the ultrasonic tomographic image formed by transmitting and receiving ultrasonic waves to and from the abdomen, and the first learning model 32 corresponding to the lower limb is a model specialized in predicting the cross-sectional type of the ultrasonic tomographic image formed by transmitting and receiving ultrasonic waves to and from the lower limb. In this case, each first learning model 32 is learned using the ultrasonic tomographic image of the corresponding part as learning data. For example, the first learning model 32 corresponding to the abdomen is learned using the ultrasonic tomographic image formed by transmitting and receiving ultrasonic waves to and from the abdomen as learning data, and the first learning model 32 corresponding to the lower limb is learned using the ultrasonic tomographic image formed by transmitting and receiving ultrasonic waves to and from the lower limb as learning data.

[0034] The second learning model 34 is also composed of a model such as a CNN, for example, but may be any model as long as it can exhibit the functions described below. The second learning model 34 takes an ultrasonic tomographic image as input data, predicts the tissue structure included in the input ultrasonic tomographic image, and outputs the prediction result. In this specification, predicting (or specifying) the tissue structure included in the ultrasonic tomographic image means predicting (or specifying) at least one of the position of the tissue structure on the ultrasonic tomographic image (hereinafter simply referred to as "the position of the tissue structure") or the label (the name of the tissue structure). The second learning model 34 according to this embodiment predicts the position and label of the tissue structure.

[0035] In this embodiment, the second learning model 34 is learned by a learning device which is a device other than the ultrasonic diagnostic apparatus 10, and the learned second learning model 34 is stored in the memory 30. The processor of the learning device uses, as learning data, a combination of an ultrasonic tomographic image and information (teacher data) indicating the position and label of the tissue structure included in the ultrasonic tomographic image to learn the second learning model 34. Specifically, the processor of the learning device inputs the ultrasonic tomographic image, which is the learning data, into the second learning model 34. The second learning model 34 predicts the position and label of the tissue structure included in the input ultrasonic tomographic image and outputs the prediction result. The processor of the learning device adjusts the parameters of the second learning model 34 so that the difference between the position of the tissue structure predicted by the second learning model 34 and the position of the tissue structure which is the teacher data becomes small. Further, the processor of the learning device adjusts the parameters of the second learning model 34 so that the difference between the label of the tissue structure predicted by the second learning model 34 and the label of the tissue structure which is the teacher data becomes small. By repeating such learning processing, the learned second learning model 34 can predict and output the position and label of the tissue structure included in the ultrasonic tomographic image. Note that when the second learning model 34 predicts only one of the position or label of the tissue structure, it is sufficient that the teacher data included in the learning data includes only one of the position or label of the tissue structure.

[0036] The memory 30 stores a plurality of second learning models 34. The plurality of second learning models 34 are each associated with each cross-sectional type of the ultrasonic tomographic image. For example, one second learning model 34 is associated with the SFJ (Sapheno Femoral Junction) cross-section, and another second learning model 34 is associated with the CFV (Common Femoral Vein) cross-section, and so on. Each second learning model 34 is a model specialized in predicting the tissue structures included in the ultrasonic tomographic image of the corresponding cross-sectional type. For example, the second learning model 34 associated with the SFJ cross-section is a model specialized in predicting the tissue structures included in the ultrasonic tomographic image whose cross-sectional type is the SFJ cross-section, and the second learning model 34 associated with the CFV cross-section is a model specialized in predicting the tissue structures included in the ultrasonic tomographic image whose cross-sectional type is the CFV cross-section. Each second learning model 34 is learned using the ultrasonic tomographic image of the corresponding cross-sectional type as learning data. For example, the second learning model 34 corresponding to the SFJ cross-section is learned using the ultrasonic tomographic image whose cross-sectional type is the SFJ cross-section as learning data, and the second learning model 34 corresponding to the CFV cross-section is learned using the ultrasonic tomographic image whose cross-sectional type is the CFV cross-section as learning data.

[0037] Although one second learning model 34 may be associated with each cross-sectional type of the ultrasonic tomographic image, in this embodiment, a plurality of different second learning models 34 are associated with one cross-sectional type. For example, a plurality of different second learning models 34 are associated with the SFJ cross-section. The plurality of second learning models 34 associated with one cross-sectional type have different parameters (including parameters adjusted by the learning process or hyperparameters not adjusted by the learning process), or different preprocessing (processing the input data prior to the prediction process by the learning model). That is, when the same ultrasonic tomographic image is input, the prediction results for the plurality of second learning models 34 associated with one cross-sectional type can be different from each other.

[0038] In the present embodiment, the first learning model 32 and the second learning model 34 are learned by a learning device different from the ultrasonic diagnostic apparatus 10. However, the first learning model 32 and the second learning model 34 may be learned by the ultrasonic diagnostic apparatus 10. In that case, the ultrasonic diagnostic apparatus 10 is configured by a processor or has a learning processing unit (not shown in FIG. 1) realized by cooperation of hardware such as a processor and software, and the learning processing unit executes the learning processing of the first learning model 32 and the second learning model 34.

[0039] Also, as shown in FIG. 1, image quality adjustment information 36 is stored in the memory 30. Details of the image quality adjustment information 36 will be described later.

[0040] The cross-section type specifying unit 38 specifies the cross-section type of the ultrasonic tomographic image formed by the image forming unit 18. In this specification, the ultrasonic tomographic image to be processed by the cross-section type specifying unit 38 (and the tissue structure specifying unit 40) is referred to as a target image.

[0041] Specifically, the cross-section type specifying unit 38 inputs the target image to the first learning model 32, and specifies the cross-section type of the target image based on the prediction result of the first learning model 32 for the target image. In this specification, the cross-section type of the target image specified by the cross-section type specifying unit 38 is referred to as a specified cross-section.

[0042] The tissue structure specifying unit 40 specifies the tissue structure included in the target image. First, the tissue structure specifying unit 40 inputs the target image into the second learning model 34 associated with the specific cross-section specified by the cross-section type specifying unit 38 among the plurality of second learning models 34 stored in the memory 30, and based on the prediction result of the second learning model 34 for the target image, specifies the tissue structure included in the target image. When a plurality of second learning models 34 are associated with the specific cross-section, the tissue structure specifying unit 40 inputs the target image into one of the second learning models 34 determined in advance. In this specification, one second learning model 34 selected according to the specific cross-section specified by the cross-section type specifying unit 38 in this way is called the first second learning model 34. Also, based on the output of the first second learning model 34, the tissue structure specified by the tissue structure specifying unit 40 is called the first tissue structure.

[0043] The display control unit 22 causes the display 24 to display first tissue structure information, which is information about the first tissue structure specified by the tissue structure specifying unit 40. FIG. 2 is a diagram showing an example of the display of the first tissue structure information S1. In the present embodiment, the display control unit 22 causes the display 24 to display the target image TI, the cross-section information CS indicating the specific cross-section, and the first tissue structure information S1. The first tissue structure information S1 includes an icon indicating the position of the first tissue structure on the target image TI, or an image icon or character indicating a label. In the example of FIG. 2, as the first tissue structure information S1, an icon indicating the position of the first tissue structure on the target image TI is displayed superimposed on the target image TI. In addition to this, characters indicating the label of the first tissue structure may also be displayed.

[0044] Here, assume that the first tissue structure is incorrect. The first tissue structure being incorrect includes cases where the position and label of the identified first tissue structure do not match the tissue structure expected by the user, although they match the facts. For example, at the position indicated by the first tissue structure information S1, there is indeed the great saphenous vein, and its label is also the great saphenous vein. However, as a user, when the user wants to identify the position of the femoral artery. Also, the first tissue structure being incorrect includes cases where the position or label of the identified first tissue structure is different from the facts in the first place. For example, although there is the great saphenous vein at the position indicated by the first tissue structure information S1, the label is the femoral artery. In this specification, assume that when the user wants to identify the position of the femoral artery, the position and label of the great saphenous vein are identified as the first tissue structure. The position indicated by the first tissue structure information S1 in FIG. 2 is the position of the great saphenous vein.

[0045] The first factor for the tissue structure identification unit 40 to identify an incorrect first tissue structure is that the specific cross-section identified by the cross-section type identification unit 38 is incorrect. As described above, the tissue structure identification unit 40 uses the second learning model 34 associated with the specific cross-section, that is, the second learning model 34 specialized in predicting the tissue structure included in the ultrasonic tomographic image of the specific cross-section, to identify the first tissue structure. Here, if the cross-section type identification unit 38 identifies a cross-section type different from the true cross-section type of the target image TI as the specific cross-section, the tissue structure identification unit 40 will use the second learning model 34 not specialized in the true cross-section type of the target image TI to identify the first tissue structure. For example, if the true cross-section type of the target image TI is the CFV cross-section, but the tissue structure identification unit 40 identifies the SFJ cross-section as the specific cross-section, the tissue structure identification unit 40 will use the second learning model 34 specialized in the SFJ cross-section to identify the first tissue structure from the target image TI of the CFV cross-section. In such a case, an incorrect first tissue structure may be identified.

[0046] The second factor that the tissue structure specifying unit 40 specifies an incorrect first tissue structure is that although the specific cross-section specified by the cross-section type specifying unit 38 is correct, the first second learning model 34 is not suitable for the target image TI. As described above, when a plurality of second learning models 34 are associated with a specific cross-section, the tissue structure specifying unit 40 designates one of them as the first second learning model 34. In the case where the first second learning model 34 thus selected is not suitable for the target image TI. For example, when the brightness of the target image TI is too low for the first second learning model 34 to predict the correct tissue structure, an incorrect first tissue structure may be specified.

[0047] The user checks the first tissue structure information S1 displayed on the display 24 and confirms that the first tissue structure is incorrect. The user who has confirmed the error of the first tissue structure inputs an instruction from the input interface 26 indicating the position on the target image TI of the target tissue structure. This instruction is also an instruction to notify the tissue structure specifying unit 40 that the first tissue structure is incorrect, together with the position on the target image TI of the tissue structure expected by the user. In the present embodiment, the user inputs this instruction by aligning the cursor C that moves on the display 24 in response to the operation of the input interface 26 to the target position and clicking or the like. Here, it is assumed that the user has instructed the position of the femoral artery, which is the position of the cursor C shown in FIG. 2.

[0048] In response to the instruction from the user, the tissue structure specifying unit 40 inputs the target image TI to one or more second learning models 34 other than the first second learning model 34, including the second learning models 34 associated with cross-section types other than the specific cross-section specified by the cross-section type specifying unit 38. In this specification, one or more second learning models 34 other than the first second learning model 34 into which the target image TI is input here are referred to as other second learning models 34. The tissue structure specifying unit 40 specifies the tissue structure included in the vicinity of the position instructed by the user of the target image TI based on the prediction results of the other second learning models 34. In this specification, the tissue structure specified by the tissue structure specifying unit 40 based on the output of the other second learning models 34 is referred to as the second tissue structure.

[0049] Specifically, the tissue structure specifying unit 40 designates, as the second tissue structure, the tissue structure detected at the position closest to the position specified by the user among the tissue structures predicted by one or more other second learning models 34. Alternatively, the tissue structure specifying unit 40 may define a certain range centered on the position specified by the user, and designate, as the second tissue structure, the tissue structure detected within the range. In this case, when a plurality of tissue structures are detected by a plurality of other second learning models 34 within the range, the tissue structure having the label most frequently detected among the labels of the plurality of tissue structures may be designated as the second tissue structure. For example, within the range, if three other second learning models 34 detect a tissue structure having the label "femoral artery" and one other second learning model 34 detects a tissue structure having "thrombus", the tissue structure specifying unit 40 designates one of the tissue structures having the label "femoral artery" as the second tissue structure.

[0050] Further, the tissue structure specifying unit 40 may specify the second tissue structure by cutting out a certain range centered on the position specified by the user from the target image TI and inputting the cut-out image to another second learning model 34.

[0051] The specified second tissue structure may be the correct tissue structure (i.e., the position and label of the first tissue structure match the facts and it is the tissue structure expected by the user).

[0052] First, consider the case where the tissue structure specifying unit 40 specifies an incorrect first tissue structure because the specific cross-section specified by the cross-section type specifying unit 38 is incorrect. The other second learning models 34 include second learning models 34 associated with cross-section types other than the specific cross-section. That is, the other second learning models 34 include a second learning model 34 corresponding to the true cross-section type of the target image TI. Therefore, the tissue structure specifying unit 40 can specify the second tissue structure, which is the correct tissue structure, based on the output of another second learning model 34 corresponding to the true cross-section type of the target image TI.

[0053] Next, consider a case where the specific cross-section identified by the cross-section type identification unit 38 is correct, but the first second learning model 34 is not suitable for the target image TI, and the tissue structure identification unit 40 identifies an incorrect first tissue structure. Other second learning models 34 include second learning models 34 other than the first second learning model 34 associated with the specific cross-section. Such second learning models 34 may include a second learning model 34 suitable for the target image TI. For example, in order for the first second learning model 34 to predict the correct tissue structure, even if the luminance of the target image TI is too low, if preprocessing such as correcting (increasing) the luminance is performed by other second learning models 34 associated with the specific cross-section, the tissue structure identification unit 40 can identify the second tissue structure, which is the correct tissue structure, based on the output of the other second learning models 34.

[0054] The display control unit 22 causes the display 24 to display second tissue structure information, which is information regarding the second tissue structure identified by the tissue structure identification unit 40. FIG. 3 is a diagram showing an example of the display of the second tissue structure information S2. In the present embodiment, the display control unit 22 causes the display 24 to display the second tissue structure information S2 together with the target image TI and the cross-section information CS. The second tissue structure information S2 includes an icon indicating the position of the first tissue structure on the target image TI, or an image icon or character indicating a label, like the first tissue structure information S1. In the example of FIG. 3, as the second tissue structure information S2, an icon indicating the position of the second tissue structure on the target image TI is displayed superimposed on the target image TI. In addition to this, characters indicating the label of the second tissue structure may be displayed. Also, as shown in FIG. 3, the display control unit 22 may display the second tissue structure information S2 together with the first tissue structure information S1. In that case, the icon as the second tissue structure information S2 may have a different shape and color from the icon as the first tissue structure information S1.

[0055] Before displaying the second tissue structure information S2 on the display 24 or before performing prediction processing using another second learning model 34, the display control unit 22 may display a notification for confirmation to the user on the display 24. For example, a dialog or the like may be displayed in the form of a pop-up.

[0056] When the second tissue structure indicated by the second tissue structure information S2 is the correct tissue structure, the user can specify the second tissue structure (this specification also notifies the ultrasonic diagnostic apparatus 10 that the second tissue structure is the correct tissue structure) and proceed to subsequent processing, for example, measurement processing related to the second tissue structure.

[0057] Thus, according to the present embodiment, when the first tissue structure is incorrect, in response to a user's instruction, another second learning model 34 including the second learning model 34 associated with a cross-sectional type other than the specific cross-section is used to identify the second tissue structure, so that the identification accuracy of the tissue structure can be improved.

[0058] Consider a case where the tissue structure identification unit 40 has identified an incorrect first tissue structure because the specific cross-section identified by the cross-section type identification unit 38 is incorrect, and the tissue structure identification unit 40 has identified a second tissue structure based on the output of another second learning model 34 associated with a cross-sectional type other than the specific cross-section, and the second tissue structure is the correct tissue structure. In this case, since the correct tissue structure can be identified based on the output of the other second learning model 34, it is highly likely that the other second learning model 34 is a model specialized in predicting the tissue structure included in the ultrasonic tomographic image having the true cross-sectional type of the target image TI. That is, it can be said that the cross-sectional type associated with the other second learning model 34 is highly likely to be the true cross-sectional type of the target image TI.

[0059] Therefore, when the specific cross-section identified by the cross-section type identification unit 38 and the cross-section type associated with another second learning model 34 that contributed to the identification of the correct second tissue structure are different from each other, instead of the cross-section information CS (see FIG. 3) indicating the specific cross-section, as shown in FIG. 4, the display 24 may be caused to display corrected cross-section information CS' indicating the cross-section type corresponding to the other second learning model 34 that contributed to the identification of the second tissue structure. Thereby, even when the cross-section type identification unit 38 identifies an incorrect specific cross-section, the correct cross-section type of the target image TI can be notified to the user.

[0060] In the present embodiment, when the first tissue structure is incorrect, the tissue structure identification unit 40 performs the identification process of the second tissue structure using another second learning model 34. Here, a large number of second learning models 34 may be stored in the memory 30. In that case, the number of other second learning models 34 to which the target image TI should be input also becomes large, and the processing amount for identifying the second tissue structure may become enormous.

[0061] In consideration of the problem, it is advisable to group a plurality of second learning models 34 in advance corresponding to each part of the subject. For example, a plurality of second learning models 34 may be grouped into a plurality of groups such as a group corresponding to the abdomen and a group corresponding to the lower limbs. Then, the tissue structure identification unit 40 inputs the target image TI to other second learning models 34 belonging to the same group as the first second learning model 34 among the plurality of second learning models 34 other than the first second learning model 34 in response to an instruction from the user who has confirmed the first tissue structure information S1, and does not input the target image TI to the second learning models 34 belonging to a group different from the first second learning model 34. Then, the tissue structure identification unit 40 may identify the second tissue structure included in the target image TI based on the prediction results of the other second learning models 34 belonging to the same group as the first second learning model 34.

[0062] This is based on the fact that even if the cross-section type specifying unit 38 specifies an incorrect specific cross-section, the possibility of specifying a cross-section different from the true cross-section type and part of the target image TI as the specific part is quite low. For example, when the true cross-section type of the target image TI is the CFV cross-section of the lower limb, even if the cross-section type specifying unit 38 specifies the cross-section of the target image TI as the SFJ cross-section of the lower limb, the possibility of specifying it as the cross-section of the abdomen is quite low. When the cross-section type specifying unit 38 erroneously specifies the cross-section of the target image TI as the SFJ cross-section of the lower limb, the first second learning model 34 will be associated with the SFJ cross-section of the lower limb. And in this case, the tissue structure specifying unit 40 inputs the target image TI to other second learning models 34 corresponding to the lower limb (including other second learning models 34 associated with the CFV cross-section which is the true cross-section type) according to the instruction from the user. Thereby, without using the second learning model 34 belonging to a group other than the group corresponding to the lower limb, only other second learning models 34 belonging to the group corresponding to the lower limb can be used to specify the second tissue structure which is the correct tissue structure (in this case, the second tissue structure which is the correct tissue structure can be specified by other second learning models 34 associated with the CFV cross-section). That is, the processing amount for specifying the second tissue structure can be reduced without reducing the possibility of specifying the second tissue structure which is the correct tissue structure.

[0063] In the above embodiment, the tissue structure specifying unit 40 uses, for example, the tissue structure detected at the position closest to the position specified by the user as the second tissue structure, but the tissue structure specifying unit 40 may present a plurality of candidates for the second tissue structure to the user and let the user select the second tissue structure.

[0064] Specifically, first, the tissue structure specifying unit 40 inputs the target image TI to each of a plurality of other second learning models 34 other than the first second learning model 34 according to the instruction from the user who has confirmed the first tissue structure information S1 (see FIG. 2). Thereby, the positions and labels of a plurality of tissue structures are predicted by the plurality of other second learning models 34.

[0065] Next, the organizational structure identification unit 40 calculates the order of prediction accuracies for each of the labels of the plurality of organizational structures predicted by the plurality of other second learning models 34. For example, the organizational structure identification unit 40 arranges the predicted organizational structures in ascending order of proximity to the position specified by the user, and arranges the labels of each organizational structure in the same order without duplication (for example, when there are the same labels, use the one with the higher order and do not use the other labels), and sets the order as the order of prediction accuracies of the labels. Alternatively, a certain range centered on the position specified by the user may be defined, and the order of the number detected within the range may be set as the order of prediction accuracies of the labels.

[0066] The display control unit 22 causes the display 24 to display the labels of the plurality of organizational structures predicted by each of the plurality of other second learning models 34 in a display mode in which the calculated order of prediction accuracies is expressed. FIG. 5 is a diagram showing an example of the display of the labels of the plurality of organizational structures. In the example of FIG. 5, the levels of the plurality of predicted organizational structures are displayed in the form of a label list L. In the label list L, the plurality of labels are displayed so as to be arranged vertically, and the label displayed at the top has the highest prediction accuracy, indicating that the order of prediction accuracies decreases as going downward. Of course, the label list L is merely an example, and the display control unit 22 may display the plurality of labels in any display mode as long as the order of prediction accuracies is expressed.

[0067] The user checks the plurality of labels displayed in the display mode in which the order of prediction accuracies is expressed, and selects one label while referring to the prediction accuracies.

[0068] The tissue structure specifying unit 40 specifies, as a second tissue structure, the tissue structure related to the label selected by the user among the plurality of tissue structures predicted by each of the plurality of other second learning models 34. In the case where there are a plurality of tissue structures related to the label selected by the user, the tissue structure specifying unit 40 selects one tissue structure from among the plurality of tissue structures related to the selected label and sets this as the second tissue structure. For example, the tissue structure specifying unit 40 sets, as the second tissue structure, the tissue structure located at the position closest to the position specified by the user among the plurality of tissue structures related to the selected label.

[0069] Details of the processing of the image quality adjustment unit 20 will be described. The image quality adjustment information 36 stored in the memory 30 is information in which the cross-sectional type of the ultrasonic tomographic image and the image quality adjustment parameters for adjusting the image quality of the ultrasonic tomographic image of the cross-sectional type are associated with each other. The image quality adjustment unit 20 adjusts the image quality of the target image TI based on the cross-sectional type of the target image TI and the image quality adjustment information 36.

[0070] In particular, consider the case where the tissue structure specifying unit 40 specifies an incorrect first tissue structure due to the specific cross-section specified by the cross-section type specifying unit 38 being incorrect, and the tissue structure specifying unit 40 specifies a second tissue structure based on the output of another second learning model 34 associated with a cross-sectional type other than the specific cross-section, and the second tissue structure is the correct tissue structure. That is, consider the case where the specific cross-section and the cross-sectional type associated with another second learning model 34 that contributed to the specification of the second tissue structure are different from each other.

[0071] In this case, the image quality adjustment unit 20 may adjust the image quality of the target image TI using the image quality adjustment parameters associated with the cross-sectional type corresponding to another second learning model 34 that contributed to the specification of the second tissue structure. Thereby, the image quality of the target image TI can be adjusted using the image quality adjustment parameters suitable for the true cross-sectional type of the target image TI.

[0072] The above is the outline of the configuration of the ultrasonic diagnostic apparatus 10 according to this embodiment. Hereinafter, the basic processing flow of the ultrasonic diagnostic apparatus 10 will be described according to the flowchart shown in FIG. 6. It is assumed that the first learning model 32 and the second learning model 34 have been learned at the start point of the flowchart shown in FIG. 6.

[0073] In step S10, the image forming unit 18 forms a target image TI, which is an ultrasonic tomographic image, based on the received beam signal that has been signal-processed by the signal processing unit 16.

[0074] In step S12, the cross-section type specifying unit 38 inputs the target image TI formed in step S10 into the first learning model 32, and specifies a specific cross-section, which is the cross-section type of the target image TI, based on the prediction result of the first learning model 32 for the target image TI.

[0075] In step S14, the tissue structure specifying unit 40 inputs the target image TI into the first second learning model 34 associated with the specific cross-section specified in step S12, and specifies a first tissue structure included in the target image TI based on the prediction result of the first second learning model 34 for the target image TI.

[0076] In step S16, the display control unit 22 causes the display 24 to display first tissue structure information regarding the first tissue structure specified in step S14.

[0077] In step S18, the tissue structure specifying unit 40 determines whether or not an instruction indicating a position on the target image TI has been received from the user who has confirmed the first tissue structure information displayed on the display 24 in step S16. If the instruction has not been received, it means that the first tissue structure is not incorrect (it is the tissue structure expected by the user), so the processing of this flowchart is terminated. Thereafter, the user proceeds to measurement processing or the like regarding the first tissue structure. If the instruction has been received, the process proceeds to step S20.

[0078] In step S20, the tissue structure identification unit 40 inputs the target image TI to other second learning models 34 other than the first second learning model 34, and based on the prediction results of the other second learning models 34 for the target image TI, identifies the second tissue structure included in the target image TI.

[0079] In step S22, the display control unit 22 causes the display 24 to display the second tissue structure information regarding the second tissue structure identified in step S20.

[0080] After that, if the second tissue structure is the correct tissue structure, the user selects the second tissue structure and then proceeds to measurement processing or the like regarding the second tissue structure.

[0081] As described above, the ultrasonic tomographic image processing apparatus according to the present disclosure has been described. However, the ultrasonic tomographic image processing apparatus according to the present disclosure is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit of the present disclosure.

[0082] For example, in the above-described embodiment, the ultrasonic tomographic image processing apparatus is the ultrasonic diagnostic apparatus 10. However, the ultrasonic tomographic image processing apparatus is not limited to the ultrasonic diagnostic apparatus 10. For example, the ultrasonic tomographic image processing apparatus may be a personal computer (PC (Personal Computer)) or a server. In this case, the PC or server as the ultrasonic tomographic image processing apparatus has a display control unit 22, a display 24, a memory 30 in which the learned first learning model 32 and the learned second learning model 34 are stored, a cross-sectional type identification unit 38, and a tissue structure identification unit 40. The PC or server as the ultrasonic tomographic image processing apparatus acquires the target image TI from the ultrasonic diagnostic apparatus and performs the processing according to the above-described embodiment on the acquired target image TI.

[0083] In the above-described embodiment, the first learning model 32 and the second learning model 34 are stored in the memory 30 of the ultrasonic tomography imaging apparatus. However, as long as the first learning model 32 and the second learning model 34 are accessible from the ultrasonic tomography imaging apparatus, they may be stored in a device separate from the ultrasonic tomography imaging apparatus.

Description of Reference Numerals

[0084] 10 Ultrasonic diagnostic apparatus, 12 Ultrasonic probe, 14 Transmission / reception unit, 16 Signal processing unit, 18 Image formation unit, 20 Image quality adjustment unit, 22 Display control unit, 24 Display, 26 Input interface, 28 Control unit, 30 Memory, 32 First learning model, 34 Second learning model, 36 Image quality adjustment information, 38 Cross-section type specifying unit, 40 Tissue structure specifying unit.

Claims

1. A first learning model trained to predict and output a cross-sectional type of an input ultrasonic tomographic image, and a plurality of second learning models each associated with a respective cross-sectional type of an ultrasonic tomographic image, each of the second learning models being trained to predict and output a tissue structure included in an ultrasonic tomographic image of the corresponding cross-sectional type, the plurality of second learning models, An ultrasonic tomographic image processing apparatus accessible to, A cross-sectional type specifying unit that specifies a specific cross-section that is the cross-sectional type of the target image by inputting a target image, which is an ultrasonic tomographic image to be processed, into the first learning model, A tissue structure specifying unit that specifies a first tissue structure included in the target image by inputting the target image into a first second learning model that is the second learning model associated with the specific cross-section, A display control unit that causes a display unit to display first tissue structure information that is information about the first tissue structure, Comprising, The tissue structure specifying unit inputs the target image into the second learning models other than the first second learning model, including the second learning models associated with cross-sectional types other than the specific cross-section, in response to an instruction from a user who has confirmed the first tissue structure information and indicates a position on the target image, and based on the prediction results of the second learning models, specifies a second tissue structure included in the vicinity of the position indicated by the user on the target image. An ultrasonic tomographic image processing apparatus characterized by the above.

2. The display control unit causes the display unit to display cross-section information indicating the specific cross-section, When the specific cross-section and the cross-sectional type associated with the second learning model that contributed to the specification of the second tissue structure are different from each other, the display control unit causes the display unit to display, instead of the cross-section information indicating the specific cross-section, modified cross-section information indicating the cross-sectional type corresponding to the second learning model that contributed to the specification of the second tissue structure. The ultrasonic tomographic image processing apparatus according to claim 1, characterized by the above.

3. The plurality of second learning models are grouped corresponding to each part of the subject, The tissue structure specifying unit inputs the target image to the second learning models belonging to the same group as the first second learning model among the plurality of second learning models other than the first second learning model in response to an instruction from a user who has confirmed the first tissue structure information, and further specifies the second tissue structure included in the target image based on the prediction result of the second learning model. The ultrasonic tomography image processing apparatus according to claim 1, characterized in that.

4. The tissue structure specifying unit inputs the target image to each of the plurality of second learning models other than the first second learning model in response to an instruction from a user who has confirmed the first tissue structure information, and calculates the order of prediction accuracy for each of the labels of the plurality of tissue structures predicted by each of the plurality of second learning models. The display control unit causes the display unit to display the labels of the plurality of tissue structures predicted by each of the plurality of second learning models in a display mode in which the order of the prediction accuracy is expressed. The tissue structure specifying unit specifies, as the second tissue structure, the tissue structure related to the label selected by the user among the plurality of tissue structures predicted by each of the plurality of second learning models. The ultrasonic tomography image processing apparatus according to claim 1, characterized in that.

5. A image quality adjustment unit that adjusts the image quality of the ultrasonic tomography image based on image quality adjustment information in which the cross-sectional type of the ultrasonic tomography image and the image quality adjustment parameters for adjusting the image quality of the ultrasonic tomography image of the cross-sectional type are associated with each other, wherein when the specific cross-section and the cross-sectional type associated with the second learning model that contributed to the specification of the second tissue structure are different from each other, the image quality of the target image is adjusted using the image quality adjustment parameters associated with the cross-sectional type corresponding to the second learning model that contributed to the specification of the second tissue structure. The ultrasonic tomography image processing apparatus according to any one of claims 1 to 4, further comprising.

6. A first learning model trained to predict and output the cross-sectional type of the input ultrasonic tomography image, A plurality of second learning models each associated with a respective cross-sectional type of the ultrasonic tomography image, each of the second learning models being trained to predict and output the tissue structures included in the ultrasonic tomography image of the corresponding cross-sectional type. A computer accessible to A cross-section type specifying unit that specifies a specific cross-section, which is the cross-section type of the target image, by inputting the target image, which is an ultrasonic tomographic image to be processed, into the first learning model; A tissue structure specifying unit that specifies a first tissue structure included in the target image by inputting the target image into the first second learning model, which is the second learning model associated with the specific cross-section; A display control unit that causes a display unit to display first tissue structure information, which is information about the first tissue structure; Function as; The tissue structure specifying unit inputs the target image into the second learning model other than the first second learning model, including the second learning model associated with a cross-section type other than the specific cross-section, in response to an instruction from a user who has confirmed the first tissue structure information and indicates a position on the target image, and specifies a second tissue structure included in the vicinity of the position indicated by the user on the target image based on the prediction result of the second learning model; An ultrasonic tomographic image processing program characterized by the above.

Images