Method for producing learned model, ultrasonic diagnostic device, and device for producing learned model

Abstract

To efficiently produce a learned model that specifies at least one of the type and position of an organ included in input ultrasonic data.SOLUTION: The present disclosure relates to a method for producing a learned model that specifies at least one of the type and position of an organ included in input ultrasonic data. The method receives at least one of the type of an organ and the position of the organ included in the ultrasonic data, executes machine learning using at least one of the type of the organ and the position of the organ and the ultrasonic data as teacher data and produces a learned model (460). The ultrasonic data substantially includes only one corresponding to one or multiple specific cross-sections. Each of the one or multiple specific cross-sections is associated with a specific position and a specific direction of a subject.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to methods and apparatus for producing trained models using ultrasound data. [Background technology]

[0002] Previously, attempts have been made to identify the location of organs in ultrasound images using mathematical models such as convolutional neural networks (CNN) (e.g., WO2017210690A1, “Comparison and verification of high-speed organ detection methods in ultrasound images using CNN,” Igarashi Tatsuki et al., Transactions of the Society of Instrument and Control Engineers, Vol. 56, No. 12, December 2020, pp. 560-569).

[0003] An ultrasound diagnostic device that collects such ultrasound images can emit ultrasound waves in any direction from a probe placed at any position on the body surface of a subject, and can collect ultrasound data (echo signals, sound ray signals) based on the ultrasound waves to depict cross sections. Therefore, ultrasound diagnostic devices have the advantage of having a higher degree of freedom in depicting cross sections compared to other modalities such as MR and CT.

[0004] In addition, in many other modalities such as MR and CT, the reconstructed slices are often associated with position information such as slice position, but in most cases, in ultrasound diagnostic devices, the position information is not recorded on the reconstructed slices. For this reason, during or after the examination, it may be difficult for beginners or third parties to determine whether the appropriate organs have been recorded with appropriate image quality. In addition, it is not realistic to record all images obtained in real time, considering the limitations of recording media and the effort required for interpretation.

[0005] For this reason, there is a demand for the use of trained artificial intelligence models that can identify the type and / or location of organs in ultrasound images with high accuracy.However, ultrasound images generated by ultrasound diagnostic devices have a feature that they have a higher degree of freedom in the cross-sections they depict than images generated by other modalities such as MR and CT.Therefore, it is not easy to prepare training data for training an artificial intelligence model. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2018-079000 A [Patent Document 2] WO2017210690A1 publication [Non-Patent Document 1] "Comparison and Verification of High-Speed ​​Organ Detection Methods on Ultrasound Images Using CNN" Tatsuki Igarashi et al., Transactions of the Society of Instrument and Control Engineers, Vol. 56, No. 12, December 2020, P560-569) Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, what is needed is a low-cost method for producing a trained model that identifies at least one of the type and location of an organ contained in input ultrasound data, and a simple method for more efficiently training an artificial intelligence model to identify at least one of the type and location of an organ. [Means for solving the problem]

[0008] A method for producing a trained model for identifying at least one of the type and position of an organ included in input ultrasound data, according to a first embodiment of the present invention, includes: receiving the ultrasound data; receiving at least one of a type of organ and a position of the organ included in the ultrasound data; (1) performing machine learning using at least one of the type of organ and the position of the organ and (2) the ultrasound data as training data to produce the trained model; Including, the ultrasound data substantially only includes data corresponding to one or more particular slices; Each of the one or more specific slices is associated with a specific position and a specific direction of the subject.

[0009] In a second embodiment of the present invention, The ultrasound data includes raw data before image processing or image data after image processing; The method according to the first embodiment is provided, wherein the one or more specific cross sections are cross sections of representative positions that cover a necessary and sufficient range for screening a specific site.

[0010] In a third embodiment of the present invention, making the trained model available to one or more ultrasound diagnostic devices; performing automatic or manual re-training of the trained model; Including, The step of automatically retraining the trained model includes: A step of automatically detecting the type and position of an organ included in new ultrasound data corresponding to the one or more specific cross sections newly collected by the one or more ultrasound diagnostic devices using the trained model; A step of updating the trained model by re-learning based on the new ultrasound data and the type and position of the organ corresponding to the new ultrasound data; Including, The step of manually re-learning the trained model includes: receiving a user input including a type and a position of an organ included in new ultrasound data corresponding to the one or more specific cross sections newly acquired by the one or more ultrasound diagnostic devices; A step of updating the trained model by re-learning based on the new ultrasound data and the type and position of the organ corresponding to the new ultrasound data; The method according to the first or second embodiment is provided, comprising:

[0011] In a fourth embodiment of the present invention, The step of receiving at least one of a type and a position of an organ included in the ultrasound data includes a step of automatically or manually receiving the type and the position of the organ included in the ultrasound data; The step of automatically receiving the type and the location of the organ contained in the ultrasound data includes: applying pattern recognition to an ultrasound image generated based on the ultrasound data to identify the type and location of the organ; The step of manually receiving the type and the location of the organ contained in the ultrasound data includes: displaying an ultrasound image generated based on the ultrasound data on a monitor; receiving coordinate information identifying a geometric figure positioned by the operator with respect to the organ included in the ultrasound image, the geometric figure being displayed on the monitor; receiving organ type information that identifies the organ inputted in association with the geometric figure; The method according to the first to third embodiments includes the steps of:

[0012] In a fifth embodiment of the present invention, accessing a database storing the ultrasound data and / or the ultrasound images via a communications connection; receiving the ultrasound data and / or the ultrasound image from the database; Including, the database is in communication with a plurality of ultrasound diagnostic devices that acquire the ultrasound data and / or the ultrasound images; Each of the plurality of ultrasonic diagnostic devices is configured to be programmed with a predetermined examination procedure in advance, so that acquisition parameters corresponding to the one or more specific cross sections are automatically set and an operator is prompted to examine the plurality of different specific cross sections in a predetermined order; A method according to the first to fourth embodiments is provided, in which each of the plurality of ultrasound diagnostic devices is configured to tag and store the ultrasound data and / or the ultrasound image with at least one of the type of organ and the position of the organ corresponding to each of the plurality of different specific cross sections.

[0013] In a sixth embodiment of the present invention, The training data includes at least one of a shape of the organ, information on a lesion included in the organ, and quality of the ultrasound data; When the trained model is deployed for use with one or more ultrasound diagnostic devices, detecting a shape of the organ included in an ultrasound image newly acquired by the one or more ultrasound diagnostic devices; Identifying the presence or absence of a pathology in the organ in the newly acquired ultrasound images by the one or more ultrasound diagnostic devices; assessing the quality of newly acquired ultrasound data from the one or more ultrasound diagnostic devices; The method according to the first to fifth embodiments is provided, which provides one or more of the following:

[0014] In a seventh embodiment of the present invention, An ultrasound diagnostic device having a trained model for identifying at least one of the type and position of an organ included in input ultrasound data, The training data for generating the trained model includes: An ultrasound image corresponding substantially only to one or more specific cross sections, and at least one of a type and a position of an organ included in the ultrasound image; An ultrasonic diagnostic apparatus is provided, in which each of the one or more specific cross sections is associated with a specific position and a specific direction of a subject.

[0015] In an eighth embodiment of the present invention, An apparatus for producing a trained model that identifies at least one of the type and position of an organ contained in input ultrasound data, an input port for receiving the ultrasound data; (1) a processor that performs machine learning using at least one of the type of organ and the position of the organ and (2) the ultrasound data as training data to produce the trained model; Including, the ultrasound data substantially only includes data corresponding to one or more particular slices; An apparatus is provided, wherein each of the one or more specific slices is associated with a particular location and a particular orientation of the subject.

[0016] In a ninth embodiment of the present invention, the processor is configured to automatically assess, or prompt an operator to assess, quality of the ultrasound data; If the quality of the ultrasound data does not reach a predetermined evaluation standard, the ultrasound data is excluded from the training data; The predetermined evaluation criteria are: deviation of the ultrasound data from the particular location; deviation of the ultrasound data from the particular direction; low image quality of an ultrasound image based on the ultrasound data; A user input instructing to exclude the ultrasound data from the training data; The description of the eighth embodiment is provided, wherein the description is any one or more of the following:

[0017] In a tenth embodiment of the present invention, the device is in communication with a database that stores the ultrasound data and / or the ultrasound images; the database is in communication with a plurality of ultrasound diagnostic devices that acquire the ultrasound data and / or the ultrasound images; Each of the plurality of ultrasonic diagnostic devices is configured to automatically set acquisition parameters corresponding to the one or more specific cross sections by programming a predetermined examination procedure in advance, and to prompt an operator to examine the plurality of different specific cross sections in a predetermined order; The description of the eighth or ninth embodiment is provided, in which each of the plurality of ultrasound diagnostic devices is configured to tag and store the ultrasound data and / or the ultrasound image with at least one of the type of organ and the position of the organ corresponding to each of the plurality of different specific cross sections.

[0018] In an eleventh embodiment of the present invention, The ultrasound data is collected in A mode, B mode, M mode or Doppler mode, as described in the eighth to tenth embodiments.

[0019] In a twelfth embodiment of the present invention, The processor, displaying on the monitor names of a plurality of organs corresponding to the cross-section specifying information in a prioritized order; Switching the model of the training target according to the cross-section identification information; The eighth to eleventh embodiments are configured to perform any one of the above. Effect of the Invention

[0020] According to the above embodiment of the invention, a trained model that identifies at least one of the type and position of an organ contained in input ultrasound data can be produced at low cost. [Brief description of the drawings]

[0021] [Figure 1] 1 is a block diagram showing an outline of the overall configuration of a system according to an embodiment of the present invention. [Diagram 2] 2 is a block diagram showing the configuration of an ultrasonic diagnostic device that constitutes the system shown in FIG. 1. [Diagram 3]FIG. 1 is a blinking diagram showing a trained model production system. [Figure 4] FIG. 13 is a diagram showing how annotations are added and the type and position of organs are recognized by a deep learning model. [Diagram 5] FIG. 1 illustrates an exemplary embodiment of a trainer and modeler. [Figure 6] FIG. 1 illustrates an exemplary implementation of a deep learning neural network. [Figure 7] FIG. 1 illustrates an exemplary implementation of an image analysis convolutional neural network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. A system 100 shown in Fig. 1 includes a plurality of ultrasound diagnostic devices 101, 102, and 103 and a server 104. The plurality of ultrasound diagnostic devices 101, 102, and 103 are communicatively connected to each other via a network 105. Each of the plurality of ultrasound diagnostic devices 101, 102, and 103 is also communicatively connected to the server 104 via the network 105. A trained model production terminal 107 that produces a trained model is also communicatively connected to the server 104 via the network 105.

[0023] The configuration of each of the ultrasound diagnostic devices 101, 102, and 103 is shown in FIG. 2 as an ultrasound diagnostic device 200. The ultrasound diagnostic device 200 includes a transmission beamformer 203 that drives a plurality of transducer elements 202a arranged in an ultrasound probe 202 to generate a pulsed ultrasound signal, and a transmitter 204 that radiates the generated pulsed ultrasound signal to a subject (not shown). The pulsed ultrasound signal generates an echo that is reflected in the subject and returns to the transducer elements 202a. The echo is converted into an electric signal by the transducer elements 202a, and the electric signal is received by a receiver 205. The electric signal representing the received echo, i.e., the echo signal, is amplified by a required gain in the receiver 205 and then input to a reception beamformer 206, where reception beamforming is performed. The reception beamformer 206 outputs ultrasound data after reception beamforming.

[0024] The receive beamformer 206 may be a hardware beamformer or a software beamformer. When the receive beamformer 206 is a software beamformer, the receive beamformer 206 may include one or more processors 207 including any one or more of a graphics processing unit (GPU), a microprocessor, a central processing unit (CPU), a digital signal processor (DSP), or other types of processors capable of performing logical operations. The processor constituting the receive beamformer 206 may be a processor separate from the processor 207 described below, or may be composed of the processor 207. The echo signal before the receive beamforming and the ultrasound data after the receive beamforming are stored in the memory 209.

[0025] The processor 207 can process the data in real-time during a scanning session as the echo signals are received. For purposes of this disclosure, the term "real-time" is defined to include procedures that occur without any intentional delay.

[0026] Additionally, data may be temporarily stored in a buffer (not shown) during an ultrasound scan and processed non-real-time for live or offline operation. In this disclosure, the term "data" may be used to refer to one or more data sets acquired using an ultrasound imaging device.

[0027] The ultrasound data may be processed by the processor 207 in other or different mode-related modules (e.g., B-mode, color Doppler, M-mode, color M-mode, spectral Doppler, contrast mode, elastography, TVI, strain, strain rate, etc.) to produce data for an ultrasound image. For example, one or more modules may generate an ultrasound image such as B-mode, color Doppler, M-mode, color M-mode, spectral Doppler, contrast mode, elastography, TVI, strain, strain rate, and combinations thereof.

[0028] A video processor module may be provided to read image frames from the memory and display the image frames in real time while a procedure is being performed on the subject. The video processor module may store the image frames in the image memory, and the ultrasound images are read from the image memory and displayed on the display 208.

[0029] As used herein, the term "image" broadly refers to both a visible image and data representing a visible image, and the term "data" may include raw data, which is ultrasound data (echo signals or sound ray signals) before a scan conversion operation, and image data, which is data after a scan conversion operation.

[0030] When processor 207 includes multiple processors, the processing tasks described above for processor 207 may be performed by the multiple processors. For example, a first processor may be used to demodulate and decimate the RF signals, and a second processor may be used to further process the data before displaying an image. Also, for example, when receive beamformer 206 is a software beamformer, its processing functions may be performed by a single processor or multiple processors.

[0031] The display 208 is a light emitting diode (LED) display, a liquid crystal display (LCD), a micro LED display, an organic electro-luminescence (EL) display, or the like.

[0032] The memory 209 is any known data storage medium. In one example, the ultrasound image display device 1 includes a plurality of memories 209, including a non-transient storage medium and a transient storage medium as the memory 209. The non-transient storage medium is, for example, a non-volatile storage medium such as a hard disk drive (HDD) or a read only memory (ROM). The non-transient storage medium may include a portable storage medium such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray Disc (registered trademark). The non-transient storage medium stores a program executed by the processor 207. The non-transient storage medium also stores a protocol described below. The transient storage medium can be a volatile storage medium such as a random access memory (RAM). All of these may be stored in the same memory 209, or at least one of them may be stored in a different memory 209. The memory 209 may also be a plurality of data storage media deployed on the cloud.

[0033] The user interface 210 can accept input from an operator. For example, the user interface 210 accepts input of instructions and information from an operator. The user interface 210 is configured to include a keyboard, hard keys, soft keys, and the like. The user interface 210 may include various input devices such as pointing devices such as a mouse, a touch panel, a pen tablet, a touch pad, a trackball, and a joystick, as well as eye tracking and voice input.

[0034] The speaker 211 outputs sound under the control of the processor 207. In one example, the speaker 211 outputs sound based on a signal input from the processor 207.

[0035] 3 shows a trained model production system 300. The trained model production system 300 includes an ultrasound diagnostic device 310, a data store 320, a model trainer 330, a modeler 340, an output processor 350, a feedback unit 360, a selector 370, and an annotation adding unit 380. Each functional block can be executed by one or more processors. In a specific example, the data store 320 is disposed in a database of the server 104, and the data store 320, the model trainer 330, the modeler 340, the output processor 350, the feedback unit 360, the selector 370, and the annotation adding unit 380 are disposed in the trained model production terminal 107. In another specific example, the trained model production terminal 107 is not required, and the data store 320, the model trainer 330, the modeler 340, the output processor 350, the feedback unit 360, the selector 370, and the annotation adding unit 380 are arranged in the server 104. In yet another specific example, the data store 320, the model trainer 330, the modeler 340, the output processor 350, the feedback unit 360, the selector 370, and the annotation adding unit 380 are arranged in one or more of the ultrasound diagnostic devices 101, 102, and 103. The data store 320, the data store 320, the model trainer 330, the modeler 340, the output processor 350, the feedback unit 360, the selector 370, and the annotation adding unit 380 can also be processed by multiple processing devices distributed over a network. The ultrasound data is stored in a data store 320 (e.g., a database, a data structure, a hard drive, solid state memory, flash memory, other computer memory, etc.), which may consist of multiple storage devices distributed across a network.

[0036] The selector 370 receives the images and / or echo signals from the data store 320, determines whether they correspond to one or more predetermined specific slices, and tags them to correspond to one or more predetermined specific slices. The training network 420 (FIG. 5) and / or trained model 460 (FIG. 5) to be used are selected according to the tag. In a particular embodiment, the selector 370 receives the images and / or echo signals from the data store 320, determines whether they correspond to one or more predetermined specific slices, and excludes images and / or echo signals that do not correspond to one or more predetermined specific slices from processing and are not tagged. In yet another embodiment, the images and / or echo signals stored in the data store 320 are associated with slice identification information, and in such an embodiment, the selector 370 is not required. The selector 370 is configured to automatically evaluate the quality of the images and / or echo signals from the data store 320 or to prompt the operator to evaluate the quality, and excludes the images and / or echo signals from the training data if the quality of the images and / or echo signals does not meet a predetermined evaluation standard. The predetermined evaluation criteria may be one or more of deviation from a specific position, deviation from a specific direction, low image quality of the ultrasound image, and user input instructing exclusion from the training data.

[0037] In a preferred embodiment of the present invention, the one or more specific slices are slices at representative positions that cover a necessary and sufficient range for screening a specific site such as the kidney or pancreas. In a preferred embodiment of the present invention, the slices correspond to one or more of the 25 slices recommended for recording defined by the Japanese Society of Digestive Cancer Screening. The one or more specific slices may be a plurality of different specific slices acquired with different acquisition parameters, or may be a plurality of different specific slices acquired with the same acquisition parameters.

[0038] In a preferred embodiment of the present invention, the selector 370 receives the image and / or echo signal from the data store 320, detects low-quality images such as insufficient contact, shadows, inappropriate brightness (too low and / or too high), low contrast, blurred images, and excludes them from processing. In a specific embodiment, when the selector 370 detects low-quality images, it only annotates the images according to the type of cause of low quality, but does not exclude them from processing. In a specific embodiment, when the selector 370 detects low-quality images, it excludes images with quality below a predetermined level from processing, and annotates images with low but acceptable quality according to the type of cause of low quality, but does not exclude them from processing.

[0039] In recent years, deep learning techniques have made it possible to detect objects in images and classify images. Learning requires a large number of images and accompanying information (annotations). Ultrasound images can be obtained in real time, making it easy to collect large amounts of data, but as mentioned above, the freedom of the cross-sections is so high that it is difficult to collect and annotate the images required for learning without bias.

[0040] In the present invention, in order to train a deep learning model, only ultrasound images of one or more specific cross sections associated with a specific position and a specific direction are adopted as learning data, which makes it possible to efficiently detect the type and position of organs and learn image quality classification.

[0041] 4, the annotation adding unit 380 displays an image from the data store 320 on the display device of the trained model production terminal 107. However, if it is unclear from what position and in what direction the displayed image 401 was collected on the subject, it is not necessarily clear what organs are included in the image, and therefore the operator of the trained model production terminal 107 cannot easily add annotations to the image.

[0042] On the other hand, the Japanese Society of Digestive Cancer Screening has determined 25 types of recommended recording sections (ultrasonic B-mode images), and the position and direction in which the operator places the ultrasonic probe 202 on the subject are determined for each of the sections. In such a case, for example, the kidney and liver may be noted in the 25th section out of the 25 recommended recording sections, while the approximate position of the kidney in the B-mode image 402 is known. The operator of the trained model production terminal 107 can easily add annotations to the image. Similarly, the pancreas may be noted in the 8th section out of the 25 recommended recording sections, while the approximate position of the pancreas in the B-mode image 403 is known. The operator of the trained model production terminal 107 can easily add annotations to the image. In the example of FIG. 4, the organ name is added as an annotation, but the position, shape, image quality, etc. of the organ can also be added as an annotation. Also, the type of cause of image quality degradation can be manually annotated, such as occurrence of insufficient contact, presence of shadows, inappropriate brightness (too low brightness and / or too high brightness), low contrast, presence of image blur, etc. Furthermore, the operator can use the annotation adding unit 380 to delete images in order to exclude undesirable images from processing.

[0043] The specific cross sections associated with the specific positions and specific directions of the subject, as represented by the 25 types of recommended recording cross sections, and the annotations therefor are ideal teacher data for training the deep learning model. In the example of FIG. 4, the deep learning model is trained by adding annotations 404 indicating that the organ names are "kidney" and "liver" to the specific cross section image 402. When the specific cross section image of another subject is analyzed using this trained deep learning model, a display 406 indicating that the kidney and liver are included and position information 407, 408 of each organ are displayed. In another example, the deep learning model is trained by adding annotation 405 indicating that the organ name is "pancreas" to the specific cross section image 403. When the specific cross section image of another subject is analyzed using this trained deep learning model, a display 406 indicating that the pancreas is included and position information 409 of each organ are displayed. In this embodiment, a rectangular geometric figure is used as position information indicating the position of an organ, but various geometric figures can be used, such as a circle, an ellipse, a polygon, or a complex curve drawn by edge detection.

[0044] Manual annotation can be performed in a variety of ways. In certain embodiments of the invention, the operator locates the organ by surrounding the desired organ with a geometric shape such as a circle, oval, polygon, etc. In other embodiments, the operator locates the organ by selecting the center of the organ. Once the organ is located, a pull-down menu appears on the display prompting the operator to identify the organ type. The operator selects the organ type from the pull-down menu, completing the annotation of the organ type.

[0045] As described above, the annotation adding unit 380 provides the operator with a function of manually adding annotations. However, in another preferred embodiment of the present invention, the annotation adding unit 380 can also provide a function of automatically adding annotations. The annotation adding unit 380 applies pattern recognition to an ultrasound image generated based on ultrasound data to identify the type and position of a recognized organ, and adds a corresponding annotation. In a specific embodiment, the annotation adding unit 380 applies pattern recognition to an ultrasound image generated based on ultrasound data to identify the type and position of a recognized organ, and presents the corresponding annotation to the operator so that the annotation can be corrected. The annotation adding unit 380 can automatically annotate not only the type and position of an organ, but also the type of cause of image quality degradation, such as the shape of the organ, good or bad image quality, occurrence of insufficient contact, the presence of a shadow, inappropriate brightness (too low brightness and / or too high brightness), low contrast, and the presence of blur in the image.

[0046] Although the function of the annotation adding unit 380 has been described, in a specific embodiment of the present invention, the annotation adding unit 380 is not necessary. In an embodiment in which the annotation adding unit 380 is not necessary, special software for navigating the acquisition of a specific cross section is installed in each of the ultrasound diagnostic devices 101 to 103. For example, there are a plurality of examination procedures in which the examination procedures are generally fixed, such as ultrasound examinations in medical checkups.

[0047] The navigation software installed in each of the ultrasound diagnostic devices 101 to 103 navigates the collection of a plurality of specific cross sections included in the selected examination procedure in a predetermined order determined in consideration of examination efficiency, etc., in response to the selection of an examination procedure by the operator (technologist). This navigation software is not an essential component in certain embodiments of the present invention. In the case of ultrasound examinations for medical checkups as described above, the navigation software can be replaced by an in-house manual describing in what order what cross sections are to be examined and images to be saved, an agreement between operators verbally or in writing on paper or electronic media, instructions from a supervisor, etc. In the case of ultrasound examinations for medical checkups as described above, the navigation software can be replaced by an in-house manual describing in what order what cross sections are to be examined and images to be saved, an agreement between operators verbally or in-house e-mail, instructions from a supervisor, etc. The navigation software displays on the display 208 the position and angle at which the operator should contact the ultrasound probe on the subject and the posture the subject should be instructed to take. Such instructions can also be issued from the speaker 211. The navigation software checks whether the collected images are collected at the specified position and angle, and if not, displays a warning on the display 208 (or outputs a voice from the speaker 211). The navigation software also checks the quality of the collected images, and displays a warning on the display 208 according to the type of problem. The warning may be that the ultrasonic probe 202 is not insufficiently contacting the subject, or that the brightness adjustment is inappropriate. If the collected images are collected at the specified position and angle and there is no problem with the quality of the images, the navigation software recommends the operator to save the images. When the operator gives an instruction to save the images accordingly, the navigation software adds specific cross-section identification information that identifies the collected specific cross-section to the images as an annotation. In a preferred embodiment of the present invention, the specific cross-section identification information is a number corresponding to the number of the examination procedure and the number of the cross-section in the examination procedure.In another preferred embodiment, the specific cross-section identification information is a number associated with a specific position and a specific direction that are common to a plurality of examination procedures. In yet another embodiment, the specific cross-section identification information is information on the type and approximate position of an organ included in the cross-section. In addition to automatic annotation by the navigation software, the operator of the ultrasound diagnostic device 101-103 can perform manual annotation. For example, when either the gallbladder or the kidney has been completely removed from the subject, or when a part of the liver or the pancreas has been removed, the operator can manually add an annotation corresponding to the case. By following the procedure of such navigation software, annotation can be added without providing an annotation adding unit 380. Such manual annotation can also be a learning target.

[0048] In a preferred embodiment of the present invention, the navigation software allows the acquisition of multiple specific cross sections that are not included in the selected examination procedure. For example, when a lesion is found during an examination according to the selected examination procedure, the operator may wish to observe the condition and characteristics of the lesion in detail and save the images. In such a case, the operator can instruct the navigation software to interrupt the normal examination procedure and collect and save the necessary images. The navigation software adds an annotation indicating that the image was saved during the interruption of the normal procedure. The selector 370 excludes the image with the annotation indicating that the image was saved during the interruption of the normal procedure from the processing target.

[0049] In a preferred embodiment of the present invention, the annotation adding unit 380 can provide the operator with a function of manually adding annotations to one or more specific cross sections collected using navigation software. In this case, the same procedure can be followed as for one or more specific cross sections collected without using navigation software. In a specific embodiment, the annotation adding unit 380 can provide a pull-down menu of organ names corresponding to one or more specific cross sections based on the specific cross section identification information added as annotations to the one or more specific cross sections. For example, since the hepatic vein and the diaphragm are often the focus of attention in the 21st cross section of the 25 types of recommended recording cross sections, a pull-down menu is provided in which the hepatic vein and the diaphragm are placed at the top. Also, a pull-down menu is provided in which the pancreas is placed at the top of the 3rd cross section of the 25 types of recommended recording cross sections. In a specific embodiment of the present invention, the examination data of multiple subjects is accumulated and sorted based on the specific cross section identification information. The operator who manually adds annotations can process a large number of cross section images that have the same position and direction at the same time. This can improve the efficiency and accuracy of the work.

[0050] Returning to FIG. 3, the explanation continues. The trainer 330 and the modeler 340 receive image and / or echo signals from the data store 320 directly or via a selector 370. The trainer 330 trains and tests deep learning networks and / or other machine learning networks (e.g., convolutional neural networks, recurrent neural networks, random forests, etc.) that are deployed as models in the modeler 340. The trained network models are used by the modeler 340 to analyze image data from the data store 320. As learning models, YOLO for object detection, ResNet for classification, etc., which are conventionally used by those skilled in the art, may also be used.

[0051] Modeler 340 uses a deployed artificial intelligence model (trained model, e.g., neural network, random forest, etc.) to process the incoming ultrasound data, identify organ types and / or locations of the organs in the ultrasound data, and generate a composite image that incorporates an indication of the organ types and / or locations of the organs in the ultrasound data into the ultrasound image. The composite image generated by modeler 340 is provided to output processor 350 for generating an image output for display, storage, processing by another system (e.g., for computer-aided diagnosis to detect lesions, tumors, etc.), communication with another device (e.g., tablet or smartphone, etc.), etc.

[0052] Feedback from the image output can be captured by feedback unit 360 and provided to trainer 330, which can use the feedback to improve the artificial intelligence model. The feedback can be used to train and / or test the model, and can be used to periodically trigger regeneration by trainer 330 of the model that is deployed to modeler 340.

[0053] In a preferred embodiment of the present invention, the trained model 460 is transmitted from the trained model production terminal 107 to each of the ultrasound diagnostic devices 101 to 103. In each of the ultrasound diagnostic devices 101 to 103, automatic re-learning or manual re-learning, or both, of the trained model 460 is performed.

[0054] In the automatic re-learning of the trained model 460, first, one or more specific cross sections are newly acquired by each of the ultrasound diagnostic devices 101 to 103, and the trained model 460 is used to automatically detect the type and position of an organ contained in new ultrasound data corresponding to the newly acquired specific cross section. Next, the new ultrasound data and the type and position of an organ corresponding to the new ultrasound data are input to the trainer 330, and re-learning is performed to update the trained model 460. By such automatic re-learning, it is possible to provide an enhanced trained model 460 that takes into account differences in the shape, size, brightness, position, etc. of an organ that change according to individual differences of the subject.

[0055] In a specific embodiment, in the automatic re-learning of the trained model 460, first, each of the ultrasound diagnostic devices 101 to 103 newly collects one or more specific cross sections, and transmits the newly collected one or more specific cross sections to one or more locations where the trained model 460 is automatically re-learned. The one or more locations may be the server 104, the trained model production terminal 107, or other ultrasound diagnostic devices. Using the trained model 460 at one or more locations, the type and position of the organ contained in the new ultrasound data corresponding to the newly collected specific cross section are automatically detected. In a specific embodiment, if the specific cross section corresponds to one or more specific cross sections but has characteristics different from other specific cross sections, the data is transmitted to a different location determined according to the characteristics. At a location that receives a specific cross section with a special characteristic, the trained model 460 is automatically re-learned using the special specific cross section. As a result, even if the special characteristic is based on a lesion based on a rare case, a large number of ultrasound diagnostic devices collect a large number of images, and the trained model 460 after re-learning can be used to detect similar rare lesions, etc. contained in the newly input image. In addition, for example, the execution of re-learning can be triggered when an examiner adds information to a specific cross section during an examination using navigation software. When re-learning, the added information can also be used as a target. For example, if the added information is a possibility of a tumor, a trained model 460 specialized for tumors may be selected as a target for re-learning.

[0056] In certain embodiments of the present invention, if a predicted organ is not detected and / or an incorrect organ is detected in each cross-section of a predetermined examination procedure, it is automatically determined that the image and organ type are to be used for re-learning. In other embodiments, the examiner is allowed to select whether re-learning is necessary in this case. In certain embodiments of the present invention, only ultrasound data collected during an examination using the navigation software is used for training or re-learning the trained model 460. In other embodiments, if ultrasound data is not collected during an examination using the navigation software, but is automatically analyzed to be equivalent to ultrasound data collected during an examination using the navigation software, it is used for training or re-learning the trained model 460.

[0057] In a preferred embodiment of the present invention, the trained model 460 is transmitted from the trained model production terminal 107 to each of the ultrasound diagnostic devices 101 to 103. In each of the ultrasound diagnostic devices 101 to 103, manual re-learning of the trained model 460 is executed.

[0058] In manual re-learning of the trained model 460, first, one or more specific cross sections are newly collected by each of the ultrasound diagnostic devices 101 to 103, and the trained model 460 is used to automatically detect the type and position of an organ contained in new ultrasound data corresponding to the newly collected specific cross section. Next, at least one of the type of organ contained in the newly collected specific cross section and the position of the organ is displayed on the display 208 of the ultrasound diagnostic devices 101 to 103 by superimposing it on the ultrasound image of the newly collected specific cross section. The operator of the ultrasound diagnostic devices 101 to 103 can correct at least one of the type of organ and the position of the organ via the user interface 210 as necessary. When manually specifying the position of an organ, a rectangular geometric figure can be used as position information indicating the position of the organ. Information that can specify the specified rectangle (for example, the coordinates of the upper left and the coordinates of the lower right) is added as an annotation to the specific cross section by the operator. However, various geometric figures can be used to specify the position of an organ, such as a circle, an ellipse, a polygon, or a complex curve drawn by edge detection.

[0059] The model trainer 330 arranged in the ultrasound diagnostic devices 101-103 that have newly collected one or more specific cross sections re-trains the training network 420 using at least one of the organ type and the organ position corrected by the operator and the newly collected one or more specific cross sections as teacher data, and updates the trained model 460. By such manual re-learning, it is possible to provide an enhanced trained model 460 that can display positions in a way that is easy for the operator to understand.

[0060] In the above-mentioned embodiment, the manual re-learning of the trained model 460 is performed in the ultrasound diagnostic devices 101-103 that newly collected one or more specific cross sections, but it can be performed instead of or in addition to this at other locations. In a specific embodiment, in the manual re-learning of the trained model 460, first, each of the ultrasound diagnostic devices 101-103 newly collects one or more specific cross sections, and transmits the newly collected one or more specific cross sections to one or more locations where the trained model 460 is manually re-learned. The one or more locations may be the server 104, the trained model production terminal 107, or other ultrasound diagnostic devices. Using the trained model 460 at one or more locations, the type of organ included in the new ultrasound data corresponding to the newly collected specific cross section and its position are automatically detected. At least one of the type of organ included in the newly collected specific cross section and the position of the organ is displayed on the display 208 at one or more locations so as to be superimposed on the ultrasound image of the newly collected specific cross section. An operator at one or more locations can modify at least one of the type of organ and the position of the organ via the user interface as necessary.

[0061] The model trainer 330 arranged at one or more locations retrains the training network 420 using at least one of the type of organ and the position of the organ corrected by the operator and one or more newly collected specific cross sections as teacher data, and updates the trained model 460. By such manual retraining, it is possible to provide an enhanced trained model 460 capable of displaying positions in a way that is easy for the operator to understand. According to such an embodiment, since the retraining can be performed by concentrating the work of adding annotations on an operator skilled in adding annotations, it is possible to produce a training network 420 with high detection accuracy and accurate position identification.

[0062] FIG. 5 illustrates an exemplary implementation of the trainer 330 and the modeler 340. As shown in the example of FIG. 5, the trainer 330 includes an input processor 410, a training network 420, an output validator 430, and a feedback processor 440. The exemplary modeler 340 includes a pre-processor 450, a trained model 460, which is a deployed model, and a post-processor 470. In the example of FIG. 5, inputs such as images and / or echo signals received directly from the data store 320 or via a selector 370 are provided to the input processor 410, which selects a training network 420 from among multiple training networks 420 corresponding to the input, and prepares the data to be input to the training network 420. For example, the data can be filtered, supplemented, and / or modified in a manner such as sanitizing to make the images and / or echo signals input to the network 420 more suitable for learning. In certain embodiments, the trainer 330 does not have an input processor 410.

[0063] A typical training network 420 analyzes input data from the input processor 410 and generates an output, which is verified by the output validator 430. Thus, the training network 420 receives input image data and develops connections to dynamically form a learning algorithm that identifies the type and location of organs contained in the image. The output validator 430 can, for example, verify the accuracy of the output from the training network 420. If the output of the network 420 is not accurate, the network 420 can be modified (e.g., by adjusting the network weights, changing the node connections, etc.) to, for example, update the network 420 and generate a correct output. Once the output of the training network 420 is verified by the output validator 430, the training network 420 can be used to generate and deploy a learned model 460 for the modeler 340.

[0064] Periodically, feedback can be provided to a feedback processor 440, which processes the feedback and evaluates whether to trigger an update or regeneration of the network 420. For example, if the output of the deployed trained model 460 continues to be accurate, there may be no reason to perform an update. However, for example, if the output of the trained model 460 becomes inaccurate, a regeneration or other update of the network 420 can be triggered and an updated trained model 460 can be deployed (e.g., based on additional data, new constraints, updated configurations, etc.).

[0065] The developed trained model 460 is used by the modeler 340 to process input ultrasound data and identify organ types and locations. Input data, such as from the data store 320, is prepared by the pre-processor 450, which then provides the input data to a trained model 460 (e.g., a deep learning network model, a machine learning model, another network model, etc.) selected by the pre-processor 450 in response to the input data. The pre-processor 450 can adjust the contrast, brightness, levels, artifacts / noise, etc. of the input image data before the image data is provided to the trained model 460. The trained model 460 processes the ultrasound data from the pre-processor 450 and identifies organ types and locations. Output from the trained model 460 is post-processed by the post-processor 470, which can clean up, organize, and / or otherwise modify the output data to form a composite 2D image. In certain examples, the post-processor 470 may verify and / or otherwise perform quality checks on the output image data before the composite image is stored, displayed, transmitted to another system, etc.

[0066] 6 shows a typical deep learning neural network 500. Typical neural network 500 includes layers 520, 540, 560, and 580. Layers 520 and 540 are connected by neural connection 530. Layers 540 and 560 are connected by neural connection 550. Layers 560 and 580 are connected by neural connection 570. Data flows forward from the input layer 520 via inputs 512, 514, 516 to the output layer 580 and output 590.

[0067] Layer 520 is an input layer that includes multiple nodes 522, 524, 526 in the example of FIG. 6. Layers 540 and 560 are hidden layers that include nodes 542, 544, 546, 548, 562, 564, 566, 568 in the example of FIG. 6. Neural network 500 may include more or fewer hidden layers 540 and 560 than are shown. Layer 580 is an output layer that includes node 582 having output 590 in the example of FIG. 6. Each input 512-516 corresponds to a node 522-526 in input layer 520, and each node 522-526 in input layer 520 has a connection 530 to a respective node 542-548 in hidden layer 540. Each node 542-548 in hidden layer 540 has a connection 550 to a respective node 562-568 in hidden layer 560. Each node 562-568 in hidden layer 560 has a connection 570 to output layer 580. Output layer 580 has an output 590 that provides the output from the exemplary neural network 500.

[0068] Among the connections 530, 550, and 570, certain exemplary connections 532, 552, 572 may be given additional weighting, while other exemplary connections 534, 554, 574 may be given lighter weighting in the neural network 500. The input nodes 522-526 are activated, for example, by receiving input data via the inputs 512-516. The nodes 542-548 and 562-568 in the hidden layers 540 and 560 are activated by the forward flow of data in the network 500 via the connections 530 and 550, respectively. The node 582 in the output layer 580 is activated after the data processed in the hidden layers 540 and 560 is sent via the connection 570. When the output node 582 in the output layer 580 is activated, the node 582 outputs an appropriate value based on the processing accomplished in the hidden layers 540 and 560 of the neural network 500. In this way, a model can be created by using the images, the names of organs contained in the images, and their location information for learning object detection, shape detection (segmentation), and classification.

[0069] 7 shows an exemplary embodiment of an image analysis convolutional neural network 700. The convolutional neural network 700 receives input image data 702 and abstracts the image data in a convolutional layer 704 to identify learned features 710-722. In a second convolutional layer 730, the image data is transformed into a plurality of images 730-738 in which each of the learned features 710-722 is enhanced in a respective sub-image 730-738. The images 730-738 are further processed to focus on the features 710-722 of interest in images 740-748. The resulting images 740-748 are then processed through a pooling layer that reduces the size of the images 740-748 to isolate portions 750-754 of the images 740-748 that contain the features 710-722 of interest. The outputs 750-754 of the convolutional neural network 700 receive values ​​from the last non-output layer and classify the image based on the data received from the last non-output layer. Such classification and its shape identify the type and location of the corresponding organ and its shape. Similarly, the type and location of a lesion of a given shape and its shape can be identified. Furthermore, the convolutional neural network 700 can evaluate the quality of newly acquired ultrasound data. In certain examples, the convolutional neural network 700 can include many different variations of convolutional layers, pooling layers, learned features, and outputs.

[0070] Although the present invention has been described above with a focus on the most preferred embodiment, it will be apparent to those skilled in the art that the present invention can be implemented by making various changes and modifications to the embodiment within the technical scope of the present invention. [Explanation of symbols]

[0071] 100 Systems 101, 102, 103, 200, 310 Ultrasound diagnostic equipment 104 Server 105 Network 107 Trained model production terminal 202 Ultrasonic Probe 202a Vibration element 203 Transmit Beamformer 204 Transmitter 205 Receiver 206 Receive Beamformer 207 Processor 208 Display 209 Memory 210 User Interface 211 Speaker 300 Trained Model Production System 320 Data Store 330 Model Trainer 340 Modeler 350 Output Processor 360 Feedback Unit 370 Selector 380 Annotation Addition Section 410 Input Processor 420 Training Network 430 Output Validator 440 Feedback Processor 450 Preprocessor 460 trained models 470 Post Processor

Claims

1. A method for producing a trained model for identifying at least one of a type and a position of an organ contained in input ultrasound data, comprising: receiving the ultrasound data; receiving at least one of a type of organ and a position of the organ included in the ultrasound data; (1) performing machine learning using at least one of the type of organ and the position of the organ and (2) the ultrasound data as training data to produce the trained model; Including, the ultrasound data substantially only includes data corresponding to one or more particular slices; Each of the one or more specific cross sections is associated with a particular position and a particular orientation of the subject.

2. The ultrasound data includes raw data before image processing or image data after image processing; The method according to claim 1 , wherein the one or more specific cross sections are cross sections of representative positions that cover a necessary and sufficient range for screening a specific site.

3. making the trained model available to one or more ultrasound diagnostic devices; performing automatic or manual re-training of the trained model; Including, The step of automatically retraining the trained model includes: Using the trained model, automatically detecting the type and position of an organ included in new ultrasound data corresponding to the one or more specific cross sections newly collected by the one or more ultrasound diagnostic devices; A step of updating the trained model by re-learning based on the new ultrasound data and the type and position of the organ corresponding to the new ultrasound data; Including, The step of manually re-learning the trained model includes: receiving a user input including a type and a position of an organ included in new ultrasound data corresponding to the one or more specific cross sections newly acquired by the one or more ultrasound diagnostic devices; A step of updating the trained model by re-learning based on the new ultrasound data and the type and position of the organ corresponding to the new ultrasound data; The method of claim 1 , comprising:

4. The step of receiving at least one of a type and a position of an organ included in the ultrasound data includes a step of automatically or manually receiving the type and the position of the organ included in the ultrasound data; The step of automatically receiving the type and the location of the organ contained in the ultrasound data includes: applying pattern recognition to an ultrasound image generated based on the ultrasound data to identify the type and location of the organ; The step of manually receiving the type and the location of the organ contained in the ultrasound data includes: displaying an ultrasound image generated based on the ultrasound data on a monitor; receiving coordinate information identifying a geometric figure positioned by the operator with respect to the organ included in the ultrasound image, the geometric figure being displayed on the monitor; receiving organ type information that identifies the organ inputted in association with the geometric figure; The method of claim 1 , comprising:

5. accessing a database storing the ultrasound data and / or the ultrasound images via a communications connection; receiving the ultrasound data and / or the ultrasound image from the database; Including, the database is in communication with a plurality of ultrasound diagnostic devices that acquire the ultrasound data and / or the ultrasound images; Each of the plurality of ultrasonic diagnostic devices is configured to be programmed with a predetermined examination procedure in advance, so that acquisition parameters corresponding to the one or more specific cross sections are automatically set and an operator is prompted to examine the plurality of different specific cross sections in a predetermined order; 2. The method according to claim 1, wherein each of the plurality of ultrasound diagnostic devices is configured to tag and store the ultrasound data and / or the ultrasound image with at least one of the type of organ and the position of the organ corresponding to each of the plurality of different specific cross sections.

6. The training data includes at least one of a shape of the organ, information on a lesion included in the organ, and quality of the ultrasound data; When the trained model is arranged for use by one or more ultrasound diagnostic devices, Detection of the shape of the organ included in the ultrasound images newly acquired by the one or more ultrasound diagnostic devices; Identifying the presence or absence of a lesion in the organ contained in the newly acquired ultrasound images by the one or more ultrasound diagnostic devices; assessing the quality of newly acquired ultrasound data from the one or more ultrasound diagnostic devices; The method of claim 1 , further comprising providing one or more of:

7. An ultrasound diagnostic device having a trained model for identifying at least one of the type and position of an organ included in input ultrasound data, The training data for generating the trained model includes: An ultrasound image corresponding substantially only to one or more specific cross sections, and at least one of a type and a position of an organ included in the ultrasound image, An ultrasound diagnostic apparatus, wherein each of the one or more specific cross sections is associated with a specific position and a specific direction of a subject.

8. An apparatus for producing a trained model that identifies at least one of the type and position of an organ contained in input ultrasound data, an input port for receiving the ultrasound data; (1) a processor that performs machine learning using at least one of the type of organ and the position of the organ, and (2) the ultrasound data as training data to produce the trained model; Including, the ultrasound data substantially only includes data corresponding to one or more particular slices; An apparatus, wherein each of the one or more specific cross sections is associated with a specific position and a specific direction of the subject.

9. the processor is configured to automatically assess, or prompt an operator to assess, quality of the ultrasound data; If the quality of the ultrasound data does not reach a predetermined evaluation standard, the ultrasound data is excluded from the training data; The predetermined evaluation criteria are: deviation of the ultrasound data from the particular location; deviation of the ultrasound data from the particular direction; low image quality of an ultrasound image based on the ultrasound data; A user input instructing to exclude the ultrasound data from the training data; The apparatus of claim 8, wherein the

10. the device is in communication with a database storing the ultrasound data and / or the ultrasound images; the database is in communication with a plurality of ultrasound diagnostic devices that acquire the ultrasound data and / or the ultrasound images; Each of the plurality of ultrasonic diagnostic devices is configured to automatically set acquisition parameters corresponding to the one or more specific cross sections by programming a predetermined examination procedure in advance, and to prompt an operator to examine the plurality of different specific cross sections in a predetermined order; 9. The device according to claim 8, wherein each of the plurality of ultrasound diagnostic devices is configured to tag and store the ultrasound data and / or the ultrasound image with at least one of the type of organ and the position of the organ corresponding to each of the plurality of different specific cross sections.

11. The apparatus of claim 10 , wherein the ultrasound data is collected in A-mode, B-mode, M-mode or Doppler mode.

12. The processor, displaying on the monitor names of a plurality of organs corresponding to the cross-section specifying information in a prioritized order; Switching the model of the training target according to the cross-section identification information; The apparatus of claim 10 , configured to perform any of the following:

Images