Ultrasound diagnostic device and ultrasound diagnostic system

Abstract

The invention provides an ultrasonic diagnostic apparatus and an ultrasonic diagnostic system. The present invention addresses the problem of implementing a mechanism in which an internal analysis model is replaced with an external analysis model. An evaluation unit (37) evaluates the performance of an analysis model (internal analysis model) (36) on the basis of internal data extracted from the analysis model (internal analysis model) (36) and output data of the analysis model (36). The determination unit (48) determines the use of the external analysis model (56) on the basis of the evaluation result of the evaluation unit (37). In this case, an input image (tomographic image) is transmitted to an information processing device (54) via a communication unit (50). The analysis result of the analysis model (external analysis model) (56) is transmitted to the calculation control unit (30) via the communication unit (50).

Description

Technical Field

[0001] The present invention relates to an ultrasonic diagnostic device and an ultrasonic diagnostic system, and more particularly to a technology for selectively using a plurality of analysis models. Background Art

[0002] Ultrasonic diagnostic devices are used in ultrasonic examinations. Among the ultrasonic diagnostic devices, there are devices equipped with an analysis model (a model generated through a machine learning process) for analyzing ultrasonic images. As such analysis models, there are known models for identifying the type of tomographic images, models for detecting lesions in tomographic images, and models for analyzing lesions in tomographic images. The analysis model is, for example, built on a CNN (Convolutional Neural Network).

[0003] Patent Document 1 describes an ultrasonic diagnostic apparatus including an analysis model for identifying the type of a tomographic image. Patent Document 2 describes an ultrasonic diagnostic apparatus including a plurality of analysis models.

[0004] In addition, a technique for generating a class activation map based on internal data extracted from a CNN is described in Non-Patent Documents 1 and 2. A class activation map is a map showing the response distribution of a CNN, and is also called a saliency map (or heat map).

[0005] Patent Document 1: Japanese Patent No. 6423540

[0006] Patent Document 2: Japanese Patent Application Publication No. 2020-204970

[0007] Non-patent literature 1: Zhou et al., Learning Deep Features for Difficult Localization, arxiv 2015, 1512.04150.

[0008] Non-patent literature 2: Selvaraju et al., Grad-CAM: Visual Explanations from DeepNetworks via Gradient-based Localization, arxiv 2016, 1610.02391.

[0009] Due to various structural or operational limitations, a high-performance analysis model is sometimes not provided inside the ultrasonic diagnostic device. Alternatively, it is also possible to consider providing an analysis model with higher performance after the ultrasonic diagnostic device equipped with the analysis model is used. In a situation where the analysis model provided inside the ultrasonic diagnostic device (internal analysis model) does not function sufficiently, it is desirable to implement a mechanism for temporarily or auxiliary use of an analysis model provided outside the ultrasonic diagnostic device (external analysis model).

[0010] Furthermore, Patent Documents 1 and 2 do not describe a configuration for using an analysis model provided outside the ultrasonic diagnostic apparatus. Summary of the invention

[0011] An object of the present invention is to provide an ultrasonic diagnostic apparatus having a mechanism for selectively using an internal analysis model and an external analysis model. Alternatively, an object of the present invention is to provide an ultrasonic diagnostic apparatus having a function of using an external analysis model when the use of the internal analysis model is inappropriate.

[0012] The ultrasonic diagnostic device involved in the present invention is an ultrasonic diagnostic device, which is used in ultrasonic examination of a subject, and is characterized in that it includes: a generator, which generates an ultrasonic image based on data obtained from the subject; an internal analysis model, which is arranged inside the ultrasonic diagnostic device and analyzes the ultrasonic image; an evaluator, which evaluates the internal analysis model; a determiner, which determines the use of an external analysis model arranged outside the ultrasonic diagnostic device based on the evaluation result of the evaluator; and a communicator, which transmits the ultrasonic image to the external analysis model and receives the analysis result of the external analysis model when the use of the external analysis model is determined.

[0013] The ultrasonic diagnostic system involved in the present invention is characterized in that it includes: an ultrasonic diagnostic device, which is used in ultrasonic examination of a subject; and an information processing device, which is connected to the ultrasonic diagnostic device via a network, the information processing device includes an external analysis model, and the external analysis model analyzes the ultrasonic image outside the ultrasonic diagnostic device, and the ultrasonic diagnostic device includes: a generator, which generates the ultrasonic image based on data obtained from the subject; an internal analysis model, which analyzes the ultrasonic image inside the ultrasonic diagnostic device; an evaluator, which evaluates the internal analysis model; a determiner, which determines the use of the external analysis model based on the evaluation result of the evaluator; and a communicator, which, when it is determined that the external analysis model is used, transmits the ultrasonic image to the external analysis model and receives the analysis result of the external analysis model.

[0014] Effects of the Invention

[0015] According to the present invention, the internal analysis model and the external analysis model can be selectively used. Alternatively, according to the present invention, when the use of the internal analysis model is inappropriate, the external analysis model can be used. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a block diagram showing an ultrasonic diagnostic system according to an embodiment.

[0017] Figure 2 It is a figure which shows a display example.

[0018] Figure 3 It is a diagram showing the difference between the internal analysis model and the external analysis model.

[0019] Figure 4 This is a diagram showing an example of an input image evaluation method.

[0020] Figure 5 is a diagram showing an example of a saliency map generation method.

[0021] Figure 6 This is a diagram showing an example of a saliency map evaluation method.

[0022] Figure 7 is a diagram showing another example of a saliency map evaluation method.

[0023] Figure 8 It is a diagram showing the first example of the confidence history and an example of the confidence history evaluation method.

[0024] Fig. 9 This is a diagram showing the second example of the confidence history.

[0025] Fig.10 This is a diagram showing the third example of the confidence history.

[0026] Fig.11 This is a diagram showing another example of the confidence history evaluation method.

[0027] Fig.12 It is a diagram showing an operation example of the determination unit.

[0028] Fig.13 This is a block diagram showing an ultrasonic diagnostic system according to another embodiment.

[0029] Explanation of symbols

[0030] 10-ultrasonic diagnostic device, 12-processor, 26-image generating unit, 30-operation control unit, 34-image analyzing unit, 36-analysis model (internal analysis model), 38-input image evaluation unit, 40-image generating unit, 42-image evaluation unit, 44-history generating unit, 46-history evaluation unit, 48-determination unit (selection unit), 54-information processing device, 56-analysis model (external analysis model). DETAILED DESCRIPTION

[0031] Hereinafter, embodiments will be described based on the drawings.

[0032] (1) Overview of Implementation

[0033] The ultrasonic diagnostic apparatus involved in the embodiment is an ultrasonic diagnostic apparatus used in ultrasonic examination of a subject, and includes a generator, an internal analysis model, an evaluator, a determiner, and a communicator. The generator generates an ultrasonic image based on data obtained from the subject. The internal analysis model is provided inside the ultrasonic diagnostic apparatus and analyzes the ultrasonic image. The evaluator evaluates the internal analysis model. The determiner determines the use of an external analysis model provided outside the ultrasonic diagnostic apparatus based on the evaluation result of the evaluator. When the use of the external analysis model is determined, the communicator transmits the ultrasonic image to the external analysis model and receives the analysis result of the external analysis model.

[0034] According to the above configuration, when the use of the internal analysis model is inappropriate, the external analysis model is used instead of the internal analysis model. That is, the external analysis model can be used as needed while giving priority to the internal analysis model.

[0035] A part or all of the evaluation unit described later corresponds to the above-mentioned evaluator. The determination unit described later corresponds to the above-mentioned determiner. The communication unit described later corresponds to the above-mentioned communicator. Both the internal analysis model and the external analysis model are models generated through a machine learning process. Generally speaking, the performance of the external analysis model is higher than that of the internal analysis model.

[0036] In an embodiment, the evaluator evaluates the internal analysis model based on at least one of internal data extracted from the internal analysis model and output data output from the internal analysis model. The internal data is data reflecting the analysis process of the internal analysis model, and the output data is data indicating the analysis result of the internal analysis model. By referring to one or both of the internal data and the output data, it is possible to determine whether the internal analysis model fully functions in relation to the ultrasonic image input to the internal analysis model.

[0037] In the embodiment, the evaluator evaluates the internal analysis model based on both the internal data and the output data. According to this configuration, it is possible to comprehensively or more accurately determine whether the internal analysis model has fully functioned in relation to the ultrasonic image input to the internal analysis model.

[0038] In an embodiment, the evaluator includes a graph generator and a graph evaluator. The graph generator generates a saliency graph based on a plurality of feature graphs as internal data extracted from an internal analysis model. The graph evaluator evaluates the saliency graph. The determiner determines the use of an external analysis model based on the evaluation result of the saliency graph. The graph generator described below is equivalent to the graph generator, and the graph evaluator described below is equivalent to the graph evaluator.

[0039] The saliency map represents the response distribution, i.e., the activation distribution, of the internal analysis model. Based on the saliency map, the response degree or response mode of the internal analysis model can be evaluated. For example, multiple feature maps are taken from multiple convolutional layers, and the saliency map is generated by weighted addition of the multiple feature maps.

[0040] In an embodiment, the evaluator includes a history generator and a history evaluator. The history generator generates a confidence history representing time changes of confidences of multiple classes based on output data output from the internal analysis model. The history evaluator evaluates the confidence history. The determiner determines the use of the external analysis model based on the evaluation result of the confidence history.

[0041] Generally speaking, when the internal analysis model works effectively on the input image, only the confidence of a specific class becomes high. On the contrary, when the internal analysis model does not work effectively on the input image, the confidence of any class becomes low. Alternatively, the confidence of multiple classes becomes high. By referring to the confidence history, the performance of the internal analysis model and its time changes can be comprehensively evaluated. In an embodiment, the confidence history is composed of multiple confidence curve graphs corresponding to multiple classes. The multiple classes are multiple classification classes, for example, multiple cross-section types.

[0042] The ultrasonic diagnostic apparatus according to the embodiment includes an input image evaluator that evaluates an ultrasonic image input to an internal analysis model, that is, an input image. A determiner determines the use of an external analysis model based on the evaluation result of the input image. In the embodiment, when the input image satisfies an image quality condition and the evaluation result of the internal analysis model does not satisfy the model performance condition, the determiner determines the use of the external analysis model. According to this structure, the use of the external analysis model can be correctly determined.

[0043] The ultrasonic diagnostic device involved in the embodiment includes a controller that controls the action of the ultrasonic diagnostic device. Both the internal analysis model and the external analysis model are models for identifying cross sections. Before determining the use of the external analysis model, the controller controls the action of the ultrasonic diagnostic device based on the cross section identified by the internal analysis model. On the other hand, after determining the use of the external analysis model, the controller controls the action of the ultrasonic diagnostic device based on the cross section identified by the external analysis model. According to this structure, the action control based on the cross section type is optimized. For example, the action condition is switched based on the identified cross section, the measurement is performed, or the body mark is generated.

[0044] In an embodiment, the evaluator continues to evaluate the internal analysis model even after determining the use of the external analysis model. After determining the use of the external analysis model, the determiner determines to restart the use of the internal analysis model based on the evaluation result of the evaluator. Generally speaking, good responsiveness can be obtained by using the internal analysis model. According to the above structure, the internal analysis model can be used preferentially.

[0045] The ultrasonic diagnostic system involved in the embodiment includes an ultrasonic diagnostic device and an information processing device. The information processing device includes an external analysis model that analyzes ultrasonic images outside the ultrasonic diagnostic device. The ultrasonic diagnostic device includes a generator, an internal analysis model, an evaluator, a determiner, and a communicator. The generator generates an ultrasonic image based on data obtained from a subject. The internal analysis model analyzes the ultrasonic image inside the ultrasonic diagnostic device. The evaluator evaluates the internal analysis model. The determiner determines the use of the external analysis model based on the evaluation result of the evaluator. When the use of the external analysis model is determined, the communicator transmits the ultrasonic image to the external analysis model and receives the analysis result of the external analysis model.

[0046] (2) Details of implementation methods

[0047] exist Figure 1 1 shows an ultrasonic diagnostic apparatus 10 according to an embodiment of the present invention. The ultrasonic diagnostic apparatus 10 is a medical apparatus installed in a medical institution such as a hospital and used for ultrasonic examination of a subject. The ultrasonic diagnostic apparatus 10 includes a processor 12 that functions as an information processing unit. The processor 12 is, for example, a CPU that executes a program. Figure 1 In FIG. 1 , multiple functions performed by the processor 12 are represented by multiple blocks.

[0048] The ultrasonic probe 14 is a device that transmits ultrasonic waves into a living body and receives reflected waves from the living body. The ultrasonic probe 14 has a transducer array composed of a plurality of transducers. The transducer array forms an ultrasonic beam 16. The ultrasonic beam 16 is electronically scanned to form a beam scanning plane 18.

[0049] Electronic linear scanning and electronic sector scanning are known as electronic scanning methods for the ultrasonic beam 16. A two-dimensional transducer array may be provided as the transducer array. By two-dimensionally scanning the ultrasonic beam using the two-dimensional transducer array, volume data can be acquired from the three-dimensional space in the living body.

[0050] The transmission circuit 20 is a circuit that functions as a transmission beamformer, and outputs a plurality of transmission signals in parallel to the transducer array during transmission, thereby forming a transmission beam.

[0051] The receiving circuit 22 is a circuit that functions as a receiving beam former. When receiving, it applies phasing and addition to a plurality of receiving signals output in parallel from the transducer array, thereby generating receiving beam data. As the electronic scanning is repeated, a receiving frame data sequence is output from the receiving circuit 22. Each receiving frame data is composed of a plurality of receiving beam data arranged in the electronic scanning direction. Each receiving beam data is composed of a plurality of echo data arranged in the depth direction.

[0052] The data processing circuit 24 is a circuit for processing each received frame data and includes a detection circuit, a logarithmic conversion circuit, a filter, and the like.

[0053] The image generation unit 26 has a digital scan converter (DSC). The DSC has a coordinate conversion function, a pixel interpolation function, etc. The DSC generates a display frame data column based on the received frame data column. In the embodiment, the display frame data column is equivalent to a tomographic image (B-mode tomographic image) as a dynamic image. Each display frame data constituting the display frame data column is equivalent to a tomographic image as a static image. Ultrasonic images other than tomographic images (for example, blood flow images, elasticity images) can also be generated.

[0054] A cine memory 28 is provided at the subsequent stage of the image generation unit 26. The display frame data sequence is temporarily stored in the cine memory 28. The display frame data sequence read from the cine memory 28 is sent to the calculation control unit 30. The cine memory 28 has a ring buffer structure. The cine memory 28 is composed of, for example, a semiconductor memory.

[0055] The calculation control unit 30 has a function of controlling the operation of each component element constituting the ultrasonic diagnostic apparatus and a function of executing various calculations. In the illustrated structural example, the calculation control unit 30 also functions as a display processing unit. A display frame data column is sent from the calculation control unit 30 to the display 32. A tomographic image as a dynamic image is displayed on the display 32. Other ultrasonic images may also be displayed on the display 32. The display 32 is composed of an organic EL device, a liquid crystal display, etc. An operation panel not shown in the figure is connected to the calculation control unit 30.

[0056] When a dynamic image analysis is performed, the display frame data sequence read from the cine memory 28 is sent to the image analysis unit 34. When a still image analysis is performed, specific display frame data stored in the cine memory 28 is sent to the image analysis unit 34. For example, in real-time operation, the display frame data sequence is sent from the cine memory 28 to the image analysis unit 34, and in a frozen state where transmission and reception are stopped by a freeze operation, specific display frame data is sent from the cine memory 28 to the image analysis unit 34.

[0057] The specific display frame data is the display frame data corresponding to the still image displayed on the display 32. The user selects the specific display frame data in sequence as needed. The user can select either the dynamic image analysis or the still image analysis. Figure 1 In the illustrated configuration example, the cine memory 28 is provided between the image generation unit 26 and the calculation control unit 30, but the cine memory 28 may be provided between the data processing circuit 24 and the image generation unit 26. In this case, the received frame data sequence or specific received frame data is sent to the image analysis unit 34.

[0058] The image analysis unit 34 has an analysis model (internal analysis model) 36 which is AI (Artificial Intelligence). The analysis model 36 is a model generated through a machine learning process. In an embodiment, the analysis model 36 is built on a CNN (Convolutional Neural Network). The analysis model 36 involved in the embodiment is a model for identifying the type of tomographic image (i.e., the type of cross-section). As other analysis models, there can be cited a model for detecting the lesion contained in the tomographic image, a model for analyzing the lesion contained in the tomographic image, and the like. The analysis result of the analysis model 36, i.e., the output data, is sent to the operation control unit 30.

[0059] The calculation control unit 30 controls the operation of the ultrasonic diagnostic apparatus based on the cross-sectional type identified by the analysis model 36. For example, the operation control unit 30 controls the progress of the examination protocol based on the identified cross-sectional type, starts the measurement suitable for the identified cross-sectional type, or selects the body marker corresponding to the identified cross-sectional type.

[0060] The evaluation unit 37 is a module (evaluator) that evaluates whether the analysis model 36 accurately performs its function for each display frame data (input image) input to the analysis model 36. The evaluation unit 37 evaluates the performance of the analysis model 36 for each input image and / or for each series of input images. Specifically, the evaluation unit 37 includes a graph generation unit 40, a graph evaluation unit 42, a history generation unit 44, and a history evaluation unit 46. In the embodiment, the evaluation unit 37 further includes an input image evaluation unit 38.

[0061] The input image evaluation unit 38 is an evaluator for evaluating the quality of the input image. The evaluation result of the input image evaluation unit 38 is sent to the determination unit 48. Since it is inappropriate to evaluate the performance of the analysis model 36 when the quality of the input image is degraded, that is, in order to correctly evaluate the performance of the analysis model 36, the input image evaluation unit 38 is provided. The details of the input image evaluation method will be described later.

[0062] The graph generation unit 40 extracts the internal data generated or used in the analysis process of the analysis model 36 from the analysis model 36, and generates a saliency map based on the extracted internal data. More specifically, the internal data includes a plurality of feature maps generated by the action of a plurality of convolutional layers. The graph generation unit 40 generates a saliency map by weighted addition of a plurality of feature maps. The saliency map represents the response distribution of the analysis model, and is composed of a plurality of intensities (a plurality of activity levels) arranged in two dimensions.

[0063] The graph evaluation unit 42 evaluates whether the analysis of the input image is appropriate by evaluating the saliency map. For example, the saliency map is evaluated based on the number of high intensities or the distribution of high intensities in the saliency map. The graph evaluation unit 42 calculates the graph evaluation value. The details of the graph generation method and the graph evaluation method will be described later.

[0064] The history generating unit 44 generates a confidence history based on the output data of the analysis model 36. The output data of the analysis model 36 includes a plurality of confidences (confidence sets) corresponding to a plurality of classes (i.e., a plurality of identifiable cross-section types). The history generating unit 44 generates a confidence history based on a plurality of confidence sets sequentially generated by sequentially analyzing a plurality of input images. The confidence of each class can also be referred to as a class probability.

[0065] The history evaluation unit 46 evaluates the confidence history. By evaluating the confidence history, it is possible to determine whether the analysis of the input image is appropriate. The history evaluation unit 46 calculates the history evaluation value. The details of the confidence history generation method and the confidence history evaluation method will be described later. In addition, only the latest confidence set can be evaluated. That is, the evaluation value can also be calculated based on the latest confidence set.

[0066] The determination unit 48 is a determiner that determines whether to use an external analysis model instead of the analysis model (internal analysis model) 36 based on the evaluation result of the input image, the image evaluation value, and the history evaluation value. That is, the determination unit 48 has a function of selecting the internal analysis model or the external analysis model, and from this point of view, the determination unit 48 can be called a selection unit. The determination result of the determination unit 48 is sent to the calculation control unit 30 and the communication unit 50.

[0067] The graph evaluation value is an evaluation value generated by spatial evaluation. The history evaluation value is an evaluation value generated by temporal evaluation. By considering these two evaluation values ​​simultaneously, it is possible to accurately determine whether an external analysis model needs to be used. Of course, depending on the situation, only the graph evaluation value or only the history evaluation value may be considered. In the embodiment, since the input image is evaluated, it is possible to avoid erroneous model performance evaluation caused by a decrease in the quality of the input image.

[0068] The communication unit 50 is connected to an information processing device 54 via a network 52. The information processing device 54 has an analysis model (external analysis model) 56. Similar to the analysis model 36 in the ultrasonic diagnostic device 10, the analysis model 56 identifies the type of tomographic image generated by the ultrasonic diagnostic device 10 (section type).

[0069] When the communication unit 50 determines that an external analysis model is to be used, the input image is transmitted to the information processing device 54. In this case, a plurality of images constituting a dynamic image are transmitted in sequence, or a still image is transmitted. The analysis model 56 analyzes each transmitted image and outputs an analysis result. The analysis result is sent from the information processing device 54 to the operation control unit 30 via the communication unit 50. In the case of dynamic image analysis, a series of analysis results are sent to the operation control unit 30 in sequence. In the case of still image analysis, the analysis result of a specific still image is sent to the operation control unit 30. As described above, the operation control unit 30 controls the operation of the ultrasonic diagnostic apparatus based on the analysis result of the analysis model 36 or the analysis result of the analysis model 56.

[0070] exist Figure 2 2 shows a display example. An image 200 displayed on the display includes a tomographic image 202. The type of the tomographic image 202, that is, the type of the cross section, is identified by the internal analysis model or the external analysis model. The identified cross section type is displayed as text information 204.

[0071] The operation of the ultrasonic diagnostic apparatus is controlled according to the identified section type. This is described in detail. In the illustrated example, a process list 206 representing multiple processes constituting an inspection protocol (inspection steps) is included in the image 200. The process list 206 is composed of multiple rows 208 representing multiple processes, and each row 208 includes a header or an inspection column indicating an executed process. Through an internal analysis model or an external analysis model, the next process is started at the timing when a specific section type is identified. For example, processing conditions, display conditions, etc. corresponding to the next process are automatically set in the ultrasonic diagnostic apparatus.

[0072] The measurement of the tomographic image may be automatically performed based on the recognized cross-section type. For example, a series of processing such as detecting the contour of the tissue 214, setting two marks 212a and 212b on the contour, and measuring the distance between the two marks 212a and 212b may be automatically performed.

[0073] Alternatively, a model for performing tissue detection may be provided as an analysis model. In this case, a specific tissue (e.g., a lesion) 214 in the tomographic image is automatically detected. Figure 2 In FIG. 2 , a frame 216 surrounding a specific tissue 214 is displayed.

[0074] exist Figure 3 The differences between the internal analysis model and the external analysis model are summarized in. For example, the internal analysis model is a model suitable for a specific examination site (specific tissue). In contrast, the external analysis model is a model suitable for all examination sites (or multiple examination sites). From another perspective, the internal analysis model can identify a relatively small number of cross sections, while the external analysis model can identify a very large number of cross sections.

[0075] For example, the internal analysis model is a model generated through a machine learning process using a relatively small amount of learning data, and the external analysis model is a model generated through machine learning using a large amount of learning data. A plurality of learning data may be generated in a plurality of medical institutions.

[0076] Generally speaking, when an internal analysis model is used, good responsiveness can be obtained. On the other hand, when an external analysis model is used, sufficient responsiveness may not be obtained depending on the situation. In addition, depending on the network status, it is sometimes best to avoid sending and receiving large amounts of data. Therefore, in the embodiment, the internal analysis model is basically used preferentially, and when the internal analysis model cannot perform sufficient analysis, the external analysis model is used temporarily or as an auxiliary. For example, when the internal analysis model cannot perform sufficient analysis on a still image displayed in a frozen state, the still image may be analyzed by an external analysis model.

[0077] exist Figure 4 An example of an input image evaluation method is shown in FIG. Figure 4 express Figure 1 The processing contents of the input image evaluation unit 38 are shown.

[0078] exist Figure 4In the example shown in the figure, the analysis value set 62 is generated by analyzing the input image 60. In the example shown in the figure, the analysis value set 62 includes the average brightness 64, the brightness variance value 66, the S / N ratio 68, the proportion of the low brightness area 70, etc. For example, when the ultrasonic probe is separated from the body surface, the S / N ratio 68 decreases, or the proportion of the low brightness area increases. Based on the content of the analysis value set or its change, the appropriateness of the input image 60 is comprehensively determined (reference symbol 72). The evaluation conditions to be used can be selected by the user or automatically.

[0079] In addition, when the quality of the input image is low, a reference image prepared in advance can be displayed. In other words, information to support the user can be provided so that the user can obtain a good tomographic image while referring to the displayed reference image.

[0080] exist Figure 5 An example of a graph generation method is shown in FIG. Figure 5 express Figure 1 The processing contents of the graph generating unit 40 are shown.

[0081] exist Figure 5 In the example, the entity of the analysis model 36 is a CNN. An input image 74 is input to the CNN, and output data 76 is output from the CNN. The CNN has a plurality of convolutional layers 78. The plurality of convolutional layers 78 generate a plurality of feature maps 84-1 to 84-n.

[0082] In the embodiment, a plurality of feature maps 84-1 to 84-n are extracted from the CNN. The extracted plurality of feature maps 84-1 to 84-n form a feature map column 82. Each feature map 84-1 to 84-n represents the response distribution of each convolution layer 78. A plurality of weights 86 are multiplied with the plurality of feature maps 84-1 to 84-n. A saliency map 88 is generated by adding the plurality of multiplication results.

[0083] On the premise that the CNN includes a GAP (Global Average Pooling) layer, multiple weights 86 can be taken out from between the GAP layer and the fully connected layer (output layer) 80 (refer to non-patent document 1). Multiple weights 86 can also be calculated by other methods. Various existing saliency map generation methods can be used (for example, refer to non-patent document 2 (Grad-CAM method)). In addition, saliency maps corresponding to all classes can be generated, and multiple saliency maps corresponding to multiple classes can be generated separately (reference symbol 94).

[0084] exist Figure 6 An example of a graph evaluation method is shown in FIG. Figure 6 express Figure 1 The processing contents of the evaluation unit 42 are shown in FIG.

[0085] exist Figure 6 In the example shown, binarization processing is applied to the saliency map 90 to generate a binarized saliency map 92. Labeling processing or region extraction processing is applied to the binarized saliency map 92 to extract a plurality of regions #1 to #4.

[0086] The maximum intensity is determined for each region #1 to #4, thereby generating table 96. Table 96 includes multiple maximum intensities representing multiple regions. One or more maximum intensities exceeding a threshold value are determined among the multiple maximum intensities (reference symbol 97), and a graph evaluation value 98 is calculated based on the one or more maximum intensities exceeding the threshold value. For example, the graph evaluation value may be calculated based on the number of maximum intensities exceeding the threshold value. It is believed that this number represents the breadth or strength of the CNN's response. The graph evaluation value may also be calculated based on the area of ​​each region, the coordinates of each region, etc.

[0087] exist Figure 7 Another graph evaluation method is shown in . A saliency map 90 is given to an evaluation model 100 generated through a machine learning process. Thus, a graph evaluation value 101 is output from the evaluation model 100. If the saliency map 90 is analyzed using the results of machine learning, the saliency map 90 can be correctly evaluated.

[0088] exist Figure 8 An example of the confidence history and an example of the history evaluation method are shown in FIG. Figure 8 Yes means Figure 1 The diagram shows the processing contents of the history generating unit 44 and the processing contents of the history evaluating unit 46.

[0089] exist Figure 8 In the upper section, the confidence history 102 is composed of a plurality of confidence graphs 104A to 104D corresponding to a plurality of classes (a plurality of cross-section types). Each confidence graph 104A to 104D is composed of a plurality of confidences arranged in chronological order. In the following, the confidence is simply expressed as a score depending on the situation.

[0090] The horizontal axis in the confidence history 102 is the time axis. ta represents the current time, and tb represents a time point in the past N frames back. The period between ta and tb is T. For each class, within the period T, an average value (average score) is calculated from a plurality of confidences.

[0091] The reference frame number N can be made different when performing dynamic image analysis and when performing still image analysis. The reference frame number N can be set by the user or automatically according to the situation. For example, N can be specified from 1 to several thousand, or T can be specified from several seconds to several hours.

[0092] The vertical axis in the confidence history 102 indicates the magnitude of the confidence. α1 is a threshold. In the illustrated confidence history 102, only one confidence curve 104A is higher than the threshold α1, and the other confidence curves 104B to 104D are lower than the threshold α1.

[0093] exist Figure 8 The lower section shows the evaluation method of the confidence history. In the example shown in the figure, when the total score of a certain class in N frames is greater than the threshold value α 1 (reference condition C1), 1.0 is determined as the evaluation value. When only one average score is greater than the threshold value α 1 (reference condition C2), 0.9 is determined as the evaluation value. In addition, when both conditions C1 and C2 are satisfied, condition C1 is preferentially satisfied.

[0094] In the case where the two average scores are greater than the threshold value α 1 (reference condition C3), 0.4 is determined as the evaluation value. This is because, as a result, it can be said that the screening is insufficient. This method is shown in Fig. 9 .exist Fig. 9 In the confidence history shown, two confidence curve graphs 106A and 106D are greater than the threshold α1, but the other two confidence curve graphs 106B and 106C are less than the threshold α1.

[0095] exist Figure 8 In the case where all average scores are less than the threshold α 1, 0.1 is determined as the evaluation value. Fig.10 .exist Fig.10 In the confidence history shown, four confidence curve graphs 108A to 108D are smaller than the threshold α1.

[0096] exist Fig.11 Another evaluation method of the confidence history is shown in . In S10, an average score is calculated for each class within a certain period. In S12, the suboptimal ratio is calculated by dividing the first average score (the largest average score) by the second average score (the second largest average score).

[0097] In S14, it is determined whether the suboptimal ratio is greater than a first threshold. In S16, it is determined whether all average scores are less than a second threshold. In S18, it is determined whether other predetermined conditions are satisfied.

[0098] In S20, the historical evaluation value is calculated based on the determination results of S14, S16, and S18. Various calculation methods can be used to perform this calculation. As other prescribed conditions applied in S18, conditions related to the temporal stability of each score can be cited. Figure 8 and Fig.11 The evaluation method shown is an example, and various evaluation methods can be adopted.

[0099] exist Fig.12 It is shown in Figure 1 The processing contents of the determination unit (selection unit) 48 are shown. Whether to use an external analysis model is determined based on the input image evaluation result 110, the image evaluation value 112, and the history evaluation value 114. That is, the internal analysis model or the external analysis model is selected as the object to be used (reference symbol 115). Symbol 118 indicates the selection of the external analysis model, and symbol 120 indicates the selection of the internal analysis model.

[0100] In an embodiment, when the quality of the input image satisfies a prescribed input image quality condition and the graph evaluation value and the history evaluation value do not satisfy a prescribed model performance condition, an external analysis model is selected. In this case, when at least one of the graph evaluation value and the history evaluation value exceeds a prescribed threshold value, it can be judged that the prescribed model performance condition is satisfied. Alternatively, when both the graph evaluation value and the history evaluation value exceed a prescribed threshold value, it can be judged that the prescribed model performance condition is satisfied. Depending on the situation, only one of the graph evaluation value and the history evaluation value can be referenced. When the quality of the input image does not satisfy the prescribed input image quality condition, the selection of the model is not performed, and the control based on the analysis result is temporarily stopped (reference symbol 116). In this case, a message indicating that the image quality has deteriorated can be provided to the user.

[0101] In the embodiment, the internal analysis model is also in operation while the external analysis model is being used, and the performance of the internal analysis model is continuously evaluated. When the quality of the input image satisfies the predetermined input image quality condition and the image evaluation value and the history evaluation value satisfy the predetermined model performance condition, the determination unit 48 determines to restart the use of the internal analysis model (reference symbol 115).

[0102] exist Fig.13 . An ultrasonic diagnostic system according to another embodiment is described in . An ultrasonic diagnostic apparatus 121 is connected to a network 122. Information processing apparatuses 124 and 126 are connected to the network 122. The ultrasonic diagnostic apparatus 121 has an internal analysis model 128 and a model selection unit 130. The information processing apparatus 124 has a first external analysis model 132. The information processing apparatus 126 has a second external analysis model 134.

[0103] The internal analysis model 128, the first external analysis model, and the second external analysis model respectively identify the type of tomographic image generated in the ultrasonic diagnostic apparatus. The model selection unit 130 selects the analysis model to be used based on the performance evaluation result of the internal analysis model 128. Specifically, it is selected whether to use the internal analysis model 128 or the external analysis model, and when using the external analysis model, it is selected whether to use the first external analysis model 132 or the second external analysis model 134. The model selection unit 130 may select the external analysis model to be used based on the content or quality of the tomographic image. The external analysis model to be used may also be selected based on the performance and performance of the first external analysis model 132 and the performance and performance of the second external analysis model 134.

[0104] In addition, a plurality of internal analysis models may be provided inside the ultrasonic diagnostic apparatus and selectively used. In this case, the analysis model selection technique according to the embodiment may also be applied.

Claims

1. An ultrasonic diagnostic device used in an ultrasonic examination of a subject, the ultrasonic diagnostic device comprising: a generator that generates an ultrasound image based on data obtained from the subject; an internal analysis model, which is disposed inside the ultrasonic diagnostic device and analyzes the ultrasonic image; an evaluator for evaluating the internal analysis model; a determiner that determines the use of an external analysis model provided outside the ultrasonic diagnostic apparatus based on an evaluation result of the evaluator; and The communicator transmits the ultrasonic image to the external analysis model and receives an analysis result of the external analysis model when determining the use of the external analysis model.

2. The ultrasonic diagnostic apparatus according to claim 1, wherein: The evaluator evaluates the internal analysis model based on at least one of internal data extracted from the internal analysis model and output data output from the internal analysis model.

3. The ultrasonic diagnostic apparatus according to claim 2, wherein: The evaluator evaluates the internal analysis model based on both the internal data and the output data.

4. The ultrasonic diagnostic apparatus according to claim 1, wherein: The evaluator comprises: a map generator that generates a saliency map based on a plurality of feature maps that are internal data extracted from the internal analysis model; and a map evaluator for evaluating the saliency map, The determiner determines use of the external analysis model based on an evaluation result of the saliency map.

5. The ultrasonic diagnostic apparatus according to claim 1, wherein: The evaluator comprises: a history generator that generates a confidence history representing temporal changes in confidences of a plurality of classes based on output data output from the internal analysis model; and a history evaluator, evaluating the confidence history, The determiner determines use of the external analysis model based on an evaluation result of the confidence history.

6. The ultrasonic diagnostic apparatus according to claim 1, wherein: The ultrasonic diagnostic apparatus includes an input image evaluator for evaluating an ultrasonic image input to the internal analysis model, that is, an input image. The determiner determines use of the external analysis model based on an evaluation result of the input image.

7. The ultrasonic diagnostic apparatus according to claim 6, wherein: The determiner determines the use of the external analysis model when the input image satisfies an image quality condition and an evaluation result of the internal analysis model does not satisfy a model performance condition.

8. The ultrasonic diagnostic apparatus according to claim 1, wherein: The ultrasonic diagnostic device includes a controller, which controls the operation of the ultrasonic diagnostic device. The internal analysis model and the external analysis model are both models for identifying cross sections. The controller controls the operation of the ultrasonic diagnostic apparatus based on the cross section identified by the internal analysis model before determining the use of the external analysis model, and controls the operation of the ultrasonic diagnostic apparatus based on the cross section identified by the external analysis model after determining the use of the external analysis model.

9. The ultrasonic diagnostic apparatus according to claim 1, wherein: The evaluator continues to evaluate the internal analysis model even after determining the use of the external analysis model. The determiner determines to restart the use of the internal analysis model based on an evaluation result of the evaluator after determining the use of the external analysis model.

10. An ultrasonic diagnostic system, characterized in that: include: Ultrasonic diagnostic equipment for use in ultrasonic examination of a subject; and an information processing device connected to the ultrasonic diagnostic device via a network, The information processing device includes an external analysis model that analyzes an ultrasonic image outside the ultrasonic diagnostic device. The ultrasonic diagnostic device comprises: a generator that generates the ultrasound image based on data obtained from the subject; an internal analysis model for analyzing the ultrasonic image inside the ultrasonic diagnostic device; an evaluator for evaluating the internal analysis model; a determiner that determines the use of the external analysis model based on the evaluation result of the evaluator; and The communicator transmits the ultrasonic image to the external analysis model and receives an analysis result of the external analysis model when determining the use of the external analysis model.

Images