Data generation device and data generation method

Abstract

To provide a data generation device and a data generation method which can efficiently generate a learning data set.SOLUTION: A data generation device acquires a medical image, a type of a lesion in the medical image determined by a learned model and a confidence degree applied to the type of the lesion. The data generation device associates the medical image with the type of the lesion and stores them in the learning data set when determining that the confidence degree exceeds a threshold. The data generation device transmits data of the medical image to a first terminal device when determining that the confidence degree is equal to or less than the threshold.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a data generating device and a data generating method. [Background technology]

[0002] Conventionally, a method for diagnosing a lesion from an image using machine learning has been known (for example, Patent Document 1).

[0003] The '1999 patent describes extracting regions of interest from digital mammograms that are suspected of containing microcalcifications and converting the regions of interest into numerical data, which is then input into a neural network that is trained to detect microcalcifications. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 6-343627 Summary of the Invention [Problem to be solved by the invention]

[0005] Machine learning requires training data, and conventional techniques require efficient generation of training data.

[0006] In view of the above, an object of the present disclosure is to provide a data generation device and a data generation method capable of efficiently generating training data. [Means for solving the problem]

[0007] A data generating device according to an embodiment of the present disclosure acquires a medical image, a type of lesion in the medical image determined by a trained model, and a confidence level assigned to the type of lesion. When the data generating device determines that the confidence level exceeds a threshold, the data generating device associates the medical image with the type of lesion and stores it in a learning dataset. When the data generating device determines that the confidence level is equal to or less than the threshold, the data generating device transmits data of the medical image to a first terminal device.

[0008] A data generation method according to an embodiment of the present disclosure includes acquiring a medical image, a type of lesion in the medical image determined by a trained model, and a confidence level assigned to the type of lesion. The data generation method includes storing the medical image in a learning dataset in association with the type of lesion when it is determined that the confidence level exceeds a threshold. The data generation method includes transmitting data of the medical image to a first terminal device when it is determined that the confidence level is equal to or less than the threshold. Effect of the Invention

[0009] According to one embodiment of the present disclosure, a data generating device and a data generating method capable of efficiently generating a training dataset can be provided. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a schematic configuration of a data generation system according to a first embodiment of the present disclosure. [Diagram 2] 5 is a flowchart showing an operation of the data generating device according to the first embodiment of the present disclosure. [Diagram 3] FIG. 11 is a diagram showing a training data set according to the second embodiment of the present disclosure. [Figure 4] 10 is a flowchart showing an operation of the data generating device according to the second embodiment of the present disclosure. [Diagram 5] 10 is a flowchart showing an operation of the data generating device according to the second embodiment of the present disclosure. [Figure 6]FIG. 13 is a diagram showing an updated data set according to the third embodiment of the present disclosure. [Figure 7] FIG. 13 is a diagram showing changes over time in a learning dataset according to a third embodiment of the present disclosure. [Figure 8] 13 is a flowchart showing an operation of the data generating device according to the third embodiment of the present disclosure. [Figure 9] 13 is a flowchart showing an operation of the data generating device according to the third embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] (First embodiment) 1 is a diagram showing a schematic configuration of a data generation system 1 according to a first embodiment of the present disclosure. The data generation system 1 generates a learning dataset 33. The learning dataset 33 can be used for machine learning of a medical diagnostic device. The diagnostic device may be, for example, a device that diagnoses the type of a lesion in a medical image from the medical image.

[0013] In the present disclosure, a "lesion in a medical image" refers to a lesion in a specimen that is included as an image in a medical image. In addition, a medical image may be any type of image of a specimen, such as a tissue slice or a cell. For example, a medical image may be an X-ray CT (Computed Tomography) image, an X-ray image, an MRI (Magnetic Resonance Imaging) image, a nuclear medicine image, an ultrasound image, or the like.

[0014] The data generation system 1 includes terminal devices 10A, 10B, and 10C, an information processing device 20, and a data generation device 30. The terminal devices 10A to 10C, the information processing device 20, and the data generation device 30 can communicate with each other via a network 2. The network 2 may be any network including a mobile communication network, the Internet, and the like. The information processing device 20 and the data generation device 30 may be able to communicate with each other directly without going through the network 2. Hereinafter, it is assumed that the information processing device 20 and the data generation device 30 communicate with each other directly without going through the network 2. However, the information processing device 20 and the data generation device 30 may communicate with each other via the network 2.

[0015] Hereinafter, when there is no particular distinction between the terminal device 10A, the terminal device 10B, and the terminal device 10C, they will be collectively referred to as "terminal device 10." In the configuration shown in FIG. 1, the number of terminal devices 10 included in the data generation system 1 is three. However, the number of terminal devices 10 included in the data generation system 1 may be any number. For example, the number of terminal devices 10 included in the data generation system 1 may be two or less, or may be four or more.

[0016] Different terminal devices 10 may be used by different workers. For example, terminal device 10A may be used by worker A. Terminal device 10B may be used by worker B. Terminal device 10C may be used by worker C.

[0017] An operator who uses the terminal device 10 may be a medical professional such as a doctor. The operator uses the terminal device 10 to perform annotation work. The annotation work is a work of labeling a medical image with the type of lesion in the medical image. The operator may have a regular job in addition to the annotation work. For example, if the operator is a doctor, the regular job is examining patients, etc. The operator is required to perform the annotation work during a time when the operator is not performing the regular job.

[0018] The terminal device 10 may be a personal computer, a tablet terminal device, etc. The terminal device 10 includes a communication unit 11, a display unit 12, an input unit 13, a storage unit 14, and a control unit 15.

[0019] The communication unit 11 may be configured to include at least one communication module connectable to the network 2. For example, the communication module may be a communication module compatible with a mobile communication standard such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation). For example, the communication module may be a communication module compatible with a standard such as a wired LAN (Local Area Network) or a wireless LAN. The communication unit 11 may be connected to the network 2 via a wired LAN or a wireless LAN by a communication module compatible with a standard such as a wired LAN or a wireless LAN.

[0020] The display unit 12 may be configured to include any display such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display. The display unit 12 displays medical images under the control of the control unit 15. An operator can visually recognize the medical images from the display unit 12. By visually recognizing the medical images from the display unit 12, the operator can diagnose the type of lesion in the medical image.

[0021] The input unit 13 includes one or more interfaces capable of detecting a user input. For example, the input unit 13 may include at least one of a physical key, a capacitive key, and a touch screen. The operator may input the type of lesion in the medical image from the input unit 13.

[0022] The storage unit 14 may be configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read only memory (EEPROM). The storage unit 14 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 14 stores various programs that cause the control unit 15 to function and various information used by the control unit 15. For example, the storage unit 14 stores information on the identifier of the worker.

[0023] The control unit 15 includes one or more processors and memories. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field programmable gate array (FPGA). The control unit 15 may be either a system-on-a-chip (SoC) or a system-in-a-package (SiP) in which one or more processors work together.

[0024] The control unit 15 may acquire data of the medical image and information indicating an identifier of the medical image from the data generating device 30 via the network 2 by the communication unit 11. When the control unit 15 acquires data of the medical image from the data generating device 30, it causes the medical image to be displayed on the display unit 12. When the medical image is displayed on the display unit 12, the type of lesion in the medical image may be input by the operator from the input unit 13. The control unit 15 may detect the input of the type of lesion from the operator by the input unit 13. When the control unit 15 detects the input of the type of lesion, it may transmit information indicating the identifier of the operator, information indicating the identifier of the medical image, and information indicating the type of lesion to the data generating device 30 via the network 2 by the communication unit 11.

[0025] The information processing device 20 may perform annotation work. The information processing device 20 may be a general-purpose computer, or may be a computer dedicated to performing annotation work. The information processing device 20 has a communication unit 21, a storage unit 22, and a control unit 24.

[0026] The communication unit 21 may be configured to include at least one communication module connectable to the network 2. For example, the communication module connectable to the network 2 may be a communication module compatible with a mobile communication standard such as LTE, 4G, or 5G. The communication unit 21 may be configured to include a communication module capable of communicating with the data generating device 30. The communication module capable of communicating with the data generating device 30 may be a communication module compatible with a standard such as a wired LAN or a wireless LAN. The communication unit 21 may directly communicate with the data generating device 30 via a wired LAN or a wireless LAN by a communication module compatible with a standard such as a wired LAN or a wireless LAN. The communication unit 21 may also be connected to the network 2 via a wired LAN or a wireless LAN by a communication module compatible with a standard such as a wired LAN or a wireless LAN.

[0027] The memory unit 22 may be configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM or a ROM. The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The memory unit 22 may function as a main memory device, an auxiliary memory device, or a cache memory. The memory unit 22 stores various programs that cause the control unit 24 to function and various information used by the control unit 24. The memory unit 22 includes a trained model 23.

[0028] The trained model 23 may be generated by machine learning using an existing training data set. The existing training data set may be generated from data on which an operator has already performed annotation work. The machine learning algorithm for generating the trained model 23 may correspond to the machine learning algorithm of a medical diagnostic device that is trained by machine learning using the training data set 33. For example, the machine learning algorithm for generating the trained model 23 may be at least one of a support vector machine (SVM), Bayesian estimation, and deep learning (DNN), etc.

[0029] The control unit 24 includes one or more processors and memories. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application-specific IC. The processor may include a programmable logic device. The PLD may include an FPGA. The control unit 24 may be either an SoC or a SiP in which one or more processors work together.

[0030] The control unit 24 may acquire information indicating the data of the medical image and an identifier of the medical image from the data generating device 30 via the communication unit 21. When acquiring the data of the medical image, the control unit 24 determines the type of lesion in the medical image using the trained model 23.

[0031] The control unit 24 assigns a confidence level to the determined type of lesion. The confidence level may be an index indicating the likelihood of the type of lesion determined by the trained model 23. When the machine learning algorithm used to generate the trained model 23 is deep learning, the control unit 24 may calculate the confidence level based on a matching rate between the feature amount extracted from the medical image by the trained model 23 and the feature amount of the determined type of lesion. The feature amount may include at least any of the brightness and color of the pixel of the specimen included as an image in the medical image, the shape of the specimen, the number of pixels constituting the specimen, and the feature amount extracted by deep learning. The control unit 24 may acquire, as the confidence level, a parameter indicating the likelihood of the type of lesion determined by the trained model 23, among the parameters used in the machine learning used to generate the trained model 23. For example, when the machine learning algorithm is a support vector machine, the control unit 24 may acquire, as the confidence level, the “Probability” output at the time of judgment. Instead of the control unit 24 calculating or acquiring the confidence level, the trained model 23 may be machine-learned to determine the type of lesion in a medical image and output the confidence level to be assigned to that type of lesion.

[0032] The control unit 24 transmits information indicating the identifier of the medical image, information indicating the determined type of lesion, and information indicating the degree of confidence assigned to the type of lesion to the data generating device 30 via the communication unit 21.

[0033] The data generating device 30 generates a learning dataset 33. The data generating device 30 may be a general-purpose computer, or may be a computer dedicated to generating the learning dataset 33. The data generating device 30 includes a communication unit 31, a storage unit 32, and a control unit 34.

[0034] The communication unit 31 may be configured to include at least one communication module connectable to the network 2. For example, the communication module connectable to the network 2 may be a communication module compatible with a mobile communication standard such as LTE, 4G, or 5G. The communication unit 31 may be configured to include a communication module capable of communicating with the information processing device 20. The communication module capable of communicating with the information processing device 20 may be a communication module compatible with a standard such as a wired LAN or wireless LAN. The communication unit 31 may directly communicate with the information processing device 20 via a wired LAN or wireless LAN by a communication module compatible with a standard such as a wired LAN or wireless LAN. The communication unit 31 may also be connected to the network 2 via a wired LAN or wireless LAN by a communication module compatible with a standard such as a wired LAN or wireless LAN.

[0035] The storage unit 32 may be configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM or a ROM. The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The storage unit 32 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 32 stores various programs that cause the control unit 34 to function and various information used by the control unit 34. The storage unit 32 stores a learning data set 33 generated by a process described below. The learning data set 33 may be generated as a database.

[0036] The control unit 34 includes one or more processors and memories. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application-specific IC. The processor may include a programmable logic device. The PLD may include an FPGA. The control unit 34 may be either an SoC or a SiP in which one or more processors work together.

[0037] The control unit 34 may acquire data of medical images on which annotation work has not been performed and information indicating an identifier of the medical images from outside via the communication unit 31. When acquiring data of medical images on which annotation work has not been performed, the control unit 34 generates or updates the learning dataset 33. In this embodiment, the control unit 34 generates or updates the learning dataset 33 by appropriately saving data in the learning dataset 33, as described below.

[0038] Specifically, when the control unit 34 acquires information indicating data and an identifier of a medical image on which annotation work has not been performed, the control unit 34 transmits the acquired medical image data and information indicating the identifier of the medical image to the information processing device 20 via the communication unit 31. When the information processing device 20 receives the medical image data and the like from the data generating device 30, the information processing device 20 determines the type of lesion in the medical image using the trained model 23 as described above. When the information processing device 20 determines the type of lesion in the medical image, information indicating the identifier of the medical image, information indicating the type of lesion determined by the information processing device 20, and information indicating the confidence level assigned to the type of lesion are transmitted from the information processing device 20 to the data generating device 30.

[0039] The control unit 34 may receive information indicating an identifier of a medical image, information indicating a type of lesion, and information indicating a confidence level from the information processing device 20 via the communication unit 31. That is, the control unit 34 may receive the type of lesion in the medical image determined by the trained model 23 of the information processing device 20, and the confidence level assigned to the type of lesion. The control unit 34 determines whether the received confidence level is equal to or lower than a threshold value.

[0040] The threshold may be set appropriately depending on the lesion determination accuracy by the trained model 23. The higher the lesion determination accuracy by the trained model 23, the smaller the threshold may be. The threshold may be a constant value. The threshold may be a variable value that varies depending on the lesion determination accuracy by the trained model 23. For example, assume that the trained model 23 is generated by machine learning using an existing training dataset. In this case, when the amount of data of the existing training dataset increases, the determination accuracy by the trained model 23 may be higher than before the amount of data of the existing training dataset increases. In this case, the threshold after the amount of data of the existing training dataset increases and the determination accuracy by the trained model 23 increases may be smaller than the threshold before the amount of data of the existing training dataset increases.

[0041] When the confidence level exceeds the threshold, the type of lesion to which the confidence level is assigned is highly likely to be correct. In other words, the type of lesion determined by the trained model 23 of the information processing device 20 is highly likely to be correct. Therefore, in this embodiment, the operator is not allowed to perform annotation work on the medical image used to determine the type of lesion to which the confidence level exceeding the threshold is assigned. The control unit 34 stores the medical image used to determine the type of lesion to which the confidence level exceeding the threshold is assigned in the training dataset 33 in association with the type of lesion determined by the trained model 23. Specifically, when the control unit 34 determines that the confidence level received from the information processing device 20 exceeds the threshold, the control unit 34 stores the medical image corresponding to the identifier received from the information processing device 20 in association with the type of lesion received from the information processing device 20 in the training dataset 33. The control unit 34 may store the medical image in association with the confidence level in the training dataset 33.

[0042] When the confidence level is equal to or lower than the threshold, the type of lesion to which the confidence level is assigned is unlikely to be correct. In other words, the type of lesion determined by the trained model 23 of the information processing device 20 is unlikely to be correct. Therefore, the control unit 34 causes the operator to perform annotation work for the medical image used to determine the type of lesion to which the confidence level equal to or lower than the threshold is assigned. Specifically, when the control unit 34 determines that the confidence level received from the information processing device 20 is equal to or lower than the threshold, the control unit 34 transmits data of the medical image used to determine the type of lesion to which the confidence level is assigned to the first terminal device via the network 2 by the communication unit 31. The control unit 34 transmits information indicating the identifier of the medical image together with the data of the medical image used to determine the type of lesion to which the confidence level is assigned to the first terminal device via the network 2 by the communication unit 31. Here, the first terminal device is a terminal device 10 appropriately selected from the terminal devices 10 included in the data generation system 1. The control unit 34 may appropriately select a first terminal device from among the terminal devices 10A to 10C so that the annotation work burden is evenly distributed among the workers A to C. For example, the control unit 34 may select a first terminal device from among the terminal devices 10A to 10C so that the number of frames of medical images transmitted from the data generating device 30 to each of the terminal devices 10A to 10C is equalized. With this configuration, the number of medical images on which the workers A to C perform annotation work can be equalized.

[0043] The terminal device 10 as the first terminal device may receive information indicating data of the medical image and an identifier of the medical image from the data generating device 30. When the terminal device 10 receives the data of the medical image, etc., the worker performs annotation work using the terminal device 10 during a time period when the worker is not performing normal work. When the worker performs annotation work using the terminal device 10, information indicating the identifier of the worker, information indicating the identifier of the medical image, and information indicating the type of lesion diagnosed by the worker are transmitted from the terminal device 10 as the first terminal device to the data generating device 30.

[0044] The control unit 34 may receive information indicating an operator's identifier, information indicating an identifier of a medical image, and information indicating a type of lesion from the terminal device 10 as a first terminal device via the network 2 through the communication unit 31. The control unit 34 stores the medical image corresponding to the identifier received from the terminal device 10 in the learning dataset 33 in association with the type of lesion received from the terminal device 10. The control unit 34 may store the medical image in the learning dataset 33 in association with the type of lesion and the operator's identifier.

[0045] (Operation of data generating device) 2 is a flowchart showing the operation of the data generating device 30 according to the first embodiment of the present disclosure. When the control unit 34 acquires data of a medical image on which annotation work has not been performed and information indicating an identifier of the medical image from the outside via the communication unit 31, the control unit 34 may start the process shown in FIG.

[0046] The control unit 34 transmits data of the medical image on which annotation work has not been performed and information indicating the identifier of the medical image to the information processing device 20 via the communication unit 31 (step S10).

[0047] By executing the process of step S10, the information processing device 20 receives data of medical images, etc. from the data generating device 30. When the information processing device 20 receives the data of medical images, etc., it determines the type of lesion in the medical image, etc., using the trained model 23. By the information processing device 20 determining the type of lesion in the medical image, etc., information indicating the identifier of the medical image, information indicating the type of lesion determined by the information processing device 20, and information indicating the confidence level assigned to the type of lesion are transmitted from the information processing device 20 to the data generating device 30.

[0048] The control unit 34 receives information indicating an identifier of a medical image, information indicating a type of lesion, and information indicating a confidence level from the information processing device 20 via the communication unit 31 (step S11).

[0049] The control unit 34 determines whether the degree of confidence received in the process of step S11 is equal to or less than a threshold (step S12). When the control unit 34 determines that the degree of confidence received exceeds the threshold (step S12: No), the control unit 34 proceeds to the process of step S13. On the other hand, when the control unit 34 determines that the degree of confidence received is equal to or less than the threshold (step S12: Yes), the control unit 34 proceeds to the process of step S14.

[0050] In the process of step S13, the control unit 34 stores the medical image corresponding to the identifier received in the process of step S11 in the learning dataset 33 in association with the type of lesion received in the process of step S11.

[0051] In the process of step S14, the control unit 34 transmits the data of the medical image and information indicating the identifier of the medical image to the first terminal device via the network 2 through the communication unit 31.

[0052] By executing the process of step S14, the terminal device 10 as the first terminal device can receive data of the medical image, etc. from the data generating device 30. The worker performs annotation work using the terminal device 10 during a time period when the worker is not performing normal work. When the worker performs annotation work using the terminal device 10, information indicating the identifier of the worker, information indicating the identifier of the medical image, and information indicating the type of lesion diagnosed by the worker are transmitted from the terminal device 10 as the first terminal device to the data generating device 30.

[0053] In the processing of step S15, the control unit 34 receives information indicating the identifier of the operator, information indicating the identifier of the medical image, and information indicating the type of lesion from the first terminal device via the network 2 through the communication unit 31.

[0054] In the process of step S16, the control unit 34 stores the medical image corresponding to the identifier received in the process of step S15 in the learning dataset 33 in association with the type of lesion received in the process of step S15.

[0055] In this way, when the data generating device 30 according to the first embodiment determines that the confidence level exceeds the threshold, the data generating device 30 stores the medical image in the learning data set 33 in association with the type of lesion determined by the trained model 23 of the information processing device 20. That is, when the control unit 34 determines that the confidence level exceeds the threshold, the control unit 34 does not allow the operator to perform annotation work, and generates at least a part of the learning data set 33 according to the type of lesion determined by the trained model 23. With this configuration, when generating the learning data set 33, the operator does not need to perform annotation work on all medical images. As a result, the burden of the annotation work on the operator can be reduced. Therefore, according to the present embodiment, a data generating device 30 and a data generating method capable of efficiently generating the learning data set 33 can be provided.

[0056] Second embodiment Next, a data generation system according to a second embodiment will be described. The data generation system according to the second embodiment can adopt a configuration similar to that of the data generation system 1 according to the first embodiment. Therefore, the following will mainly describe the differences from the first embodiment with reference to FIG. 1.

[0057] As in the first embodiment, the control unit 34 can acquire data of a medical image on which annotation work has not been performed and information indicating an identifier of the medical image from the outside via the communication unit 31. When acquiring data of a medical image on which annotation work has not been performed, the control unit 34 transmits information indicating the acquired data of the medical image and an identifier of the medical image to the information processing device 20 via the communication unit 31.

[0058] As in the first embodiment, the control unit 34 may receive information indicating an identifier of the medical image, information indicating the type of lesion, and information indicating the confidence level assigned to the type of lesion from the information processing device 20 via the communication unit 31.

[0059] For example, the control unit 34 transmits data of medical images with identifiers ID1, ID2, ID3, and ID4 as shown in FIG. 3 to the information processing device 20. In FIG. 3, each of types X, Y, and Z indicates a different type of lesion. When the information processing device 20 receives data of a medical image with identifier ID1, the information processing device 20 determines the type of lesion in the medical image with identifier ID1 to be type X and assigns a confidence level of 100%. When the information processing device 20 receives data of a medical image with identifier ID2, the information processing device 20 determines the type of lesion in the medical image with identifier ID2 to be type X and assigns a confidence level of 50%. When the information processing device 20 receives data of a medical image with identifier ID3, the information processing device 20 determines the type of lesion in the medical image with identifier ID3 to be type Y and assigns a confidence level of 50%. When the information processing device 20 receives data of a medical image with identifier ID4, the information processing device 20 determines the type of lesion in the medical image with identifier ID4 to be type Z and assigns a confidence level of 40%. The control unit 34 may receive from the information processing device 20 the types of lesions and the confidence levels determined by the information processing device 20 .

[0060] The control unit 34 determines whether the received degree of confidence is equal to or lower than a threshold value, similarly to the first embodiment. When the control unit 34 determines that the degree of confidence received from the information processing device 20 exceeds the threshold value, similarly to the first embodiment, the control unit 34 stores a medical image corresponding to the identifier received from the information processing device 20 in the learning dataset 33 in association with the type of lesion received from the information processing device 20. The control unit 34 may store the medical image in the learning dataset 33 in association with the degree of confidence.

[0061] For example, in Fig. 3, the threshold is assumed to be 80%. In this case, for the medical image with identifier ID1 as shown in Fig. 3, the control unit 34 determines that a confidence level of 100% exceeds the threshold of 80%. The control unit 34 stores the medical image with identifier ID1 in the learning dataset 33 in association with the type X received from the information processing device 20 and the confidence level of 100%.

[0062] When the control unit 34 determines that the confidence level received from the information processing device 20 is equal to or lower than the threshold, the control unit 34 transmits data of the medical image used in determining the type of the lesion to which the confidence level has been assigned to the first terminal device via the network 2 by the communication unit 31. The control unit 34 transmits information indicating an identifier of the medical image together with the data of the medical image used in determining the type of the lesion to which the confidence level has been assigned to the first terminal device via the network 2 by the communication unit 31. Hereinafter, the operator using the first terminal device is also referred to as a "first operator". After transmitting the data of the medical image to the first terminal device, the control unit 34 may receive information indicating the identifier of the first operator, information indicating the identifier of the medical image, and information indicating the type of the lesion from the first terminal device via the network 2 by the communication unit 31.

[0063] For example, for a medical image with identifier ID2 as shown in Fig. 3, the control unit 34 determines that the confidence level of 50% is equal to or lower than the threshold value of 80%. The control unit 34 transmits data of the medical image with identifier ID2, etc. to the terminal device 10A as the first terminal device. The operator A diagnoses the type of lesion in the medical image with identifier ID2 as type X. The control unit 34 receives information indicating the identifier idA, information indicating the identifier ID2, and information indicating the type X from the terminal device 10A as the first terminal device. The identifier idA is the identifier of the operator A.

[0064] For example, for a medical image with identifier ID3 as shown in Fig. 3, the control unit 34 determines that the confidence level of 50% is equal to or lower than the threshold value of 80%. The control unit 34 transmits data of the medical image with identifier ID3, etc., to the terminal device 10C as the first terminal device. The operator C diagnoses the type of lesion in the medical image with identifier ID3 as type X. The control unit 34 receives information indicating the identifier idC, information indicating the identifier ID3, and information indicating the type X from the terminal device 10C as the first terminal device. The identifier idC is the identifier of the operator C.

[0065] For example, for a medical image with identifier ID4 as shown in Fig. 3, the control unit 34 determines that the confidence level of 40% is equal to or lower than the threshold value of 80%. The control unit 34 transmits data of the medical image with identifier ID4, etc. to the terminal device 10B as the first terminal device. The operator B diagnoses the type of lesion in the medical image with identifier ID4 as type X. The control unit 34 receives information indicating the identifier idB, information indicating the identifier ID4, and information indicating the type X from the terminal device 10B as the first terminal device. The identifier idB is the identifier of the operator B.

[0066] The control unit 34 determines whether the type of lesion received from the first terminal device matches the type of lesion received from the information processing device 20, i.e., the type of lesion determined by the trained model 23. When the control unit 34 determines that the type of lesion received from the first terminal device matches the type of lesion received from the information processing device 20, the control unit 34 stores the medical image in the training dataset 33 in association with the type of lesion received from the first terminal device and the identifier of the first operator. Furthermore, the control unit 34 may store the medical image in the training dataset 33 in association with the type of lesion and the confidence level received from the information processing device 20.

[0067] 3, the control unit 34 determines that the type X received from the information processing device 20 matches the type X received from the terminal device 10A as the first terminal device. The control unit 34 stores the medical image with the identifier ID2 in the learning dataset 33 in association with the type X and the confidence level of 50% received from the information processing device 20, and the type X and the identifier idA of the first operator received from the terminal device 10A as the first terminal device.

[0068] When the control unit 34 determines that the type of lesion received from the first terminal device does not match the type of lesion received from the information processing device 20, the control unit 34 transmits the medical image data to the second terminal device via the network 2 by the communication unit 31. The second terminal device is a terminal device 10 different from the first terminal device. Hereinafter, the operator using the second terminal device is also referred to as a "second operator." By transmitting the medical image data to the second terminal device, which is a terminal device 10 different from the first terminal device, annotation work on the medical image can be performed by the second operator different from the first operator.

[0069] 3, the control unit 34 determines that the type X received from the terminal device 10C as the first terminal device does not match the type Y received from the information processing device 20. The control unit 34 transmits the data of the medical image with the identifier ID3 to the terminal device 10B as the second terminal device different from the terminal device 10C. The operator B diagnoses the type of the lesion in the medical image with ID3 as type Y.

[0070] 3, the control unit 34 determines that the type X received from the terminal device 10B as the first terminal device does not match the type Z received from the information processing device 20. The control unit 34 transmits the data of the medical image with the identifier ID4 to the terminal device 10C as the second terminal device different from the terminal device 10B. The operator C diagnoses the type of the lesion in the medical image with the identifier ID4 as type X.

[0071] The control unit 34 may receive information indicating the identifier of the second operator, information indicating the identifier of the medical image, and information indicating the type of lesion from the second terminal device via the network 2 through the communication unit 31. The control unit 34 stores the medical image corresponding to the received identifier in the learning dataset 33 in association with the type of lesion received from the information processing device 20, the type of lesion received from the first terminal device, and the type of lesion received from the second terminal device. Furthermore, the control unit 34 may store the medical image in the learning dataset 33 in association with the confidence level received from the information processing device 20, the identifier of the first operator received from the first terminal device, and the identifier of the second operator received from the second terminal device.

[0072] For example, the control unit 34 receives information indicating the identifier idB, information indicating the identifier ID3, and information indicating the type Y from the terminal device 10B as the second terminal device for a medical image with an identifier ID3 as shown in Fig. 3. The control unit 34 associates the medical image with the identifier ID3 with the type Y received from the information processing device 20, the type X received from the terminal device 10C as the first terminal device, and the type Y received from the terminal device 10B as the second terminal device, and stores the medical image with the identifier ID3 in the learning dataset 33. Furthermore, the control unit 34 associates the medical image with the identifier ID3 with a confidence level of 50%, the identifier idC of the first operator, and the identifier idB of the second operator, and stores the medical image with the identifier ID3 in the learning dataset 33.

[0073] For example, for a medical image with an identifier ID4 as shown in Fig. 3, the control unit 34 receives information indicating the identifier idC, information indicating the identifier ID4, and information indicating the type X from the terminal device 10C as the second terminal device. The control unit 34 associates the medical image with the identifier ID4 with the type Z received from the information processing device 20, the type X received from the terminal device 10B as the first terminal device, and the type X received from the terminal device 10C as the second terminal device, and stores the medical image with the identifier ID4 in the learning dataset 33. Furthermore, the control unit 34 associates the medical image with the identifier ID4 with a confidence level of 40%, the identifier idB of the first operator, and the identifier idC of the second operator, and stores the medical image with the identifier ID4 in the learning dataset 33.

[0074] (Operation of data generating device) 4 and 5 are flowcharts showing the operation of the data generating device 30 according to the second embodiment of the present disclosure. When the control unit 34 acquires data of a medical image on which annotation work has not been performed and information indicating an identifier of the medical image from the outside via the communication unit 31, the control unit 34 may start the process shown in FIG.

[0075] The control unit 34 executes the processes of steps S20, S21, S22, S23, S24, and S25 in the same manner as the processes of steps S10, S11, S12, S13, S14, and S15 shown in FIG.

[0076] In the process of step S26, the control unit 34 determines whether or not the type of lesion received from the first terminal device in the process of step S25 matches the type of lesion received from the information processing device 20 in the process of step S21.

[0077] When the control unit 34 determines that the type of lesion received from the first terminal device and the type of lesion received from the information processing device 20 match (step S26: Yes), the control unit 34 proceeds to the process of step S27. On the other hand, when the control unit 34 determines that the type of lesion received from the first terminal device and the type of lesion received from the information processing device 20 do not match (step S26: No), the control unit 34 proceeds to the process of step S28 as shown in FIG.

[0078] In the process of step S27, the control unit 34 associates the medical image with the type of lesion received from the first terminal device, and stores it in the learning dataset 33. Furthermore, the control unit 34 may also associate the medical image with the type of lesion and the confidence level received from the information processing device 20, and the identifier of the first operator, and store it in the learning dataset 33.

[0079] In the process of step S28 as shown in FIG. 5, the control unit 34 transmits the data of the medical image and information indicating the identifier of the medical image to the second terminal device via the network 2 through the communication unit 31.

[0080] By executing the process of step S28, the terminal device 10 as the second terminal device can receive data of the medical image, etc. from the data generating device 30. The second worker performs annotation work using the terminal device 10 during times when the second worker is not performing normal work, etc. When the second worker performs annotation work using the terminal device 10, information indicating the identifier of the second worker, information indicating the identifier of the medical image, and information indicating the type of lesion are transmitted from the terminal device 10 as the second terminal device to the data generating device 30.

[0081] In the processing of step S29, the control unit 34 receives information indicating the identifier of the second worker, information indicating the identifier of the medical image, and information indicating the type of lesion from the second terminal device via the network 2 through the communication unit 31.

[0082] In the process of step S30, the control unit 34 associates the medical image with the type of lesion received from the information processing device 20, the type received from the first terminal device, and the type of lesion received from the second terminal device, and stores the medical image in the learning dataset 33. Furthermore, the control unit 34 may also associate the medical image with the confidence level received from the information processing device 20, the identifier of the first operator received from the first terminal device, and the identifier of the second operator, and store the medical image in the learning dataset 33.

[0083] Here, in the generation of a general learning data set, two or more workers are required to perform annotation work on one medical image. Furthermore, in the generation of a general learning data set, when the diagnosis results of each worker on one medical image match, the learning data set is generated by associating the diagnosis results with the medical image. In the generation of a general learning data set, since two or more workers are required to perform annotation work on one medical image, the burden on the workers may increase. Furthermore, after the first worker finishes the annotation work on the prepared medical image, the second worker starts the annotation work on the prepared medical image. In other words, the second worker must wait to start the annotation work until the first worker finishes the annotation work. For example, assume that medical images with identifiers ID1 to ID4 as shown in FIG. 3 are prepared, the first worker is worker A, and the second worker is worker B. In this case, the worker B must wait to start the annotation work on the medical images with the identifiers ID1 to ID4 until the worker A finishes the annotation work on the medical images with the identifiers ID1 to ID4.

[0084] When the data generating device 30 according to the second embodiment determines that the type of lesion received from the first terminal device matches the type of lesion received from the information processing device 20, the data generating device 30 associates the medical image with the type of lesion received from the first terminal device and stores the medical image in the learning dataset 33. In other words, when the type of lesion received from the first terminal device matches the type of lesion received from the information processing device 20, it is not necessary for two or more workers to perform annotation work on the medical image used to determine the type of lesion. With such a configuration, the burden on the workers can be reduced.

[0085] Furthermore, when the data generating device 30 according to the second embodiment determines that the confidence level is equal to or lower than the threshold, it transmits data of the medical image to the first terminal device. With this configuration, data of a plurality of medical images can be transmitted to a plurality of first terminal devices at the same time. For example, the control unit 34 can transmit data of medical images with identifiers ID2, ID3, and ID4 as shown in FIG. 3 to the terminal devices 10A, 10C, and 10B as the first terminal devices at the same time. By transmitting data of a plurality of medical images to a plurality of first terminal devices at the same time, a plurality of first operators can respectively perform annotation work on a plurality of medical images as the first operator who performs the annotation work. With this configuration, it is possible to eliminate a situation in which the second operator has to wait to start annotation work until the first operator finishes the annotation work, as in the generation of a general learning data set.

[0086] Furthermore, when transmitting data of a medical image to a second terminal device, the data generating device 30 according to the second embodiment does not transmit to the second terminal device the type of lesion determined by the information processing device 20 and the type of lesion diagnosed by the first operator, etc. With this configuration, the second operator can diagnose the lesion in the medical image without relying on the diagnosis result, etc., of the first operator.

[0087] Other configurations and effects of the data generating device 30 according to the second embodiment are similar to those of the data generating device 30 according to the first embodiment.

[0088] Third embodiment Next, a data generation system according to a third embodiment will be described. The data generation system according to the third embodiment can employ a configuration similar to that of the data generation system 1 according to the first embodiment. Therefore, the following description will focus on the differences from the first embodiment with reference to FIG. 1.

[0089] When generating or updating the learning dataset 33 as described above in the first and second embodiments, the control unit 34 may receive information indicating the type of lesion from the terminal device 10 via the network 2 by the communication unit 31. In the third embodiment, when generating or updating the learning dataset 33, the control unit 34 generates an update dataset for updating the trained model 23 of the information processing device 20 based on the information indicating the type of lesion received from the terminal device 10. In other words, the control unit 34 generates an update dataset based on a medical image on which an operator has performed an annotation work. The trained model 23 may be updated by machine learning using the update dataset, as described later. By updating the trained model 23, the accuracy of determining the type of lesion by the trained model 23 may be improved.

[0090] The control unit 34 may generate the updated dataset using data already stored in the learning dataset 33. Alternatively, the control unit 34 may temporarily store data in the storage unit 32 when generating or updating the learning dataset 33 as described above in the first and second embodiments. For example, the control unit 34 may temporarily store information indicating an operator's identifier, information indicating an identifier of a medical image, and information indicating a type of lesion, which are received from the terminal device 10, in the storage unit 32. In this case, the control unit 34 may generate the updated dataset using the information indicating the type of lesion temporarily stored in the storage unit 32. The control unit 34 may periodically generate the updated dataset. In this embodiment, as described below, the control unit 34 generates the updated dataset by appropriately storing data in the updated dataset.

[0091] The control unit 34 may generate an updated data set based on at least one of the type of lesion received from the first terminal device and the type of lesion received from the second terminal device.

[0092] When the control unit 34 receives the type of lesion for a medical image only from the first terminal device, the control unit 34 may store the medical image in the update data set in association with the type of lesion and a first weighting factor. The first weighting factor may be 1.

[0093] The medical image associated with the first weighting coefficient may be associated with only one type of lesion received from the first terminal device together with the first weighting coefficient. The trained model 23 may be trained to determine that the type of lesion in the medical image associated with the first weighting coefficient is only the type of lesion associated with the medical image when updating by machine learning using an update data set described later. At this time, the trained model 23 may be trained so that the confidence of the determined type of lesion approaches a value according to the first weighting coefficient. For example, the value according to the first weighting coefficient may be 100% when a confidence level is adopted as an index indicating the confidence level of the determined type of lesion. In this case, when the trained model 23 is configured to output a confidence level, the trained model 23 may be trained so that the confidence level approaches 100% for the type of lesion associated with the first weighting coefficient.

[0094] When using data already stored in the learning dataset 33, the control unit 34 may extract medical images associated only with the identifier of the first operator as medical images for which the type of lesion has been received only from the first terminal device, from the data already stored in the learning dataset 33. The control unit 34 may store the extracted medical images in the update dataset in association with the type of lesion and the first weighting coefficient received from the first terminal device.

[0095] For example, it is assumed that data as shown in Fig. 3 is used as data already stored in the learning dataset 33. Only the identifier idA of the first operator is associated with the medical image with the identifier ID2. In this case, the control unit 34 may store the medical image with the identifier ID2 in the update dataset in association with the type X and the first weighting coefficient as shown in Fig. 6. In the example shown in Fig. 6, the control unit 34 assigns 1 as the first weighting coefficient.

[0096] When the control unit 34 receives the type of lesion for the same medical image from both the first terminal device and the second terminal device, the control unit 34 may generate an update dataset by the medical image. In this case, the control unit 34 may extract, from the data already stored in the learning dataset 33, a medical image associated with both the identifier of the first operator and the identifier of the second operator as a medical image for which the type of lesion has been received from both the first terminal device and the second terminal device.

[0097] For example, it is assumed that data as shown in Fig. 3 is used as data stored in the learning dataset 33. A medical image with an identifier ID3 is associated with an identifier idC of a first operator and an identifier idB of a second operator. A medical image with an identifier ID4 is associated with an identifier idB of a first operator and an identifier idC of a second operator. The control unit 34 extracts the medical images with the identifiers ID3 and ID4 from the learning dataset 33 as medical images for which the types of lesions have been received from both the first terminal device and the second terminal device.

[0098] When the control unit 34 receives the type of lesion from both the first terminal device and the second terminal device for the same medical image, it may determine whether the type of lesion received from the first terminal device matches the type of lesion received from the second terminal device.

[0099] The control unit 34 may determine that the type of lesion received from the first terminal device and the type of lesion received from the second terminal device for the same medical image do not match. In this case, the control unit 34 may store the medical image in the update data set in association with the type of lesion received from each of the first terminal device and the second terminal device. The second weighting factor is smaller than the first weighting factor. The control unit 34 may calculate the second weighting factor by dividing the first weighting factor by the number of terminal devices 10 that have transmitted the type of lesion for the same medical image. For example, when the first weighting factor is 1 and the number of terminal devices 10 that have transmitted the type of lesion for the same medical image is two, the first terminal device and the second terminal device, the second weighting factor may be 0.5.

[0100] The medical scene image associated with the second weighting coefficient may be associated with two types of lesions received from the first terminal device and the second terminal device together with the second weighting coefficient. When updating by machine learning using an update data set described later, the trained model 23 may be trained to determine that the type of lesion in the medical image associated with the second weighting coefficient corresponds to each of the two types of lesions associated with the medical image. At this time, the trained model 23 may be trained so that the likelihood of each of the two types of lesions corresponding to the type of lesion in the medical image approaches a value corresponding to a second weighting coefficient smaller than the first weighting coefficient. For example, the value corresponding to the second weighting coefficient may be 50% when a confidence level is adopted as an index indicating the likelihood of the type of lesion determined and the second coefficient is 0.5. In this case, when the trained model 23 is configured to output the confidence level, the trained model 23 may be trained so that the confidence level approaches 50% for the type of lesion associated with the second weighting coefficient.

[0101] For example, the control unit 34 determines that the type X received from the terminal device 10C as the first terminal device does not match the type Y received from the terminal device 10B as the second terminal device for a medical image with an identifier ID3 as shown in Fig. 3. In this case, the control unit 34 associates the medical image with the identifier ID3 with the type X, the type Y, and the second weighting coefficient of 0.5, and stores the medical image in the updated data set as shown in Fig. 6.

[0102] The control unit 34 may determine that the type of lesion received from the first terminal device and the type of lesion received from the second terminal device for the same medical image match each other. In this case, the control unit 34 may store the medical image in the update data set in association with the type of lesion and the first weighting factor received from each of the first terminal device and the second medical image.

[0103] For example, the control unit 34 determines that the type X received from the terminal device 10B as the first terminal device matches the type X received from the terminal device 10C as the second terminal device for a medical image with an identifier ID4 as shown in Fig. 3. In this case, the control unit 34 associates the medical image with the identifier ID4 with the type X and the first weighting coefficient of 1, and stores the medical image in the updated data set as shown in Fig. 6.

[0104] The control unit 34 transmits the generated updated data set to the information processing device 20 via the communication unit 31. In the information processing device 20, the control unit 24 may receive the updated data set from the data generating device 30 via the communication unit 21. Upon receiving the updated data set, the control unit 24 updates the trained model 23 by machine learning using the updated data set. Upon completing the update of the trained model 23, the control unit 24 may transmit a notification indicating the completion of the update to the data generating device 30 via the communication unit 21.

[0105] The control unit 34 may receive a notification indicating the end of the update from the information processing device 20 through the communication unit 31. When the control unit 34 receives the notification indicating the end of the update, it extracts medical images stored in the learning dataset 33 to which only the type of lesion received from the information processing device 20, i.e., the type of lesion determined by the trained model 23, is associated. For example, when the learning dataset 33 has a configuration as shown in FIG. 3, the control unit 34 extracts a medical image with an identifier ID1. The control unit 34 transmits information indicating the data of the extracted medical image and the identifier of the medical image to the information processing device 20 through the communication unit 21. In the information processing device 20, the control unit 24 may receive the data of the medical image and the identifier of the medical image through the communication unit 21. When the control unit 24 receives the data of the medical image, etc., it determines the type of lesion of the medical image using the updated trained model 23 and assigns a degree of confidence to the determined type of lesion. The control unit 24 transmits information indicating the identifier of the medical image, information indicating the determined type of lesion, and information indicating the degree of confidence assigned to the type of lesion to the data generating device 30 via the communication unit 21.

[0106] The control unit 34 may receive information indicating an identifier of a medical image, information indicating a type of lesion, and information indicating a confidence level from the information processing device 20 via the communication unit 31. The control unit 34 may update the data already stored in the learning dataset 33 based on the received information indicating the type of lesion, etc.

[0107] Fig. 7 shows the change over time of the training data set 33 according to the third embodiment of the present disclosure. Fig. 7 shows the type of lesion and the confidence level determined by the trained model 23 for the medical image with identifier ID1 in the first update, the second update, and the third update of the trained model 23.

[0108] The more times the trained model 23 is updated, the more accurate the determination of the type of lesion by the trained model 23 can be. With this configuration, even if the type of lesion determined by the trained model 23 for the same medical image is the same before and after the update, the confidence level assigned after the update may be lower than the confidence level assigned before the update. For example, as shown in FIG. 7, in the first and second updates, the trained model 23 determines the type of lesion in the medical image with identifier ID1 to be the same type X. The confidence level assigned to the second update is 50%, which is lower than the confidence level of 80% assigned to the first update.

[0109] As the accuracy of lesion type determination by the trained model 23 improves, the type of lesion determined for the same medical image by the trained model 23 before the update may differ from the type of lesion determined by the trained model 23 after the update. For example, as shown in Fig. 7, in the second update, the trained model 23 determines the medical image with identifier ID1 as type X. In contrast, in the third update, the trained model 23 determines the medical image with identifier ID1 as type Y.

[0110] (Operation of data generating device) 8 and 9 are flowcharts showing the operation of the data generating device 30 according to the third embodiment of the present disclosure. In the following, it is assumed that the control unit 34 generates an updated dataset using data already stored in the learning dataset 33. The control unit 34 may periodically execute the process as shown in FIG. 8. For example, the control unit 34 may execute the process as shown in FIG. 8 every time the number of medical images already stored in the learning dataset 33 increases by a predetermined value. The predetermined value may be appropriately determined taking into consideration the accuracy of lesion type determination required for the trained model 23.

[0111] The control unit 34 extracts medical images associated only with the identifier of the first operator from the data stored in the learning dataset 33 as medical images for which the type of lesion has been received only from the first terminal device (step S40).

[0112] The control unit 34 stores the medical images extracted in the process of step S40 in an updated data set in association with the type of lesion and the first weighting factor received from the first terminal device (step S41). The control unit 34 may repeatedly execute the process of step S41 the same number of times as the number of medical images extracted in the process of step S40.

[0113] The control unit 34 extracts medical images associated with both the first operator's identifier and the second operator's identifier from the data stored in the learning dataset 33 as medical images for which the type of lesion has been received from both the first terminal device and the second terminal device (step S42).

[0114] The control unit 34 determines whether the type of lesion received from the first terminal device and the type of lesion received from the second terminal device for the same medical image extracted in the process of step S42 match (step S43). When the control unit 34 determines that the type of lesion received from the first terminal device and the type of lesion received from the second terminal device for the same medical image do not match (step S43: No), the control unit 34 proceeds to the process of step S44. When the control unit 34 determines that the type of lesion received from the first terminal device and the type of lesion received from the second terminal device match for the same medical image (step S43: Yes), the control unit 34 proceeds to the process of step S45.

[0115] In the processing of step S44, the control unit 34 stores the medical image used in the judgment processing of step S43 in an updated data set in association with the type of lesion and the second weighting coefficient received from each of the first terminal device and the second terminal device.

[0116] In the processing of step S45, the control unit 34 stores the medical images used in the judgment processing of step S43 in an updated data set in association with the type of lesion and the first weighting coefficient received from each of the first terminal device and the second medical image.

[0117] The control unit 34 may repeatedly execute the processes of steps S43 to S45 the number of times corresponding to the number of medical images extracted in the process of step S42.

[0118] In the process of step S46, the control unit 34 transmits the generated update data set to the information processing device 20 via the communication unit 31.

[0119] After executing the process of step S46, the control unit 34 proceeds to the process shown in FIG.

[0120] By executing the process of step S46, in the information processing device 20, the control unit 24 can receive the updated data set from the data generating device 30 via the communication unit 21. Upon receiving the updated data set, the control unit 24 updates the trained model 23 by machine learning using the updated data set. Upon completing the update of the trained model 23, the control unit 24 transmits a notification indicating the completion of the update to the data generating device 30 via the communication unit 21.

[0121] In the process of step S47, the control unit 34 receives, via the communication unit 31, a notification from the information processing device 20 indicating the end of the update.

[0122] In the processing of step S48, the control unit 34 extracts medical images from the training dataset 33 that are associated with only the type of lesion received from the information processing device 20, i.e., the type of lesion determined by the trained model 23.

[0123] In the process of step S49, the control unit 34 transmits, via the communication unit 21, to the information processing device 20, the data of the medical image extracted in the process of step S48 and information indicating the identifier of the medical image.

[0124] By executing the process of step S49, in the information processing device 20, the control unit 24 can receive the data of the medical image and the identifier of the medical image via the communication unit 21. When the control unit 24 receives the data of the medical image, etc., it determines the type of lesion in the medical image using the updated trained model 23 and assigns a degree of confidence to the determined type of lesion. The control unit 24 transmits information indicating the identifier of the medical image, information indicating the determined type of lesion, and information indicating the degree of confidence assigned to the type of lesion to the data generating device 30 via the communication unit 21.

[0125] In the process of step S50, the control unit 34 receives, via the communication unit 31, from the information processing device 20, information indicating the identifier of the medical image, information indicating the type of lesion, and information indicating the degree of confidence.

[0126] In the process of step 51, the control unit 34 updates the data already stored in the learning dataset 33 based on the information indicating the type of lesion received in the process of step 50 and the like.

[0127] After executing the process of step S50, the control unit 34 may proceed to the process of step S22 as shown in FIG. 4, instead of proceeding to the process of step S51.

[0128] Furthermore, if there is no medical image in the learning dataset 33 that is associated only with the identifier of the first operator, the control unit 34 does not need to execute the processes of steps S40 and S41.

[0129] Furthermore, if there is no medical image associated with both the identifier of the first operator and the identifier of the second operator in the learning dataset 33, the control unit 34 does not need to execute the processes of steps S42 to S45.

[0130] In this way, the data generating device 30 according to the third embodiment generates an update data set for updating the trained model 23 based on at least one of the type of lesion received from the first terminal device and the type of lesion received from the second terminal device. By updating the trained model 23, the accuracy of the trained model 23 in determining the type of lesion can be improved. By improving the accuracy of the trained model 23 in determining the type of lesion, the accuracy of the trained data set 33 can be improved.

[0131] The present disclosure is not limited to the above-described embodiments. For example, multiple blocks shown in the block diagram may be integrated, or one block may be divided. Instead of executing multiple steps shown in the flowchart in chronological order as described, each step may be executed in parallel or in a different order depending on the processing capacity of the device executing each step, or as necessary. Other modifications are possible without departing from the spirit of the present disclosure.

[0132] For example, in the above-described first to third embodiments, the information processing device 20 and the data generating device 30 have been described as separate devices as shown in FIG. 1. However, the information processing device 20 and the data generating device 30 may be configured as one device or one cloud server. When the information processing device 20 and the data generating device 30 are configured as one device or cloud server, the trained model 23 may be stored in the storage unit 32 of the data generating device 30. In addition, the control unit 34 may determine the type of lesion in the medical image using the trained model 23, and assign a degree of confidence to the determined type of lesion.

[0133] For example, in the above-described first embodiment, the threshold has been described as being appropriately determined according to the amount of learning of the trained model 23. However, when the trained model 23 is updated by machine learning using an updated dataset as described above in the third embodiment, the threshold may be appropriately set based on the amount of data in the updated dataset. For example, the threshold may be smaller as the amount of data in the updated dataset increases. [Explanation of symbols]

[0134] 1 Data Generation System 2 Network 10, 10A, 10B, 10C Terminal equipment 11 Communications Department 12 Display section 13 Input section 14 Storage section 15 Control section 20 Information processing device 21 Communications Department 22 Memory section 23 Pre-trained models 24 Control Unit 30 Data Generation Device 31 Communications Department 32 Storage section 33 Training Data Set 34 Control section

Claims

1. A data generating device, The data generating device includes: Obtaining a medical image, a type of lesion in the medical image determined by a trained model, and a confidence level assigned to the type of lesion; When it is determined that the confidence level exceeds a threshold, the medical image is stored in a learning dataset in association with the type of the lesion; When it is determined that the degree of confidence is equal to or less than the threshold, the data of the medical image is transmitted to a first terminal device; When it is determined that the type of lesion received from the first terminal device matches the type of lesion in the medical image determined by the trained model, the data generation device associates the medical image with the type of lesion received from the first terminal device and stores it in a training dataset.

2. 2. The data generating device according to claim 1, The data generating device includes: When it is determined that the type of the lesion received from the first terminal device does not match the type of the lesion in the medical image determined by the trained model, transmitting data of the medical image to a second terminal device different from the first terminal device; A data generating device that stores the medical image in a learning dataset in association with the type of lesion in the medical image determined by a trained model, the type of lesion received from the first terminal device, and the type of lesion received from the second terminal device.

3. 3. The data generating device according to claim 2, The data generation device generates an update dataset for updating the trained model based on at least one of the type of lesion received from the first terminal device and the type of lesion received from the second terminal device.

4. 4. The data generating device according to claim 3, When the data generating device receives the type of lesion for the medical image only from the first terminal device, the data generating device associates the medical image with the type of lesion and a first weighting coefficient and stores the medical image in the updated data set.

5. The data generating device according to claim 4, When the data generating device receives a type of lesion from both the first terminal device and the second terminal device for the same medical image, and determines that the type of lesion received from the first terminal device does not match the type of lesion received from the second terminal device, the data generating device associates the medical image with the type of lesion received from each of the first terminal device and the second terminal device and a second weighting coefficient smaller than the first weighting coefficient, and stores the medical image in the updated data set.

6. A step of a data generating device acquiring a medical image, a type of lesion in the medical image determined by a trained model, and a confidence level assigned to the type of lesion; When the data generating device determines that the confidence level exceeds a threshold, storing the medical image in a learning dataset in association with the type of the lesion; transmitting data of the medical image to a first terminal device when the data generating device determines that the degree of confidence is equal to or less than the threshold; When the data generating device determines that the type of lesion received from the first terminal device matches the type of lesion in the medical image determined by the trained model, the data generating device associates the medical image with the type of lesion received from the first terminal device and stores the medical image in a training dataset. A data generation method comprising:

Images