Non-transitory computer-readable storage medium, information processing apparatus, learning method, and inference method

Abstract

To provide a learning program, an inference program, an information processing device, a learning method, and an inference method, capable of diagnosing dysphagia by a swallowing endoscopy.SOLUTION: Machine learning of a plurality of pieces of first training data each including first image data constituting first moving image data about the inside of the body captured by an endoscope camera, first position information indicating a position of a target of at least one of a site in the body and a test meal appearing in the first image data, and a first label indicating a type of the target corresponding to the first position information; A first learning model that outputs second position information of a target appearing in second image data and a second label indicating a type of the target appearing in the second image data is generated in accordance with input of the second image data constituting the second moving image data regarding the inside of a body captured by an endoscopic camera, and the generated first learning model is stored in a storage unit.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a learning program, an inference program, an information processing device, a learning method, and an inference method. [Background technology]

[0002] For example, in Japan, which has the highest aging rate in the world, there are many elderly people with swallowing disorders, and the importance of swallowing endoscopy for diagnostic purposes is increasing significantly (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2022-179220 Summary of the Invention [Problem to be solved by the invention]

[0004] The above-mentioned swallowing endoscopy examination is preferably performed by a swallowing specialist (hereinafter simply referred to as a specialist) because, for example, diagnosing swallowing disorders is highly difficult. However, depending on the region, it may not be possible to secure a sufficient number of such specialists.

[0005] Therefore, one disclosure provides a learning program, an inference program, an information processing device, a learning method, and an inference method that enable diagnosis of swallowing disorders through swallowing endoscopy. [Means for solving the problem]

[0006] The learning program of the present disclosure generates a first learning model by machine learning multiple first training data, each of which includes first image data constituting first video data of the inside of the body captured by an endoscopic camera, first position information indicating the position of at least one of an area within the body and a test meal captured in the first image data, and a first label indicating the type of the object corresponding to the first position information, in response to input of second image data constituting second video data of the inside of the body captured by an endoscopic camera, and outputs second position information of the object captured in the second image data and a second label indicating the type of the object captured in the second image data, and stores the generated first learning model in a memory unit. [Effects of the Invention]

[0007] The learning program, inference program, information processing device, learning method, and inference method disclosed herein make it possible to diagnose swallowing disorders using a swallowing endoscopy. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an information processing system 100 according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating the hardware configuration of the information processing device 1 according to the first embodiment. [Figure 3] FIG. 3 is a block diagram of the functions of the information processing device 1 according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating the relationship between the first teacher data DT1 and the first learning model MD1. [Figure 5] FIG. 5 is a diagram illustrating the relationship between the second teacher data DT2 and the second learning model MD2. [Figure 6] FIG. 6 is a diagram illustrating the relationship between the third teacher data DT3 and the third learning model MD3. [Figure 7] FIG. 7 is a diagram illustrating the relationship between the fourth training data DT4 and the fourth learning model MD4. [Figure 8]FIG. 8 is a flowchart illustrating the learning process according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating a specific example of the first teacher data DT1. [Figure 10] FIG. 10 is a diagram illustrating a specific example of the first learning model MD1. [Figure 11] FIG. 11 is a diagram illustrating a specific example of the second teacher data DT2. [Figure 12] FIG. 12 is a diagram illustrating a specific example of the second learning model MD2. [Figure 13] FIG. 13 is a diagram illustrating a specific example of the third teacher data DT3. [Figure 14] FIG. 14 is a diagram illustrating a specific example of the third learning model MD3. [Figure 15] FIG. 15 is a diagram illustrating a specific example of the fourth teacher data DT4. [Figure 16] FIG. 16 is a diagram illustrating a specific example of the fourth teacher data DT4. [Figure 17] FIG. 17 is a diagram illustrating a specific example of the fourth learning model MD4. [Figure 18] FIG. 18 is a diagram illustrating a specific example of the fourth teacher data DT4. [Figure 19] FIG. 19 is a diagram illustrating a specific example of the fourth teacher data DT4. [Figure 20] FIG. 20 is a flowchart illustrating the inference process according to the first embodiment. [Figure 21] FIG. 21 is a diagram illustrating the inference process according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. However, such descriptions should not be interpreted in a limiting sense, and do not limit the subject matter described in the claims. Furthermore, various changes, substitutions, and modifications can be made without departing from the spirit and scope of the present disclosure. Furthermore, different embodiments can be combined as appropriate.

[0010] [Information Processing System 100 in the First Embodiment] First, a description will be given of an example of the configuration of an information processing system 100 according to the first embodiment. Fig. 1 is a diagram illustrating the configuration of the information processing system 100 according to the first embodiment.

[0011] As shown in FIG. 1, the information processing system 100 includes, for example, an information processing device 1, an endoscopic camera 2, and an operation terminal 5.

[0012] The information processing device 1 is, for example, a physical machine or a virtual machine. The endoscopic camera 2 is, for example, a camera that captures video data (not shown) of the inside of the body of the subject OB. The operation terminal 5 is, for example, a mobile terminal such as a PC (Personal Computer) or a smartphone, and is a terminal through which a practitioner (hereinafter simply referred to as a practitioner) who diagnoses dysphagia in the subject OB inputs necessary information. The information processing device 1 and the operation terminal 5 can access each other, for example, via a network NW such as the Internet (hereinafter simply referred to as a network NW). The information processing device 1 and the endoscopic camera 2 can access each other, for example, via the network NW.

[0013] The endoscopic camera 2 captures video data of, for example, the inside of the oral cavity of the subject OB. Specifically, the endoscopic camera 2 captures video data showing, for example, the state of the larynx of the subject OB. Then, the endoscopic camera 2 transmits the captured video data to the information processing device 1. Thereafter, the information processing device 1 stores the video data transmitted from the endoscopic camera 2 in the storage unit 130, for example.

[0014] 1, the information processing system 100 may further include, for example, a video data management device 1a that acquires and stores video data captured by the endoscopic camera 2. In this case, the information processing device 1 may acquire, for example, the video data stored in the video data management device 1a.

[0015] Furthermore, the video data captured by the endoscopic camera 2 may be stored in the storage unit 130 manually by an operator, for example.

[0016] The information processing device 1 performs, for example, a process of generating a learning model capable of making inferences about dysphagia for the subject OB (hereinafter also referred to as a learning process).

[0017] Specifically, the information processing device 1 in this embodiment stores a plurality of training data (hereinafter also referred to as first training data) each including, for example, image data (hereinafter also referred to as first image data) constituting video data (hereinafter also referred to as first video data) of the inside of the body of the subject OB photographed by the endoscopic camera 2, position information (hereinafter also referred to as first position information) indicating the position of at least one of an internal body part and a test meal (hereinafter also simply referred to as the target), which is photographed in the first image data, and a label (hereinafter also referred to as first label) indicating the type of the target corresponding to the first position information. By machine learning of the above-mentioned data, a learning model (hereinafter also referred to as a first learning model) is generated that outputs an inference result of other position information (hereinafter also referred to as second position information) of at least one of an internal body part and a test meal (hereinafter also referred to as another object) shown in the second image data, which constitutes other video data (hereinafter also referred to as second video data) about the inside of the body of the subject OB captured by the endoscopic camera 2. The learning model outputs an inference result of other position information (hereinafter also referred to as second position information) of at least one of an internal body part and a test meal (hereinafter also referred to as another object) shown in the second image data, and another label (hereinafter also referred to as a second label) indicating the type of the other object shown in the second image data. Then, the information processing device 1 in this embodiment stores the generated first learning model in the storage unit 130, for example.

[0018] That is, the information processing device 1 in this embodiment generates a first learning model, which is an instance segmentation model that can predict position information such as coordinates for the position of an object (part or test meal) inside the body shown in image data and a label indicating the name, etc.

[0019] As a result, the information processing device 1 in this embodiment can, for example, output each piece of image data constituting video data in a manner that corresponds to the output results (inference results of location information and labels) continuously output from the first learning model in response to the continuous input of each piece of image data constituting the video data. Specifically, the information processing device 1 in this embodiment can, for example, output the output results from the first learning model superimposed on each piece of image data. Therefore, when viewing video data, for example, a clinician can also view information indicating the position of each part of the subject OB's body and the position of each test meal, making it easier to diagnose swallowing disorders for the subject OB and preventing misdiagnosis of swallowing disorders. Therefore, the information processing device 1 in this embodiment can diagnose swallowing disorders for the subject OB even if the clinician is, for example, a doctor or nurse who is not a specialist in swallowing diagnosis.

[0020] Furthermore, the information processing device 1 in this embodiment generates a learning model (hereinafter also referred to as the second learning model) that outputs an inference result of the depth information (hereinafter also referred to as the second depth information) that indicates the depth of the position of the test meal in the second image data when the second image data is input, by machine learning of a plurality of teacher data (hereinafter also referred to as the second teacher data), each including the first image data and depth information (hereinafter also referred to as the first depth information) that indicates the depth of the position of the test meal in the body shown in the first image data. Furthermore, the information processing device 1 in this embodiment generates a learning model (hereinafter also referred to as the third learning model) that outputs an inference result of the label (hereinafter also referred to as the fourth label) that indicates the occurrence of aspiration in the body of the subject OB shown in the second image data when the second image data and the second depth information are input, by machine learning of a plurality of teacher data (hereinafter also referred to as the third teacher data), each including the first image data, the first depth information, and a label (hereinafter also referred to as the third label) that indicates the occurrence of aspiration in the body of the subject OB shown in the first image data. Then, the information processing device 1 in this embodiment stores the generated second learning model and third learning model in the storage unit 130, for example.

[0021] That is, whether or not aspiration has occurred in the body of the subject OB can be determined to be correlated with, for example, the position (depth) of the test meal in the body (trachea) of the subject OB. Therefore, the information processing device 1 in this embodiment generates a second learning model and a third learning model that can accurately infer whether or not aspiration has occurred in the body of the subject OB by using, for example, training data including depth information about the test meal shown in the image data.

[0022] As a result, the information processing device 1 in this embodiment allows a medical practitioner to view information indicating whether or not aspiration is occurring in the body of the subject OB. Therefore, the medical practitioner can use the information output from the information processing device 1 as auxiliary information when diagnosing whether or not aspiration is occurring in the body of the subject OB, for example, making it easier to diagnose whether or not aspiration is occurring and preventing an erroneous diagnosis of whether or not aspiration is occurring. Therefore, with the information processing device 1 in this embodiment, even if the medical practitioner is, for example, a doctor or nurse who is not a specialist in swallowing diagnosis, it is possible to diagnose whether or not aspiration is occurring in the body of the subject OB.

[0023] Furthermore, the information processing device 1 in this embodiment generates a learning model (hereinafter also referred to as the fourth learning model) that outputs a label (hereinafter also referred to as the sixth label) that indicates the symptom of dysphagia in the body of the subject OB shown in the second image data in response to input of the second image data, for example, by machine learning of a plurality of training data (hereinafter also referred to as the fourth training data), each of which includes the first image data and a label (hereinafter also referred to as the fifth label) related to the symptom of dysphagia in the body of the subject OB shown in the first image data. Then, the information processing device 1 in this embodiment stores the generated third learning model in the storage unit 130, for example.

[0024] That is, the information processing device 1 in this embodiment generates, for example, a fourth learning model that outputs information indicating symptoms of dysphagia in the subject OB in response to input of image data. Specifically, the information processing device 1 in this embodiment generates the fourth learning model by, for example, performing contrastive learning.

[0025] As a result, the information processing device 1 in this embodiment allows, for example, a medical practitioner to view information indicating the state of dysphagia in the body of the subject OB. Therefore, the medical practitioner can use, for example, the information output from the information processing device 1 as auxiliary information when diagnosing the state of dysphagia, making it easier to diagnose the state of dysphagia and preventing misdiagnosis of the state of dysphagia. Therefore, with the information processing device 1 in this embodiment, even if the medical practitioner is, for example, a doctor or nurse who is not a specialist in swallowing treatment, it is possible to diagnose the state of dysphagia in the body of the subject OB.

[0026] The second video data used in each of the first to fourth learning models may be, for example, video data captured by an endoscopic camera (not shown) other than the endoscopic camera 2. The second video data used in each of the first to fourth learning models may be, for example, video data captured of the inside of a subject (not shown) other than the subject OB. The first video data used in each of the first to fourth learning models may be different video data from each other. The second video data used in each of the first to fourth learning models may be different video data from each other.

[0027] [Information processing device 1 according to the first embodiment] Next, a description will be given of the configuration of the information processing device 1 in the first embodiment. Fig. 2 is a diagram illustrating the hardware configuration of the information processing device 1 in the first embodiment.

[0028] 2, the information processing device 1 is a computer device having, for example, a CPU 101 which is a processor, a memory 102, a communication device 103, and a storage medium 104. Each unit is connected to each other via, for example, a bus 105.

[0029] The storage medium 104 has a program storage area (not shown) that stores a program 110 for performing, for example, the learning process of each learning model and the process of performing inference using each learning model (hereinafter simply referred to as the inference process). Hereinafter, the learning process and the inference process will be collectively referred to simply as the learning process, etc. The storage medium 104 also has, for example, an information storage area 130 that stores various pieces of information used when performing the learning process, etc. The storage medium 104 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0030] The CPU 101 performs learning processing and the like by executing a program 110 loaded from the storage medium 104 into the memory 102, for example.

[0031] The communication device 103 accesses, for example, the operation terminal 5 via the network NW.

[0032] The information processing device 1 may include, for example, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a graphical processing unit (GPU). The learning process and the like may be executed in, for example, the FPGA, the ASIC, or the GPU.

[0033] Furthermore, the following description will be given assuming that the information processing system 100 has one information processing device 1, but the present invention is not limited to this. Specifically, the information processing system 100 may have, for example, a plurality of information processing devices 1. The learning process and the like may be performed, for example, in a distributed manner among the plurality of information processing devices 1. Specifically, the learning process and the inference process may each be executed, for example, in different information processing devices 1.

[0034] [Functions of information processing device 1] Next, the functions of the information processing device 1 will be described. FIG. 3 is a block diagram of the functions of the information processing device 1 in the first embodiment. FIG. 4 is a diagram illustrating the relationship between the first teacher data DT1 and the first learning model MD1. FIG. 5 is a diagram illustrating the relationship between the second teacher data DT2 and the second learning model MD2. FIG. 6 is a diagram illustrating the relationship between the third teacher data DT3 and the third learning model MD3. FIG. 7 is a diagram illustrating the relationship between the fourth teacher data DT4 and the fourth learning model MD4.

[0035] As shown in FIG. 3, the information processing device 1 realizes various functions including an information management unit 111, a data generation unit 112, a model generation unit 113, an information inference unit 114, and an information output unit 115 by organically cooperating with hardware such as a CPU 101 and a memory 102 and programs.

[0036] In addition, as shown in Figure 3, the information processing device 1 stores, for example, first teacher data DT1, second teacher data DT2, third teacher data DT3, fourth teacher data DT4, first learning model MD1, second learning model MD2, third learning model MD3, and fourth learning model MD4 in the information storage area 130.

[0037] First, the function for realizing the learning process will be described.

[0038] The information management unit 111 stores, for example, various data input by the worker via the operation terminal 5 in the information storage area 130.

[0039] Specifically, for each piece of image data DT11 constituting video data of the inside of the body of the subject OB photographed by the endoscopic camera 2, the information management unit 111 stores in the information storage area 130 each piece of image data DT11, position information DT12 indicating the position of the object shown in each piece of image data DT11 (the position of the object in each piece of image data DT11), a label DT13 indicating the type of object shown in each piece of image data DT11, depth information DT14 indicating the depth of the position of the test meal shown in each piece of image data DT11, a label DT15 indicating whether or not aspiration has occurred inside the body of the subject OB shown in each piece of image data DT11, and a label DT16 relating to the symptoms of swallowing disorders inside the body of the subject OB shown in each piece of image data DT11.

[0040] The label DT15 may be, for example, a label indicating the severity of aspiration in the body of the subject OB shown in each image data DT11. The label DT16 may be, for example, a label indicating the severity of dysphagia (dysphagia including aspiration) in the body of the subject OB shown in each image data DT11.

[0041] 4, for example, for each piece of image data DT11 (image data DT11 stored in the information storage area 130) constituting video data of the inside of the body of the subject OB photographed by the endoscopic camera 2, the data generation unit 112 generates a plurality of pieces of first teacher data DT1 each including the image data DT11, position information DT12 indicating the position of the object (site in the body of the subject OB or the position of the test meal) shown in each piece of image data DT11, and a label DT13 indicating the type of object shown in each piece of image data DT11. Then, the information management unit 111 stores the plurality of pieces of first teacher data DT1 generated by the data generation unit 112 in the information storage area 130 (hereinafter also referred to as information storage area 130a), for example.

[0042] 5, the data generation unit 112 generates, for example, for each piece of image data DT11 (image data DT11 stored in the information storage area 130) constituting video data of the inside of the body of the subject OB photographed by the endoscopic camera 2, a plurality of pieces of second teacher data DT2 each including each piece of image data DT11 and depth information DT14 indicating the depth of the position of the test meal within the body captured in each piece of image data DT11. Then, the information management unit 111 stores, for example, the plurality of pieces of second teacher data DT2 generated by the data generation unit 112 in the information storage area 130a.

[0043] 6, for example, for each piece of image data DT11 (image data DT11 stored in the information storage area 130) constituting video data of the inside of the body of the subject OB photographed by the endoscopic camera 2, the data generation unit 112 generates a plurality of pieces of teacher data DT3 each including the image data DT11, depth information DT14 indicating the depth of the position of the test meal within the body captured in each piece of image data DT11, and a label DT15 indicating whether or not aspiration has occurred within the body of the subject OB captured in each piece of image data DT11. Then, the information management unit 111 stores the plurality of pieces of teacher data DT3 generated by the data generation unit 112 in the information storage area 130a, for example.

[0044] 7, the data generation unit 112 generates, for example, for each piece of image data DT11 (image data DT11 stored in the information storage area 130) constituting video data of the inside of the body of the subject OB photographed by the endoscopic camera 2, a plurality of pieces of teacher data DT4 each including each piece of image data DT11 and a label DT16 relating to the symptoms of dysphagia inside the body of the subject OB captured in each piece of image data DT11. Then, the information management unit 111 stores, for example, the plurality of pieces of teacher data DT4 generated by the data generation unit 112 in the information storage area 130a.

[0045] For example, when generating the plurality of teacher data DT4, the data generating unit 112 may divide video data of the inside of the subject OB captured by the endoscopic camera 2 into a plurality of video data. Specifically, the data generating unit 112 may divide the video data into a plurality of video data including, for example, video data corresponding to a time period when swallowing is occurring in the body of the subject OB (hereinafter also referred to as third video data) and video data corresponding to a time period when swallowing is not occurring in the body of the subject OB (hereinafter also referred to as fourth video data). Then, for example, the data generating unit 112 may generate a plurality of teacher data DT4 for each of the plurality of video data, each of which includes a plurality of image data DT11 constituting each video data and a label DT16 related to the symptoms of dysphagia in the body of the subject OB captured in each video data.

[0046] Furthermore, the information management unit 111 may be configured, for example, to store at least one of the first teacher data DT1, second teacher data DT2, third teacher data DT3, and fourth teacher data DT4 in the information storage area 130a when an operator directly inputs at least one of the first teacher data DT1, second teacher data DT2, third teacher data DT3, and fourth teacher data DT4 via the operation terminal 5.

[0047] 4, the model generation unit 113 generates a first learning model MD1 by machine learning of a plurality of first teacher data DT1 stored in the information storage area 130a, for example. Then, the information management unit 111 stores the first learning model MD1 generated by the model generation unit 113 in the information storage area 130 (hereinafter also referred to as the information storage area 130b).

[0048] 5, the model generation unit 113 generates a second learning model MD2 by machine learning of the plurality of second teacher data DT2 stored in the information storage area 130a. Then, the information management unit 111 stores the second learning model MD2 generated by the model generation unit 113 in the information storage area 130b.

[0049] 6, the model generation unit 113 generates a third learning model MD3 by machine learning of a plurality of third teacher data DT3 stored in the information storage area 130a. Then, the information management unit 111 stores the third learning model MD3 generated by the model generation unit 113 in the information storage area 130b.

[0050] 7, the model generation unit 113 generates a fourth learning model MD4 by machine learning of a plurality of fourth teacher data DT4 stored in the information storage area 130a. Then, the information management unit 111 stores the fourth learning model MD4 generated by the model generation unit 113 in the information storage area 130b.

[0051] Next, the function for realizing the inference process will be described.

[0052] The information inference unit 114 inputs, for example, new image data DT11 constituting new video data (not shown) about the inside of the subject OB captured by the endoscopic camera 2 to the first learning model MD1. Then, the information inference unit 114 acquires, for example, new position information DT12 and new label DT13 output from the first learning model MD1.

[0053] The information inference unit 114 also inputs, for example, new image data DT11 constituting new video data about the inside of the subject OB captured by the endoscopic camera 2 to the second learning model MD2. Then, the information inference unit 114 acquires, for example, new depth information DT14 output from the second learning model MD2.

[0054] Furthermore, the information inference unit 114 inputs the new image data DT11 and new depth information DT14 corresponding to the new image data DT11 (the new depth information DT14 output from the second learning model MD2) to the third learning model MD3 for each new image data DT11 constituting new video data about the inside of the subject OB photographed by the endoscopic camera 2. Then, the information inference unit 114 acquires, for example, a new label DT15 output from the third learning model MD3.

[0055] The information inference unit 114 also inputs, for example, each of the new image data DT11 constituting new video data about the inside of the subject OB captured by the endoscopic camera 2 to the fourth learning model MD4 stored in the information storage area 130. Then, the information inference unit 114 acquires, for example, a new label DT16 output from the fourth learning model MD4.

[0056] The information output unit 115 outputs the information acquired by the information inference unit 114 to the operation terminal 5, for example.

[0057] Specifically, the information output unit 115 outputs, for example, information indicating the new position information DT12 and the new label DT13 that the information inference unit 114 has acquired from the first learning model MD1.

[0058] Furthermore, the information output unit 115 outputs, for example, information indicating new depth information DT14 that the information inference unit 114 has acquired from the second learning model MD2.

[0059] Furthermore, the information output unit 115 outputs, for example, information indicating the new label DT15 that the information inference unit 114 has acquired from the third learning model MD3.

[0060] Furthermore, the information output unit 115 outputs, for example, information indicating the new label DT16 that the information inference unit 114 has acquired from the fourth learning model MD4.

[0061] [Learning Process in the First Embodiment] Next, the learning process in the first embodiment will be described. FIG. 8 is a flowchart illustrating the learning process in the first embodiment. Also, FIGS. 9 to 18 are diagrams illustrating the learning process in the first embodiment. Note that, hereinafter, the first teacher data DT1, the second teacher data DT2, the third teacher data DT3, and the fourth teacher data DT4 will be collectively referred to simply as teacher data DT. Also, hereinafter, the first learning model MD1, the second learning model MD2, the third learning model MD3, and the fourth learning model MD4 will be collectively referred to simply as learning models MD.

[0062] 8, the model generation unit 113 waits until the learning timing comes (NO in S11), for example. The learning timing may be the timing when the operator inputs information indicating that the learning process will be performed.

[0063] Then, when it is time to learn (YES in S11), the model generation unit 113 generates a learning model MD, for example, by machine learning of the teacher data DT (for example, the teacher data DT generated in advance by the data generation unit 112) stored in the information storage area 130 (S12).

[0064] Specifically, the model generation unit 113 generates a first learning model MD1 by machine learning of, for example, first teacher data DT1 stored in the information storage area 130 (for example, first teacher data DT1 generated in advance by the data generation unit 112).

[0065] In addition, the model generation unit 113 generates a second learning model MD2, for example, by machine learning of second teacher data DT2 stored in the information storage area 130 (for example, second teacher data DT2 generated in advance by the data generation unit 112).

[0066] In addition, the model generation unit 113 generates a third learning model MD3, for example, by machine learning of third teacher data DT3 stored in the information storage area 130 (for example, third teacher data DT3 generated in advance by the data generation unit 112).

[0067] In addition, the model generation unit 113 generates a fourth learning model MD4, for example, by machine learning of the fourth teacher data DT4 stored in the information storage area 130 (for example, the fourth teacher data DT4 generated in advance by the data generation unit 112).

[0068] In addition, the model generation unit 113 may generate, for example, at least one of the first learning model MD1, the second learning model MD2, the third learning model MD3, and the fourth learning model MD4 in S12. Specific examples of the training data DT will be described below.

[0069] [Specific example of first training data DT1 and first learning model MD1] First, specific examples of the first teacher data DT1 and the first learning model MD1 will be described. Fig. 9 is a diagram illustrating a specific example of the first teacher data DT1. Fig. 10 is a diagram illustrating a specific example of the first learning model MD1.

[0070] The first teacher data DT1 shown in Figure 9 has, for example, the following items: "image data" in which information identifying the image data DT11 is set, "location information" in which multiple coordinates indicating the range in which the object appears in the image data DT11 (for example, multiple coordinates for multiple pixels corresponding to the location where the object is located) are set, and "label" in which a label indicating the type of object appearing in the image data DT11 is set.

[0071] Specifically, as shown in Figure 9, the data in the first row of the first teacher data DT1 has, for example, "IM01" set as "image data," "(24,42)" or the like set as "location information," and "epiglottis," a body part, set as "label."

[0072] Furthermore, as shown in Figure 9, the data in the second row of the first teacher data DT1 has, for example, "IM01" set as "image data," "(41,25)" or the like set as "location information," and "subglottis," a body part, set as "label."

[0073] Furthermore, as shown in Figure 9, the data in the fourth row of the first teacher data DT1 has, for example, "IM01" set as the "image data" and "(25,49)" or the like set as the "location information," and "test meal" indicating the test meal set as the "label."

[0074] 9, for example, "IM02" is set as "image data," "(22,31)" is set as "position information," and "epiglottis," a body part, is set as "label." Explanation of other data included in FIG. 9 will be omitted.

[0075] Then, in step S12, the model generation unit 113 generates the first learning model MD1 shown in FIG. 10 by performing machine learning on the plurality of first teacher data DT1 shown in FIG. 9, for example.

[0076] Specifically, as shown in FIG. 10, the model generation unit 113 generates a first learning model MD1 including, for example, "Spatial Attention," "Channel Attention," and "Feature Pyramid."

[0077] Note that when multiple objects appear in one image data DT11, the data generation unit 112 may, for example, generate multiple pieces of first teacher data DT1 from one piece of image data DT11. That is, the data generation unit 112 may, for example, generate multiple pieces of first teacher data DT1 for each object appearing in each image data DT11, each including image data DT11 in which each object appears, position information DT12 corresponding to each object, and a label DT13 indicating the type of each object.

[0078] [Specific example of second training data DT2 and second learning model MD2] Next, specific examples of the second teacher data DT2 and the second learning model MD2 will be described. Fig. 11 is a diagram illustrating a specific example of the second teacher data DT2. Fig. 12 is a diagram illustrating a specific example of the second learning model MD2.

[0079] The second teacher data DT2 shown in Figure 11 has, for example, the following items: "image data" in which information identifying the image data DT11 is set; "position information" in which multiple coordinates indicating the position where the test meal is shown in the image data DT11 (multiple coordinates for multiple pixels corresponding to the location where the test meal is present) are set; and "depth information" in which the depth of the position of the test meal shown in the image data DT11 (the distance from the endoscopic camera 2 for each of multiple pixels) is set.

[0080] Specifically, as shown in Figure 11, the data in the first row of the second teacher data DT2 has, for example, "IM01" set as "image data," "(28,35)" or the like set as "position information," and "5" or the like set as "depth information."

[0081] 11, for example, "IM02" is set as "image data," "(40, 43)" is set as "position information," and "19" is set as "depth information." Explanation of the other data included in FIG. 11 will be omitted.

[0082] That is, the second teacher data DT2 shown in Figure 11 indicates, for example, that the depth of the test meal in image data DT11 having identification information "IM02" is deeper than the depth of the test meal in image data DT11 having identification information "IM01".

[0083] Then, in step S12, the model generation unit 113 generates the second learning model MD2 shown in FIG. 12 by performing machine learning on the plurality of second teacher data DT2 shown in FIG. 11, for example.

[0084] When multiple test meals are shown in one image data set DT11, the data generating unit 112 may, for example, generate multiple pieces of second teacher data DT2 from one image data set DT11. That is, the data generating unit 112 may, for example, generate multiple pieces of second teacher data DT2 for each test meal shown in each image data set DT11, each of which includes image data DT11 showing each test meal, position information DT12 corresponding to each test meal, and depth information DT14 indicating the depth of each test meal.

[0085] [Specific example of the third training data DT3 and the third learning model MD3] Next, specific examples of the third teacher data DT3 and the third learning model MD3 will be described. Fig. 13 is a diagram illustrating a specific example of the third teacher data DT3. Fig. 14 is a diagram illustrating a specific example of the third learning model MD3.

[0086] 13 includes, for example, the following items: "image data" in which information identifying the image data DT11 is set; "position information" in which a plurality of coordinates indicating the position of the test meal in the image data DT11 (a plurality of coordinates for a plurality of pixels corresponding to the location of the test meal) is set; "depth information" in which the depth of the position of the test meal in the image data DT11 (the distance from the endoscopic camera 2 for each of the plurality of pixels) is set; and an "aspiration occurrence label" indicating whether or not aspiration has occurred in the area shown in the image data DT11. The "aspiration occurrence label" is set, for example, as "x" indicating that aspiration has occurred in the area shown in the image data DT11, or as "o" indicating that aspiration has not occurred in the area shown in the image data DT11.

[0087] Specifically, as shown in Figure 13, the data in the first row of the third teacher data DT3 has, for example, "IM01" set as "image data," "(28,35)" or the like set as "position information," "5" or the like set as "depth information," and "x" set as "aspiration occurrence label."

[0088] 13, for example, "IM02" is set as "image data," "(40, 43)" or the like is set as "position information," "19" or the like is set as "depth information," and "◯" is set as "aspiration occurrence label." Explanation of other data included in FIG. 13 will be omitted.

[0089] Then, in step S12, the model generation unit 113 generates the third learning model MD3 shown in FIG. 14 by performing machine learning on the plurality of third teacher data DT3 shown in FIG. 13, for example.

[0090] When multiple test meals are shown in one image data DT11, the data generating unit 112 may, for example, generate multiple pieces of third teacher data DT3 from one piece of image data DT11. That is, the data generating unit 112 may, for each test meal shown in each image data DT11, generate multiple pieces of third teacher data DT3 each including image data DT11 showing each test meal, position information DT12 corresponding to each test meal, depth information DT14 indicating the depth of each test meal, and a label DT15 indicating whether or not aspiration has occurred in the area shown in the image data DT11 showing each test meal.

[0091] [Specific example (1) of the fourth training data DT4 and the fourth learning model MD4] Next, specific examples of the fourth teacher data DT4 and the fourth learning model MD4 will be described. Figures 15 and 16 are diagrams for explaining specific examples of the fourth teacher data DT4. Figure 17 is a diagram for explaining a specific example of the fourth learning model MD4.

[0092] The fourth teacher data DT4 (hereinafter also referred to as fourth teacher data DT4a) shown in FIG. 15 has, for example, items "image data (1)" in which information identifying the image data DT11 is set, and "image data (2)" in which information identifying other image data DT11 is set. The "image data (1)" and "image data (2)" are set, for example, as a combination (hereinafter also referred to as a first combination) of image data DT11 constituting video data corresponding to a time period when swallowing is occurring in the body of the subject OB and image data DT11 constituting video data corresponding to a time period when swallowing is not occurring in the body of the subject OB. The "image data (1)" and "image data (2)" are set, for example, as a combination (hereinafter also referred to as a second combination) of image data DT11 constituting video data corresponding to a time period when swallowing is occurring in the body of the subject OB and other image data DT11 constituting video data corresponding to a time period when swallowing is occurring in the body of the subject OB. That is, the fourth teacher data DT4a shown in FIG. 15 is, for example, so-called unlabeled teacher data.

[0093] Specifically, as shown in Figure 15, the data in the first row of the fourth teacher data DT4a has, for example, "IM01" set as "image data (1)" and "IM11" set as "image data (2)".

[0094] In addition, in the data in the second row of the fourth teacher data DT4a, for example, "IM02" is set as "image data (1)" and "IM12" is set as "image data (2)" as shown in Fig. 15. Explanation of the other data included in Fig. 15 will be omitted.

[0095] 16 (hereinafter also referred to as fourth teacher data DT4b) has, for example, the following items: "Image Data (Normal)" in which information identifying image data DT11 at a timing when no laryngeal invasion or aspiration occurs in the body of the subject OB (i.e., a timing when normal swallowing occurs); "Image Data (Laryngeal Invasion)" in which information identifying image data DT11 at a timing when laryngeal invasion occurs in the body of the subject OB; and "Image Data (Aspiration)" in which information identifying image data DT11 at a timing when aspiration occurs in the body of the subject OB. That is, the fourth teacher data DT4b shown in FIG. 16 is, for example, so-called labeled teacher data. Hereinafter, a combination of image data DT11 corresponding to each of "Image Data (Normal)," "Image Data (Laryngeal Invasion)," and "Image Data (Aspiration)" will also be referred to as a third combination.

[0096] Specifically, as shown in Figure 16, the data in the first row of the fourth teacher data DT4b has, for example, "IM01" set as "image data (normal)", "IM02" set as "image data (laryngeal invasion)", and "IM03" set as "image data (aspiration)".

[0097] In addition, as shown in Fig. 16, in the data in the second row of the fourth teacher data DT4b, for example, "IM11" is set as "image data (normal)," "IM12" is set as "image data (larynx invasion)," and "IM13" is set as "image data (aspiration)." Explanation of the other data included in Fig. 16 will be omitted.

[0098] In the example shown in FIG. 16, the fourth teacher data DT4b has the items "image data (normal)," "image data (laryngeal invasion)," and "image data (aspiration)," but this is not limited to this. Specifically, the fourth teacher data DT4b may have, for example, a plurality of items of at least one of "image data (normal)," "image data (laryngeal invasion)," and "image data (aspiration)." Furthermore, the fourth teacher data DT4b may not have any of the items "image data (normal)," "image data (laryngeal invasion)," and "image data (aspiration)." Furthermore, the fourth teacher data DT4b may have, for example, only a plurality of "image data (normal)" items, or only a plurality of "image data (laryngeal invasion)" items, or only a plurality of "image data (aspiration)" items. Hereinafter, when the fourth teacher data DT4b has only a plurality of "image data (normal)" items, the combination of image data DT11 corresponding to each of the plurality of "image data (normal)" items will also be referred to as a fourth combination.

[0099] Then, in step S12, the model generation unit 113 performs machine learning (contrastive learning) of multiple fourth teacher data DT4a shown in Figure 15, and then performs machine learning (fine tuning) of multiple fourth teacher data DT4b shown in Figure 16, thereby generating the fourth learning model MD4 shown in the lower part of Figure 17.

[0100] Specifically, the model generation unit 113 performs machine learning (contrastive learning) of the multiple fourth teacher data DT4a shown in Figure 15, for example, so that the distances (distances indicating similarity) between the multiple image data DT11 corresponding to the first combination become larger, and so that the distances (distances indicating similarity) between the multiple image data DT11 corresponding to the second combination become smaller.

[0101] Thereafter, the model generation unit 113 performs machine learning (contrastive learning) of the multiple fourth teacher data DT4b shown in FIG. 16, for example, so that the distances (distances indicating similarity) between the multiple image data DT11 corresponding to the third combination become larger, and so that the distances (distances indicating similarity) between the multiple image data DT11 corresponding to the fourth combination become smaller.

[0102] 15, the first row of data in the fourth teacher data DT4a includes one image data DT11 constituting video data corresponding to a time period when swallowing is occurring in the body of the subject OB, and one image data DT11 constituting video data corresponding to a time period when swallowing is not occurring in the body of the subject OB. However, this is not limiting. Specifically, the fourth teacher data DT4a may include, for example, two or more image data DT11 constituting video data corresponding to a time period when swallowing is occurring in the body of the subject OB, and two or more image data DT11 constituting video data corresponding to a time period when swallowing is not occurring in the body of the subject OB.

[0103] 15, the second row of the fourth teacher data DT4a includes two pieces of image data DT11 constituting video data corresponding to a time period during which swallowing occurs in the body of the subject OB, but this is not limiting. Specifically, the fourth teacher data DT4a may include, for example, three or more pieces of image data DT11 constituting video data corresponding to a time period during which swallowing occurs in the body of the subject OB.

[0104] Here, when swallowing occurs in the body of the subject OB, the entrance to the trachea of ​​the subject OB is momentarily blocked by, for example, the epiglottis. Therefore, when swallowing occurs in the body of the subject OB, the light irradiated from the endoscopic camera 2 is reflected by the epiglottis, and the proportion of predetermined white pixels in the image data DT11 is higher than when the light irradiated from the endoscopic camera 2 is not reflected by the epiglottis (when the light irradiated by the endoscopic camera 2 passes into the trachea). The predetermined white is, for example, a white color in which each of the RGB values ​​is equal to or greater than a predetermined threshold. In other words, it is possible to determine, for example, that the timing at which the proportion of predetermined white pixels in the image data DT11 is higher than at other times is the timing at which swallowing occurs in the body of the subject OB.

[0105] Therefore, the data generation unit 112 divides, for example, video data of the inside of the subject OB captured by the endoscopic camera 2 into video data (third video data) that includes image data DT11 (hereinafter also referred to as specific image data DT11) in which the proportion of a predetermined white pixel is equal to or greater than a predetermined threshold, and video data (fourth video data) that does not include the specific image data DT11.

[0106] Specifically, the data generating unit 112 identifies, for example, video data (video data corresponding to "swallowing" in the upper part of FIG. 17) captured within a predetermined time (e.g., several seconds) from the time when specific image data DT11 (image data corresponding to "Whiteout" in the upper part of FIG. 17) was captured among video data captured inside the body of the subject OB by the endoscopic camera 2 as third video data corresponding to a time period when swallowing is occurring inside the body of the subject OB. Furthermore, the data generating unit 112 identifies, for example, video data other than the third video data (video data corresponding to "waiting" in the upper part of FIG. 17) among video data captured inside the body of the subject OB by the endoscopic camera 2 as fourth video data corresponding to a time period when swallowing is not occurring inside the body of the subject OB. Then, the data generation unit 112 generates multiple fourth teacher data DT4a by, for example, defining a combination of image data DT11 included in the third video data and image data DT11 included in the fourth video data as a first combination, and defining a combination of image data DT11 included in the third video data and other image data DT11 included in the third video data as a second combination.

[0107] This allows the information processing device 1 in this embodiment of the present invention to accurately divide, for example, the third moving image data and the fourth moving image data (i.e., divide the moving image data captured during a time period when swallowing is occurring and the moving image data captured during a time period when swallowing is not occurring). Therefore, the information processing device 1 in this embodiment of the present invention can improve the determination accuracy of the fourth learning model MD4, for example.

[0108] 18, the data generating unit 112 may generate a plurality of fourth teacher data DT4 having, instead of an "aspiration symptom label," a "dysphagia label" indicating the severity of the dysphagia occurring in the area captured in the image data DT11. The "dysphagia label" may be, for example, "normal to mild," indicating that the score of the findings by the specialist who viewed the video data captured by the endoscopic camera 2 is below a predetermined lower threshold; "moderate," indicating that the score of the findings by the specialist who viewed the video data captured by the endoscopic camera 2 is equal to or greater than a lower threshold and less than a predetermined upper threshold; or "severe," indicating that the score of the findings by the specialist who viewed the video data captured by the endoscopic camera 2 is equal to or greater than an upper threshold. Specific examples of the fourth teacher data DT4 having a "dysphagia label" are described below.

[0109] [Specific example (2) of the fourth training data DT4 and the fourth learning model MD4] Next, other specific examples of the fourth teacher data DT4 and the fourth learning model MD4 will be described. Figures 18 and 19 are diagrams for explaining specific examples of the fourth teacher data DT4.

[0110] The fourth teacher data DT4 (hereinafter also referred to as fourth teacher data DT4c) shown in FIG. 18 has, as in the fourth teacher data DTa, items of "image data (1)" in which information identifying the image data DT11 is set, and "image data (2)" in which information identifying other image data DT11 is set. For example, the "image data (1)" and "image data (2)" are set with a combination (first combination) of image data DT11 constituting video data corresponding to a time period when swallowing is occurring in the body of the subject OB and image data DT11 constituting video data corresponding to a time period when swallowing is not occurring in the body of the subject OB. Furthermore, the "image data (1)" and "image data (2)" are set with a combination (second combination) of image data DT11 constituting video data corresponding to a time period when swallowing is occurring in the body of the subject OB and other image data DT11 constituting video data corresponding to a time period when swallowing is occurring in the body of the subject OB. That is, the fourth teacher data DT4c shown in FIG. 18 is, for example, so-called unlabeled teacher data.

[0111] Specifically, as shown in Figure 18, the data in the first row of the fourth teacher data DT4c has, for example, "IM01" set as "image data (1)" and "IM11" set as "image data (2)".

[0112] In addition, in the data in the second row of the fourth teacher data DT4c, for example, "IM02" is set as "image data (1)" and "IM12" is set as "image data (2)" as shown in Fig. 18. Explanation of the other data included in Fig. 18 will be omitted.

[0113] 19 (hereinafter also referred to as fourth teacher data DT4d) has, for example, the following items: "Image Data (Normal to Mild)" in which information identifying image data DT11 corresponding to a case where no dysphagia occurs in the body of the subject OB or where mild swallowing occurs; "Image Data (Moderate)" in which information identifying image data DT11 corresponding to a case where moderate dysphagia occurs in the body of the subject OB is set; and "Image Data (Severe)" in which information identifying image data DT11 corresponding to a case where severe dysphagia occurs in the body of the subject OB is set. That is, the fourth teacher data DT4d shown in FIG. 19 is, for example, so-called labeled teacher data. Hereinafter, the combination of image data DT11 corresponding to each of "Image Data (Normal to Mild)," "Image Data (Moderate)," and "Image Data (Severe)" will also be referred to as the fifth combination.

[0114] Specifically, as shown in Figure 19, the data in the first row of the fourth teacher data DT4d has, for example, "IM01" set as "image data (normal to mild)", "IM02" set as "image data (moderate)", and "IM03" set as "image data (severe)".

[0115] In addition, as shown in Fig. 19, in the data of the second row of the fourth teacher data DT4d, for example, "IM11" is set as "image data (normal to mild)", "IM12" is set as "image data (moderate)", and "IM13" is set as "image data (severe)". Explanation of the other data included in Fig. 19 will be omitted.

[0116] In the example shown in FIG. 19, the fourth teacher data DT4d has the items "image data (normal to mild)," "image data (moderate)," and "image data (severe)," but this is not limited to this. Specifically, the fourth teacher data DT4d may have, for example, a plurality of items of at least one of "image data (normal)," "image data (moderate)," and "image data (severe)." Furthermore, the fourth teacher data DT4d may not have any of the items "image data (normal to mild)," "image data (moderate)," and "image data (severe)." Furthermore, the fourth teacher data DT4d may have, for example, only a plurality of "image data (normal to mild)" items, or may have only a plurality of "image data (moderate)" items, or may have only a plurality of "image data (severe)" items. Hereinafter, when the fourth teacher data DT4d has only multiple "image data (normal to mild)" as items, the combination of image data DT11 corresponding to each of the multiple "image data (normal to mild)" will also be referred to as the sixth combination.

[0117] Then, in step S12, the model generation unit 113 performs machine learning (contrastive learning) of multiple fourth teacher data DT4c shown in Figure 18, and then performs machine learning (fine tuning) of multiple fourth teacher data DT4d shown in Figure 19, thereby generating the fourth learning model MD4 shown in the lower part of Figure 17.

[0118] Specifically, the model generation unit 113 performs machine learning (contrastive learning) of the multiple fourth teacher data DT4c shown in Figure 18 so that, for example, the distances (distances indicating similarity) between the multiple image data DT11 corresponding to the first combination become larger, and the distances (distances indicating similarity) between the multiple image data DT11 corresponding to the second combination become smaller.

[0119] Thereafter, the model generation unit 113 performs machine learning (contrastive learning) of the multiple fourth teacher data DT4d shown in FIG. 19, for example, so that the distances (distances indicating similarity) between the multiple image data DT11 corresponding to the fifth combination become larger, and so that the distances (distances indicating similarity) between the multiple image data DT11 corresponding to the sixth combination become smaller.

[0120] In other words, the information processing device 1 in this embodiment becomes able to generate a fourth learning model MD4 that can infer symptoms from different perspectives regarding the subject OB, for example, by appropriately rearranging the items contained in the fourth teacher data DT4b.

[0121] [Inference Processing in the First Embodiment] Next, the inference processing in the first embodiment will be described. Fig. 20 is a flowchart illustrating the inference processing in the first embodiment. Fig. 21 is a diagram illustrating the inference processing in the first embodiment.

[0122] 20, the information inference unit 114 waits until the inference timing comes (NO in S21), for example. The inference timing may be the timing when the worker inputs information indicating that the inference process will be performed.

[0123] Then, when it is time to infer (YES in S21), the information inference unit 114 inputs input data to the learning model MD stored in the information storage area 130, for example (S22).

[0124] Next, the information inference unit 114 acquires, for example, output data output from the learning model MD in response to the input of the input data in S22 (S23).

[0125] Thereafter, the information output unit 115 outputs the output data acquired in S23 to the operation terminal 5 (S24), for example.

[0126] Specifically, the information inference unit 114 inputs, for example, new image data DT11 constituting new video data of the inside of the subject OB captured by the endoscopic camera 2 to the first learning model MD1 stored in the information storage area 130. Then, the information inference unit 114 acquires, for example, new position information DT12 of the object captured in the new image data DT11 and a new label DT13 indicating the type of the object captured in the new image data DT11, both of which are output from the first learning model MD1. Thereafter, the information output unit 115 outputs, for example, the acquired new position information DT12 and new label DT13 to the operation terminal 5.

[0127] More specifically, the information output unit 115 displays, for example, ranges corresponding to each piece of new position information DT12 on the new image data DT11 using different colors on an output device (not shown) of the operation terminal 5, as shown in Fig. 21. Then, for each range corresponding to the new position information DT12 on the new image data DT11, the information output unit 115 displays, for example, information indicating a new label DT13 corresponding to each range (for example, "(1)" or "(2)") on the output device of the operation terminal 5 in a manner that associates each range with the new label DT13, as shown in Fig. 21.

[0128] Furthermore, the information processing device 1 inputs new image data DT11 to the second learning model MD2 stored in the information storage area 130, for example. Then, the information processing device 1 outputs new depth information DT14, which indicates the depth of the position of the test meal shown in the new image data DT11 output from the second learning model MD2, for example. Thereafter, the information processing device 1 inputs the new image data DT11 and the new depth information DT14 to the third learning model MD3 stored in the information storage area 130, for example. Then, the information processing device 1 acquires a new label DT15, which indicates whether or not aspiration has occurred in the body of the subject OB shown in the new image data DT11 output from the third learning model MD3, for example. Thereafter, the information processing device 1 outputs the acquired new label DT15 to the operation terminal 5, for example.

[0129] Furthermore, the information processing device 1 in this embodiment inputs new image data DT11 to the fourth learning model MD4 stored in the information storage area 130, for example. Then, the information processing device 1 acquires a new label DT16 relating to the symptoms of dysphagia in the body of the subject OB that are reflected in the new image data DT11 output from the fourth learning model M4, for example. Thereafter, the information processing device 1 outputs the acquired new label DT16 to the operation terminal 5, for example. [Explanation of symbols]

[0130] 1: Information processing device 2: Endoscope camera 5: Operation terminal 100: Information processing system 101:CPU 102:Memory 103: Communication device 104: Storage medium 105: Bus 110: Program 111: Information management unit 112: Data generation unit 113: Model generation unit 114: Information inference unit 115: Information output unit 130: Information storage area DT1: Teacher data DT2: Teacher data DT2: Teacher data DT4: Teacher data DT4a: Teacher data DT4b: Teacher data DT11: Image data DT12: Location information DT13: Label DT14: Depth information DT15: Label DT16: Label MD1: First learning model MD2: Second learning model MD3: Third learning model MD4: Fourth learning model NW: Network OB: Target

Claims

1. a first learning model is generated by machine learning of a plurality of first training data each including first image data constituting first video data of the inside of the body captured by an endoscopic camera, first position information indicating the position of at least one of an internal site and a test meal captured in the first image data, and a first label indicating the type of the object corresponding to the first position information, the first learning model outputting, in response to input of second image data constituting second video data of the inside of the body captured by an endoscopic camera, second position information of the object captured in the second image data and a second label indicating the type of the object captured in the second image data; storing the generated first learning model in a storage unit; A learning program that causes a computer to execute a process.

2. a second learning model is generated by machine learning a plurality of second training data each including first image data constituting first video data of the inside of the body photographed by an endoscopic camera and first depth information indicating the depth of the position of the test meal within the body photographed in the first image data, the second learning model outputs second depth information indicating the depth of the position of the test meal within the body photographed in the second image data in response to input of second image data constituting second video data of the inside of the body photographed by an endoscopic camera; a third learning model is generated by machine learning a plurality of third training data each including the first image data, the first depth information corresponding to the test meal shown in the first image data, and a third label indicating whether or not aspiration has occurred in the body shown in the first image data, the third learning model outputs a fourth label indicating whether or not aspiration has occurred in the body shown in the second image data in response to input of the second image data and the second depth information; storing the generated second learning model and the generated third learning model in a storage unit; A learning program characterized by:

3. a fourth learning model is generated by machine learning a plurality of fourth training data each including first image data constituting first video data of the inside of the body captured by an endoscopic camera and a fifth label relating to a symptom of dysphagia in the inside of the body captured in the first image data, the fourth learning model outputting a sixth label relating to a symptom of dysphagia in the inside of the body captured in the second image data in response to input of second image data constituting second video data of the inside of the body captured by the endoscopic camera; The generated fourth learning model is stored in a storage unit. A learning program that causes a computer to execute a process.

4. The symptoms of dysphagia include the severity of aspiration.

4. The learning program according to claim 3.

5. The symptoms of the dysphagia include the severity of the dysphagia.

4. The learning program according to claim 3.

6. The first image data is image data at a timing when swallowing is occurring, among image data constituting the first moving image data.

4. The learning program according to claim 3.

7. The first image data includes image data in which a ratio of pixels showing a predetermined white color is equal to or greater than a threshold value among image data constituting the first moving image data.

4. The learning program according to claim 3.

8. inputting second image data constituting second video data of the inside of the body captured by the endoscopic camera into a first learning model generated by machine learning of a plurality of first training data each including first image data constituting first video data of the inside of the body captured by the endoscopic camera, first position information indicating the position of at least one of an internal body part and a test meal captured in the first image data, and a first label of the object corresponding to the first position information; acquiring second position information of the object shown in the second image data and a second label indicating a type of the object shown in the second image data, which are output from the first learning model in response to the input of the second image data; outputting, to an output device, the second image data in which the acquired second label is associated with the position corresponding to the second position information; An inference program that causes a computer to execute processing.

9. inputting second image data constituting second moving image data of the inside of the body photographed by the endoscopic camera into a second learning model generated by machine learning of a plurality of second teacher data each including first image data constituting first moving image data of the inside of the body photographed by the endoscopic camera and first depth information indicating the depth of the position of the test meal within the body photographed in the first image data; acquiring second depth information indicating a depth of a position of the test meal shown in the second image data, the second depth information being output from the second learning model in response to the input of the second image data; inputting the second image data and the second depth information into a third learning model generated by machine learning of a plurality of third training data each including the first image data, the first depth information corresponding to the test meal shown in the first image data, and a third label indicating whether or not aspiration has occurred in the body shown in the first image data; acquiring a fourth label indicating whether or not aspiration has occurred in the body shown in the second image data, the fourth label being output from the third learning model in response to the input of the second image data and the second depth information; outputting the acquired fourth label to an output device; An inference program that causes a computer to execute processing.

10. inputting second image data constituting second video data of the inside of the body captured by the endoscopic camera into a fourth learning model generated by machine learning of a plurality of fourth teacher data each including first image data constituting first video data of the inside of the body captured by the endoscopic camera and a fifth label related to a symptom of swallowing disorder in the body captured in the first image data; acquiring a sixth label related to a symptom of dysphagia in the body shown in the second image data, the sixth label being output from the fourth learning model in response to the input of the second image data; outputting the acquired sixth label to an output device; An inference program that causes a computer to execute processing.

11. a first learning model is generated by machine learning of a plurality of first training data each including first image data constituting first video data of the inside of the body captured by an endoscopic camera, first position information indicating the position of at least one of an internal site and a test meal captured in the first image data, and a first label indicating the type of the object corresponding to the first position information, the first learning model outputting, in response to input of second image data constituting second video data of the inside of the body captured by an endoscopic camera, second position information of the object captured in the second image data and a second label indicating the type of the object captured in the second image data; storing the generated first learning model in a storage unit; 1. An information processing device comprising:

12. a second learning model is generated by machine learning a plurality of second training data each including first image data constituting first video data of the inside of the body photographed by an endoscopic camera and first depth information indicating the depth of the position of the test meal within the body photographed in the first image data, the second learning model outputs second depth information indicating the depth of the position of the test meal within the body photographed in the second image data in response to input of second image data constituting second video data of the inside of the body photographed by an endoscopic camera; a third learning model is generated by machine learning a plurality of third training data each including the first image data, the first depth information corresponding to the test meal shown in the first image data, and a third label indicating whether or not aspiration has occurred in the body shown in the first image data, the third learning model outputs a fourth label indicating whether or not aspiration has occurred in the body shown in the second image data in response to input of the second image data and the second depth information; storing the generated second learning model and the generated third learning model in a storage unit; 1. An information processing device comprising:

13. a fourth learning model is generated by machine learning a plurality of fourth training data each including first image data constituting first video data of the inside of the body captured by an endoscopic camera and a fifth label relating to a symptom of dysphagia in the inside of the body captured in the first image data, the fourth learning model outputting a sixth label relating to a symptom of dysphagia in the inside of the body captured in the second image data in response to input of second image data constituting second video data of the inside of the body captured by the endoscopic camera; The generated fourth learning model is stored in a storage unit.

1. An information processing device comprising:

14. inputting second image data constituting second video data of the inside of the body captured by the endoscopic camera into a first learning model generated by machine learning of a plurality of first training data each including first image data constituting first video data of the inside of the body captured by the endoscopic camera, first position information indicating the position of at least one of an internal body part and a test meal captured in the first image data, and a first label of the object corresponding to the first position information; acquiring second position information of the object shown in the second image data and a second label indicating a type of the object shown in the second image data, which are output from the first learning model in response to the input of the second image data; outputting, to an output device, the second image data in which the acquired second label is associated with the position corresponding to the second position information; 1. An information processing device comprising:

15. inputting second image data constituting second moving image data of the inside of the body photographed by the endoscopic camera into a second learning model generated by machine learning of a plurality of second teacher data each including first image data constituting first moving image data of the inside of the body photographed by the endoscopic camera and first depth information indicating the depth of the position of the test meal within the body photographed in the first image data; acquiring second depth information indicating a depth of a position of the test meal shown in the second image data, the second depth information being output from the second learning model in response to the input of the second image data; inputting the second image data and the second depth information into a third learning model generated by machine learning of a plurality of third training data each including the first image data, the first depth information corresponding to the test meal shown in the first image data, and a third label indicating whether or not aspiration has occurred in the body shown in the first image data; acquiring a fourth label indicating whether or not aspiration has occurred in the body shown in the second image data, the fourth label being output from the third learning model in response to the input of the second image data and the second depth information; outputting the acquired fourth label to an output device; 1. An information processing device comprising:

16. inputting second image data constituting second video data of the inside of the body captured by the endoscopic camera into a fourth learning model generated by machine learning of a plurality of fourth teacher data each including first image data constituting first video data of the inside of the body captured by the endoscopic camera and a fifth label related to a symptom of swallowing disorder in the body captured in the first image data; acquiring a sixth label related to a symptom of dysphagia in the body shown in the second image data, the sixth label being output from the fourth learning model in response to the input of the second image data; outputting the acquired sixth label to an output device; 1. An information processing device comprising:

17. a first learning model is generated by machine learning of a plurality of first training data each including first image data constituting first video data of the inside of the body captured by an endoscopic camera, first position information indicating the position of at least one of an internal site and a test meal captured in the first image data, and a first label indicating the type of the object corresponding to the first position information, the first learning model outputting, in response to input of second image data constituting second video data of the inside of the body captured by an endoscopic camera, second position information of the object captured in the second image data and a second label indicating the type of the object captured in the second image data; storing the generated first learning model in a storage unit; A learning method characterized in that the processing is executed by a computer.

18. a second learning model is generated by machine learning a plurality of second training data each including first image data constituting first video data of the inside of the body photographed by an endoscopic camera and first depth information indicating the depth of the position of the test meal within the body photographed in the first image data, the second learning model outputs second depth information indicating the depth of the position of the test meal within the body photographed in the second image data in response to input of second image data constituting second video data of the inside of the body photographed by an endoscopic camera; a third learning model is generated by machine learning a plurality of third training data each including the first image data, the first depth information corresponding to the test meal shown in the first image data, and a third label indicating whether or not aspiration has occurred in the body shown in the first image data, the third learning model outputs a fourth label indicating whether or not aspiration has occurred in the body shown in the second image data in response to input of the second image data and the second depth information; storing the generated second learning model and the generated third learning model in a storage unit; A learning method characterized in that the processing is executed by a computer.

19. a fourth learning model is generated by machine learning a plurality of fourth training data each including first image data constituting first video data of the inside of the body captured by an endoscopic camera and a fifth label relating to a symptom of dysphagia in the inside of the body captured in the first image data, the fourth learning model outputting a sixth label relating to a symptom of dysphagia in the inside of the body captured in the second image data in response to input of second image data constituting second video data of the inside of the body captured by the endoscopic camera; The generated fourth learning model is stored in a storage unit. A learning method characterized in that the processing is executed by a computer.

20. inputting second image data constituting second video data of the inside of the body captured by the endoscopic camera into a first learning model generated by machine learning of a plurality of first training data each including first image data constituting first video data of the inside of the body captured by the endoscopic camera, first position information indicating the position of at least one of an internal body part and a test meal captured in the first image data, and a first label of the object corresponding to the first position information; acquiring second position information of the object shown in the second image data and a second label indicating a type of the object shown in the second image data, which are output from the first learning model in response to the input of the second image data; outputting, to an output device, the second image data in which the acquired second label is associated with the position corresponding to the second position information; An inference method characterized in that the processing is executed by a computer.

21. inputting second image data constituting second moving image data of the inside of the body photographed by the endoscopic camera into a second learning model generated by machine learning of a plurality of second teacher data each including first image data constituting first moving image data of the inside of the body photographed by the endoscopic camera and first depth information indicating the depth of the position of the test meal within the body photographed in the first image data; acquiring second depth information indicating a depth of a position of the test meal shown in the second image data, the second depth information being output from the second learning model in response to the input of the second image data; inputting the second image data and the second depth information into a third learning model generated by machine learning of a plurality of third training data each including the first image data, the first depth information corresponding to the test meal shown in the first image data, and a third label indicating whether or not aspiration has occurred in the body shown in the first image data; acquiring a fourth label indicating whether or not aspiration has occurred in the body shown in the second image data, the fourth label being output from the third learning model in response to the input of the second image data and the second depth information; outputting the acquired fourth label to an output device; An inference method characterized in that the processing is executed by a computer.

22. inputting second image data constituting second video data of the inside of the body captured by the endoscopic camera into a fourth learning model generated by machine learning of a plurality of fourth teacher data each including first image data constituting first video data of the inside of the body captured by the endoscopic camera and a fifth label related to a symptom of swallowing disorder in the body captured in the first image data; acquiring a sixth label related to a symptom of dysphagia in the body shown in the second image data, the sixth label being output from the fourth learning model in response to the input of the second image data; outputting the acquired sixth label to an output device; An inference method characterized in that the processing is executed by a computer.

Images