Processing apparatus, processing program, and processing method

Abstract

To provide a processor or the like suitable for processing an image obtained by photographing the inside of an oral cavity to be used for diagnosis in the oral cavity.SOLUTION: Acquiring, from an imaging device including a camera for capturing an image of a subject including at least a part of an oral cavity of a user, one or a plurality of assessment images of the subject captured by the camera; Acquiring, from the imaging device, living body detection information of the subject different from the one or more determination images, determining a possibility of suffering from a predetermined disease based on a learned determination model stored in a memory for determining the possibility of suffering from the predetermined disease, the acquired one or more determination images, and the living body detection information, and outputting information indicating the determined possibility of suffering.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a processing device, a processing program, and a processing method for processing an image of a subject captured by a camera. [Background technology]

[0002] It has been known for some time that doctors diagnose illnesses such as viral colds by observing changes in the condition of a user's oral cavity. Non-Patent Document 1 reports that lymph follicles that appear in the deepest part of the pharynx, located in the oral cavity, have a pattern specific to influenza. Lymph follicles with this unique pattern are called influenza follicles, and are a characteristic sign of influenza, said to appear approximately two hours after the onset of symptoms. However, such pharyngeal areas have been diagnosed by doctors through direct visual inspection, and diagnosis using images has not been performed. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Miyamoto and Watanabe, "Consideration of the meaning and value of pharyngeal examination findings (influenza follicles)," Nihon University Journal of Medicine 72(1): 11-18 (2013) Summary of the Invention [Problem to be solved by the invention]

[0004] Therefore, in light of the above-described technology, the present disclosure aims to provide, in various embodiments, a processing device, a processing program, and a processing method for determining the possibility of contracting a specified disease using a determination image of a subject obtained by photographing a user's oral cavity. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, there is provided a processing device including at least one processor, the at least one processor being configured to perform processing to acquire one or more judgment images of a subject photographed by a camera including a camera for photographing an image of the subject including at least a portion of a user's oral cavity from the photographing device, acquire biodetection information of the subject that is different from the one or more judgment images from the photographing device, determine the possibility of infection with a specified disease based on a learned judgment model stored in a memory for determining the possibility of infection with the specified disease, the acquired one or more judgment images, and the biodetection information, and output information indicating the determined possibility of infection.

[0006] According to one aspect of the present disclosure, there is provided a processing program that, when executed by at least one processor, causes the at least one processor to function as follows: acquire one or more judgment images of a subject photographed by a camera including a camera for photographing an image of the subject including at least a portion of a user's oral cavity from the photographing device; acquire biodetection information of the subject that differs from the one or more judgment images from the photographing device; determine the possibility of contracting a specified disease based on a learned judgment model stored in memory for determining the possibility of contracting the specified disease, the acquired one or more judgment images, and the biodetection information; and output information indicating the determined possibility of contracting the specified disease.

[0007] According to one aspect of the present disclosure, there is provided a processing method executed by at least one processor, the processing method including: a step of acquiring, from an imaging device including a camera for capturing an image of the subject including at least a portion of a user's oral cavity, one or more judgment images of the subject captured by the camera; a step of acquiring, from the imaging device, biodetection information of the subject that is different from the one or more judgment images; a step of determining the possibility of contracting a specified disease based on a learned judgment model stored in a memory for determining the possibility of contracting the specified disease, the acquired one or more judgment images, and the biodetection information; and a step of outputting information indicating the determined possibility of contracting the specified disease. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to provide a processing device, a processing program, and a processing method suitable for processing images obtained by photographing the inside of the oral cavity for use in intraoral diagnosis.

[0009] It should be noted that the above effects are merely illustrative for the sake of convenience and are not limiting. In addition to or instead of the above effects, any effect described in this disclosure or an effect obvious to a person skilled in the art may be achieved. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a state in which a processing system 1 according to an embodiment of the present disclosure is in use. [Figure 2] FIG. 2 is a diagram showing a state in which the processing system 1 according to an embodiment of the present disclosure is in use. [Figure 3] FIG. 3 is a schematic diagram of a processing system 1 according to one embodiment of the present disclosure. [Figure 4] FIG. 4 is a block diagram showing the configuration of a processing system 1 according to an embodiment of the present disclosure. [Figure 5] FIG. 5 is a schematic diagram showing the configuration of the top surface of the imaging device 200 according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic diagram showing a cross-sectional configuration of an imaging device 200 according to an embodiment of the present disclosure. [Figure 7A] FIG. 7A is a diagram conceptually illustrating an image management table stored in the processing device 100 according to an embodiment of the present disclosure. [Figure 7B] FIG. 7B is a diagram conceptually showing a biological detection information table stored in the processing device 100 according to an embodiment of the present disclosure. [Figure 7C] FIG. 7C is a diagram conceptually illustrating a user table stored in the processing device 100 according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a diagram showing a processing sequence executed between the processing device 100 and the imaging device 200 according to an embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a diagram showing a processing flow executed in the imaging device 200 according to an embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. [Figure 12] FIG. 12 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. [Figure 13] FIG. 13 is a diagram illustrating a processing flow for generating a trained model according to an embodiment of the present disclosure. [Figure 14] FIG. 14 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. [Figure 15] FIG. 15 is a diagram illustrating a processing flow for generating a trained model according to an embodiment of the present disclosure. [Figure 16] FIG. 16 is a diagram illustrating a processing flow for generating a trained model according to an embodiment of the present disclosure. [Figure 17] FIG. 17 is a diagram illustrating a processing flow for generating a trained model according to an embodiment of the present disclosure. [Figure 18] FIG. 18 is a diagram showing an example of a screen displayed on the processing device 100 according to an embodiment of the present disclosure. [Figure 19] FIG. 19 is a diagram illustrating an example of a screen displayed on the processing device 100 according to an embodiment of the present disclosure. [Figure 20] FIG. 20 is a schematic diagram of a processing system 1 according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] Various embodiments of the present disclosure will be described with reference to the accompanying drawings, in which common elements are designated by the same reference numerals.

[0012] First Embodiment 1. Overview of Processing System 1 The processing system 1 according to the present disclosure is primarily used to capture images of the inside of a user's oral cavity to obtain a subject image. In particular, the processing system 1 is used to capture images of the back of the throat and surrounding area of ​​the oral cavity, specifically the pharynx. Therefore, the following description will primarily focus on the case where the processing system 1 according to the present disclosure is used to capture images of the pharynx. However, the pharynx is only one example of an imaging region, and the processing system 1 according to the present disclosure can also be suitably used for other regions within the oral cavity, such as the tonsils.

[0013] The processing system 1 according to the present disclosure is used to determine the possibility of a predetermined disease from a subject image obtained by photographing a subject including at least the pharyngeal region of a user's oral cavity and other biological detection information, thereby diagnosing or assisting in the diagnosis of the predetermined disease. An example of a disease that can be determined by the processing system 1 is influenza. The possibility of influenza is typically diagnosed by examining the user's pharynx and tonsils or determining the presence or absence of findings such as follicles in the pharyngeal region. However, by using the processing system 1 to determine the possibility of influenza and output the results, it is possible to perform a diagnosis or assist in the diagnosis. Note that determining the possibility of influenza is just one example. The processing system 1 can be suitably used to determine any disease in which differences in intraoral findings occur due to the disease. Note that differences in findings are not limited to those discovered by a doctor or whose existence is medically known. For example, differences that can be recognized by persons other than doctors or that can be detected using artificial intelligence or image recognition technology can be suitably applied to the processing system 1. Examples of such diseases include influenza, as well as infectious diseases such as streptococcal infection, adenovirus infection, EB virus infection, mycoplasma infection, hand, foot and mouth disease, herpangina, and candidiasis; diseases that present with vascular or mucosal disorders such as arteriosclerosis, diabetes, and hypertension; and tumors such as tongue cancer and pharyngeal cancer.

[0014] In this disclosure, terms such as "determination" and "diagnosis" are used in relation to diseases, but these do not necessarily mean a definitive determination or diagnosis by a doctor. For example, it can also include a case where the processing system 1 of the present disclosure is used by the user himself or herself, or by an operator other than a doctor, and the processing device 100 included in the processing system 1 performs the determination or diagnosis.

[0015] In the present disclosure, the user who is the subject of image capture by the image capture device 200 may include any person, such as a patient, a test subject, a diagnostic user, or a healthy individual. In the present disclosure, the operator who holds the image capture device 200 and performs image capture operations is not limited to medical professionals such as doctors, nurses, and laboratory technicians, but may also include any person, such as the user. The processing system 1 according to the present disclosure is typically expected to be used in a medical institution. However, this is not limited to this case, and the location of use may be any, such as the user's home, school, or workplace.

[0016] In the present disclosure, as described above, the subject may include at least a portion of the user's oral cavity. The disease to be diagnosed may be any disease that results in differences in findings within the oral cavity. However, the following description will be given of a case in which the subject includes the pharynx or the area around the pharynx, and the disease is influenza.

[0017] Furthermore, in the present disclosure, the subject image and the judgment image may be one or more videos or one or more still images. As an example of the operation, when the power button is pressed, a through image is captured by the camera, and the captured through image is displayed on the display 203. Then, when the operator presses the capture button, one or more still images are captured by the camera, and the captured image is displayed on the display 203. Alternatively, when the user presses the capture button, video capture begins, and images captured by the camera during that time are displayed on the display 203. Then, when the capture button is pressed again, video capture ends. In this manner, in a series of operations, various images, such as through images, still images, and videos, are captured by the camera and displayed on the display. Furthermore, the subject image or judgment image may be subjected to various image processing for the display or processing related to the present disclosure, resulting in a processed image. However, the subject image and judgment image do not refer only to specific images among these, but may include all of these images captured by the camera.

[0018] Furthermore, in the present disclosure, the term "biological detection information" refers to information about a user's biological body that is captured by a camera of an imaging device and is different from the judgment image used in the trained judgment model. Examples of the biological detection information include detection data detected by information detection sensors such as a temperature sensor, breath sensor, heart rate sensor, infrared sensor, near-infrared sensor, ultraviolet sensor, acoustic sensor, or a combination thereof provided in the imaging device, and information such as the user's body temperature, breath, breathing sounds, heart rate, vascular status, and oxygen saturation obtained based on the detected detection data and the subject image captured by the camera, as well as combinations thereof. That is, the biological detection information may be the detection data itself detected by the information detection sensor, or information processed from the detected data. Furthermore, when information processed from the detection data is used as the biological detection information, the processing may be performed in the imaging device or in a processing device.

[0019] FIG. 1 is a diagram illustrating a state in which a processing system 1 according to an embodiment of the present disclosure is in use. According to FIG. 1, the processing system 1 according to the present disclosure includes a processing device 100 and an imaging device 200. An operator attaches an auxiliary tool 300 to the tip of the imaging device 200 so as to cover it, and inserts the imaging device 200 together with the auxiliary tool 300 into the oral cavity 710 of the user. Specifically, first, an operator (which may be the user 700 or may be a different person from the user 700) attaches the auxiliary tool 300 to the tip of the imaging device 200 so as to cover it. Then, the operator inserts the imaging device 200 with the auxiliary tool 300 attached into the oral cavity 710. At this time, the tip of the auxiliary tool 300 passes through the incisors 711 and is inserted up to the vicinity of the soft palate 713. That is, the imaging device 200 is similarly inserted up to the vicinity of the soft palate 713. At this time, the auxiliary tool 300 (which functions as a tongue depressor) pushes the tongue 714 downward, restricting the movement of the tongue 714. Furthermore, the soft palate 713 is pushed upward by the tip of the auxiliary tool 300. This allows the operator to secure a good field of view for the imaging device 200, enabling good imaging of the pharynx 715 located in front of the imaging device 200.

[0020] Furthermore, a desired information detection sensor is installed at the tip of the imaging device 200 as needed, and the information detection sensor detects the user's living body detection information. That is, when the imaging device 200 is inserted up to the vicinity of the soft palate 713, the information detection sensor is also inserted. Therefore, living body detection information can be obtained for the same region in the oral cavity 710 as the region of interest captured by the camera of the imaging device 200, or for the surrounding region. That is, it is possible to obtain living body information for the region of interest or the surrounding region. Furthermore, since the movement of the tongue 714 is restricted as described above, it is possible to eliminate adverse effects of the tongue 714, etc., on the detection of living body detection information by the information detection sensor.

[0021] The captured subject image (typically, an image including the pharynx 715) and the living body detection information are transmitted from the imaging device 200 to the processing device 100, which is communicably connected via a wired or wireless network. The processor of the processing device 100, which has received the subject image, processes a program stored in its memory, thereby selecting a determination image to be used for determination from the subject image, and determining the possibility of the subject having a predetermined disease using the determination image and the living body detection information. The result is then output to a display or the like.

[0022] FIG. 2 is a diagram illustrating a usage state of a processing system 1 according to an embodiment of the present disclosure. Specifically, FIG. 2 is a diagram illustrating a state in which an operator 600 holds an image capturing device 200 of the processing system 1. As illustrated in FIG. 2, the image capturing device 200 is composed of, from the side inserted into the oral cavity, a main body 201, a grip 202, and a display 203. The main body 201 and the grip 202 are formed in a substantially columnar shape of a predetermined length along the insertion direction H into the oral cavity. The display 203 is disposed on the opposite side of the grip 202 from the main body 201 side. Therefore, the image capturing device 200 is formed in a substantially columnar shape as a whole, and is held by the operator 600 by holding it in a manner similar to holding a pencil. In other words, since the display panel of the display 203 faces the operator 600 in usage, the image capturing device 200 can be easily handled while checking the subject image captured by the image capturing device 200 and the biological detection information detected by the image capturing device 200 in real time.

[0023] Furthermore, when the operator 600 holds the grip 202 in an orientation in which the subject image is displayed in the normal orientation on the display 203, the photographing button 220 is configured to be located on the upper surface of the grip. Therefore, when the operator 600 holds the grip, the operator 600 can easily press the photographing button 220 with the index finger or the like.

[0024] 2. Configuration of Processing System 1 3 is a schematic diagram of a processing system 1 according to an embodiment of the present disclosure. According to FIG. 3, the processing system 1 includes a processing device 100 and an image capturing device 200 communicably connected to the processing device 100 via a wired or wireless network. The processing device 100 receives an operation input from an operator and controls image capturing by the image capturing device 200. The processing device 100 also processes the subject image captured by the image capturing device 200 and the biological detection information to determine whether the user is likely to be infected with influenza. The processing device 100 then outputs the determination result and notifies the result to the user, an operator, a doctor, or the like.

[0025] The tip of the imaging device 200 is inserted into the oral cavity of a user to capture images of the oral cavity, particularly the pharynx. The imaging device 200 also detects biometric information of the subject based on an information detection sensor provided at the tip. Specific processing will be described later. The captured subject image and biometric information are transmitted to the processing device 100 via a wired or wireless network.

[0026] The processing system 1 may further include a mounting table 400 as necessary. The mounting table 400 is capable of stably mounting the imaging device 200. The mounting table 400 may be connected to a power source via a wired cable, thereby supplying power to the imaging device 200 from a power supply terminal of the mounting table 400 through a power supply port of the imaging device 200.

[0027] FIG. 4 is a block diagram showing a configuration of a processing system 1 according to an embodiment of the present disclosure. According to FIG. 4, the processing system 1 includes a processing device 100 including a processor 111, a memory 112, an input interface 113, an output interface 114, and a communication interface 115, and an imaging device 200 including a camera 211, a light source 212, a processor 213, a memory 214, a display panel 215, an input interface 210, and a communication interface 216. These components are electrically connected to each other via control lines and data lines. The processing system 1 does not need to include all of the components shown in FIG. 4; some components may be omitted, or other components may be added. For example, the processing system 1 may include a battery or the like for driving each component.

[0028] First, the processor 111 of the processing device 100 functions as a control unit that controls the other components of the processing system 1 based on a program stored in the memory 112. Based on the program stored in the memory 112, the processor 111 controls the driving of the camera 211 and the driving of the light source 212, stores the subject image and living body detection information received from the photographing device 200 in the memory 112, and processes the stored subject image and living body detection information. Specifically, the processor 111 executes the following processes based on the programs stored in the memory 112: acquiring a subject image of a subject from the camera 211, acquiring from the imaging device living body detection information of a subject different from the above-mentioned determination image, inputting the acquired subject image into a determination image selection model to acquire candidate determination images, acquiring a determination image from the acquired candidate determination images based on the similarity between the images, acquiring at least one of the user's medical interview information and attribute information, determining the possibility of influenza infection based on the trained determination model stored in the memory 112, the acquired one or more determination images, and at least one of the user's medical interview information and attribute information including the above-mentioned living body detection information, and outputting information indicating the determined possibility of infection with a predetermined disease in order to diagnose or assist in the diagnosis. The processor 111 is mainly composed of one or more CPUs, but may also be combined with a GPU, an FPGA, etc. as appropriate.

[0029] The memory 112 is composed of RAM, ROM, nonvolatile memory, HDD, etc., and functions as a storage unit. The memory 112 stores instructions and commands for various control operations of the processing system 1 according to this embodiment as programs. Specifically, the memory 112 stores programs to be executed by the processor 111, such as a process of acquiring a subject image of a subject from the camera 211, a process of acquiring, from an imaging device, living body detection information of a subject different from the above-mentioned determination image, a process of inputting the acquired subject image into a determination image selection model to acquire candidate determination images, a process of acquiring a determination image from the acquired candidate determination images based on the similarity between each image, a process of acquiring at least one of a user's medical history information and attribute information, a process of determining the possibility of influenza based on a trained determination model stored in the memory 112, one or more acquired determination images, and at least one of a user's medical history information and attribute information including the above-mentioned living body detection information, and a process of outputting information indicating the determined possibility of infection with a predetermined disease in order to diagnose or assist in the diagnosis. In addition to the program, the memory 112 also stores an image management table for managing subject images captured by the camera 211 of the imaging device 200, the images, etc., a user table for storing user attribute information, medical interview information, determination results, etc. The memory 112 also stores various trained models, such as a trained determination image selection model used to select a determination image from the subject image, and a trained determination model for determining the possibility of disease from the determination image.

[0030] The input interface 113 functions as an input unit that accepts operator input instructions to the processing device 100 and the imaging device 200. Examples of the input interface 113 include a "photography button" for instructing the imaging device 200 to start or stop recording or detecting biological information, a "confirmation button" for making various selections, a "back / cancel button" for returning to the previous screen or canceling an input confirmation operation, a cross key button for moving a pointer or the like output to the output interface 114, an on / off key for turning the power of the processing device 100 on and off, and physical key buttons such as character input key buttons for inputting various characters. The input interface 113 may also be a touch panel that is superimposed on a display functioning as the output interface 114 and has an input coordinate system corresponding to the display coordinate system of the display. In this case, icons corresponding to the physical keys are displayed on the display, and the operator selects each icon by inputting instructions via the touch panel. The touch panel may be any type, such as a capacitive type or a resistive type, for detecting user input instructions. The input interface 113 does not always need to be physically provided on the processing device 100, but may be connected via a wired or wireless network as needed.

[0031] The output interface 114 functions as an output unit for outputting the subject image captured by the imaging device 200 and the detected living body detection information, and for outputting the results determined by the processor 111. An example of the output interface 114 is a display configured with a liquid crystal panel, an organic EL display, a plasma display, or the like. However, the processing device 100 itself does not necessarily need to be equipped with a display. For example, an interface for connecting to a display or the like connectable to the processing device 100 via a wired or wireless network can also function as the output interface 114 for outputting display data to the display or the like.

[0032] The communication interface 115 functions as a communication unit for transmitting and receiving various commands related to the start of imaging and the like, image data captured by the imaging device 200, and detected living body detection information to and from the imaging device 200 connected via a wired or wireless network. Examples of the communication interface 115 include various types, such as a wired communication connector such as USB or SCSI, a wireless communication transmitting / receiving device such as a wireless LAN, Bluetooth (registered trademark), or infrared, and various connection terminals for printed circuit boards or flexible circuit boards.

[0033] Next, the camera 211 of the imaging device 200 functions as an imaging unit that detects light reflected from the oral cavity, which is the subject, and generates an image of the subject. To detect the light, the camera 211 includes, for example, a CMOS image sensor, a lens system, and a drive system for achieving the desired functions. The image sensor is not limited to a CMOS image sensor; other sensors, such as a CCD image sensor, can also be used. It is also possible to use additional imaging elements depending on the wavelength band of the reflected light to be detected. For example, a CMOS image sensor, which is an imaging element that detects wavelengths from the visible light band to the near-infrared light band, and a sensor, which is an infrared imaging element that detects wavelengths in the infrared light band, may be arranged adjacent to each other on an image sensor board. Although not shown, the camera 211 may have an autofocus function, and is preferably set to focus on a specific area, for example, on the front of the lens. Furthermore, the camera 211 may have a zoom function and is preferably set to capture images at an appropriate magnification depending on the size of the pharynx or influenza follicles.

[0034] It is known that lymph follicles appearing in the deepest part of the pharynx located in the oral cavity have a pattern specific to influenza. Lymph follicles with this unique pattern are called influenza follicles and are a characteristic sign of influenza, appearing approximately two hours after the onset of symptoms. As described above, the processing system 1 of this embodiment is used to determine the possibility of a user having influenza by, for example, photographing the pharynx of the oral cavity and detecting the follicles. Therefore, when the photographing device 200 is inserted into the oral cavity, the distance between the camera 211 and the subject becomes relatively short. Therefore, it is preferable that the camera 211 has an angle of view (2θ) such that the value calculated by [(distance from the tip of the camera 211 to the posterior pharyngeal wall) * tanθ] is 20 mm or more vertically and 40 mm or more horizontally. Using a camera with such an angle of view enables photographing a wider range even when the camera 211 and the subject are close to each other. That is, the camera 211 can be a normal camera, or a so-called wide-angle camera or an ultra-wide-angle camera.

[0035] Furthermore, in this embodiment, the main subject captured by camera 211 is the pharynx and influenza follicles formed in the pharynx. Because the pharynx is generally formed deep in the depth direction, if the depth of field is shallow, the focus will shift between the anterior and posterior pharynx, making it difficult to obtain a subject image suitable for use in determination by processing device 100. Therefore, camera 211 has a depth of field of at least 20 mm or more, preferably 30 mm or more. Using a camera with such a depth of field makes it possible to obtain a subject image that is in focus at all parts from the anterior pharynx to the posterior pharynx.

[0036] The camera 211 can also function as an information detection sensor for detecting biological detection information. For example, heartbeat information can be obtained by detecting vibrations due to pulse on the surface of oral tissue from the subject image captured by the camera 211. Furthermore, blood vessel patterns on the oral cavity surface can be detected from the subject image captured by the camera 211 to obtain blood vessel condition information. That is, the camera 211 can be made to function as an information detection sensor to obtain the captured subject image or various information detected from the subject image as biological detection information.

[0037] The light source 212 is driven by instructions from the processing device 100 or the photographing device 200, and functions as a light source unit for irradiating light into the oral cavity. The light source 212 includes one or more light sources. In this embodiment, the light source 212 is composed of one or more LEDs, and light having a predetermined frequency band is irradiated from each LED in the direction of the oral cavity. The light source 212 uses light having a desired band from the ultraviolet light band, the visible light band, or the infrared light band, or a combination thereof. When irradiating multiple bands, multiple LEDs that are preset to irradiate light of each band are switched for display.

[0038] In particular, near-infrared light has a high degree of penetration through living tissue and is therefore suitable for observing the state of blood vessels within the body. By irradiating near-infrared light and detecting the reflected light, it is possible to obtain information on the state of blood vessels in the irradiated area. Furthermore, ultraviolet light has a low tissue penetration rate and is therefore suitable for detecting minute structural changes on the surface of a living body. By irradiating ultraviolet light and detecting the reflected light, it is possible to obtain structural information on the surface of a living body. Furthermore, ultraviolet light is used as excitation light for fluorescent substances that bind to specific cells (e.g., tumor cells). By irradiating light in the ultraviolet range and detecting the excitation light from the fluorescent substance, it is possible to obtain information on tumors in the irradiated area.

[0039] The information detection sensor 232 is inserted into the oral cavity of the user and functions as a detector for acquiring biological detection information, which is information related to the user's biological activity. Examples of such information detection sensor 232 include a temperature sensor, a breath sensor, a heart rate sensor, a pulse oximeter sensor, an infrared sensor, a near-infrared sensor, an ultraviolet sensor, an acoustic sensor, or a combination thereof. In some cases, the camera 211 may also function as the information detection sensor 232. Examples of biological detection information acquired by such information detection sensor 232 include detection data detected by each sensor (including the case where the camera 211 functions as the information detection sensor 232), information on the user's body temperature, breath, heart rate, and blood vessel condition obtained based on the detected detection data, and combinations thereof.

[0040] When a temperature sensor is used as the information detection sensor 232, a thermistor sensor, a thermocouple sensor, a resistor sensor, a digital temperature sensor, an infrared sensor, or a combination thereof can be used. Among these, digital temperature sensors and infrared sensors, which are suitable for non-contact measurement, are more preferable. In medical diagnoses, a user's body temperature is typically input as interview information based on the user's report or measurements taken in the clinic. However, in this case, the measurement location and method may vary depending on the person taking the measurement. For example, there are thermometers that can measure the temperature of the skin surface, such as the forehead or wrist, without contact. However, during the winter season, when infectious diseases such as influenza are more likely to spread, these thermometers are susceptible to the influence of outside temperatures, resulting in large measurement errors. However, since the temperature inside the oral cavity is less affected by outside temperatures, more accurate information can be input as interview information by using a temperature sensor placed in the imaging device 200 inserted into the oral cavity. In other words, the temperature information detected by the temperature sensor is used as biometric information.

[0041] In particular, when an infrared sensor is used, infrared image data is obtained based on the infrared energy obtained by the sensor or the temperature information obtained by converting the infrared energy. Furthermore, segmentation image data is obtained by segmenting each part of the subject image using a segmentation method such as semantic segmentation. Then, by overlaying the infrared image data and the segmentation image data, it is possible to accurately detect the temperature of the measurement area or the area of ​​interest. Segmentation can also be performed on the infrared image data, and the temperature of the measurement area or the area of ​​interest can also be accurately detected by using this segmented infrared image data. In this way, by using an infrared sensor as the information detection sensor 232, particularly accurate temperature data of the area of ​​interest can be used as biological detection information.

[0042] When an ultraviolet sensor is used as the information detection sensor 232, it is possible to detect, for example, minute structures on the surface of a living body. Ultraviolet light generally has low tissue penetration and is largely reflected by the surface of living tissue. Therefore, compared with other wavelength bands, such as visible light, ultraviolet light is more suitable for detecting minute structural changes on the surface of a living body. For example, infectious diseases such as influenza cause characteristic structural changes, such as follicles, in the pharynx, as described above. Therefore, detecting such structural changes makes it possible to determine the presence or absence of disease. Furthermore, by overlaying ultraviolet image data obtained by the ultraviolet sensor with segmentation image data of the subject image, it is possible to more accurately detect structural changes in the measurement area or the area of ​​interest. In other words, the ultraviolet detection data detected by the ultraviolet sensor or the structural information on the living body surface obtained therefrom is used as living body detection information.

[0043] When a breath sensor is used as the information detection sensor 232, a sensor device capable of detecting specific gas components, such as ammonia and acetone, contained in breath can be used. Here, there is a correlation between the amount of specific gas components, such as ammonia and acetone, contained in breath that cause bad breath and human diseases. For example, a sweet smell is associated with infectious diseases, acetone with diabetes, the gangrene odor of ammonia and protein with certain tumors and liver disease, isoprene with hypoglycemia and sleep disorders, methyl mercaptan with oral bacteria and liver disease, carbon monoxide with stress, ethanol with alcohol use, trimethylamine with kidney disease, and nitric oxide with asthma. Therefore, by detecting the gas components that cause these odors in breath, it is possible to infer the disease. In other words, the gas component data measured by the breath sensor and the disease information inferred therefrom are used as biodetection information.

[0044] When a heart rate sensor is used as the information detection sensor 232, although sensors capable of measuring by contacting a living body are known, it is also possible to use an image sensor of the camera 211 capable of non-contact detection, a microwave or millimeter wave detection sensor, or a combination of these. For example, it is possible to estimate the user's heart rate by detecting changes in the surface position of the subject over time due to pulsation based on the subject image detected by the image sensor. Alternatively, it is possible to obtain the luminance of the G (green) component in each frame constituting the obtained subject image (video), generate a luminance waveform of the G component in the video, and estimate the heart rate from its peak value. This method utilizes the fact that hemoglobin in blood absorbs green light. In other words, the heart rate measured by the heart rate sensor is used as living body detection information.

[0045] When a near-infrared sensor is used as the information detection sensor 232, it is possible to detect, for example, the state of blood vessels passing through the surface or deep inside of the subject. Here, diabetes and high blood pressure obstruct blood flow or cause structural damage to the blood vessels themselves. Furthermore, these vascular disorders cause blood flow disorders, making the subject more susceptible to infections and the like. Therefore, detecting the state of blood vessels is useful in diagnosing diabetes, high blood pressure, infections, and the like. In other words, image data of near-infrared images measured by the near-infrared sensor and blood vessel state information estimated from them are used as biometric detection information.

[0046] When a pulse oximeter sensor is used as the information detection sensor 232, for example, it is possible to estimate the oxygen saturation of blood by irradiating the surface of a living body with wavelengths in the near-infrared light band and infrared light band and estimating the amounts of oxygenated hemoglobin and deoxygenated hemoglobin from the reflected light. In other words, the oxygen saturation of blood measured by the pulse oximeter sensor is used as living body detection information.

[0047] When an acoustic sensor is used as the information detection sensor 232, a microphone can typically be used. Breathing sounds can be an important indicator for diagnosing the possibility of respiratory diseases, etc. In addition, in certain diseases such as bronchiectasis, sounds like popping blisters are heard, and acoustic data within the oral cavity is important information as oral auscultation data. Therefore, the acoustic sensor is disposed at the tip of the imaging device 200 to obtain detection data of the user's breathing sounds and sounds within the oral cavity. In other words, the detection data of the user's breathing sounds and sounds within the oral cavity detected by the acoustic sensor is used as biological detection information.

[0048] The processor 213 functions as a control unit that controls other components of the imaging device 200 based on a program stored in the memory 214. Based on the program stored in the memory 214, the processor 213 controls the driving of the camera 211 and the information detection sensor 232 and the driving of the light source 212, as well as controls the storage of the subject image captured by the camera 211 in the memory 214 and the storage of the living body detection information detected by the information detection sensor 232 in the memory 214. The processor 213 also controls the output of the subject image, living body detection information, and user information stored in the memory 214 to the display 203 and the transmission of the subject image, living body detection information, and user information to the processing device 100. The processor 213 is mainly composed of one or more CPUs, but may be combined with other processors as appropriate.

[0049] The memory 214 is composed of RAM, ROM, non-volatile memory, HDD, etc., and functions as a storage unit. The memory 214 stores instructions and commands as programs for various controls of the imaging device 200. In addition to the programs, the memory 214 also stores subject images captured by the camera 211, living body detection information detected by the information detection sensor 232, various types of information about the user, etc.

[0050] The display panel 215 is provided on the display 203 and functions as a display unit for displaying the subject image captured by the imaging device 200 and the biological detection information detected by the information detection sensor 232. The display panel 215 is configured by a liquid crystal panel, but is not limited to a liquid crystal panel and may be configured by an organic EL display, a plasma display, or the like.

[0051] The input interface 210 functions as an input unit that accepts user input instructions to the processing device 100 and the imaging device 200. Examples of the input interface 210 include a "photography button" for instructing the imaging device 200 to start or stop recording or detecting biological information, a "power button" for turning the imaging device 200 on or off, a "confirmation button" for making various selections, a "back / cancel button" for returning to the previous screen or canceling an input confirmation operation, and physical key buttons such as a cross key button for moving icons displayed on the display panel 215. Note that these various buttons and keys may be physically provided, or may be displayed as icons on the display panel 215 and be selectable using a touch panel or the like superimposed on the display panel 215 and arranged as the input interface 210. The method for detecting user input instructions via the touch panel may be any method, such as a capacitance type or a resistive film type.

[0052] The communication interface 216 functions as a communication unit for transmitting and receiving information to and from the image capturing device 200 and / or other devices. Examples of the communication interface 216 include a wired communication connector such as a USB or SCSI, a wireless communication transmitting and receiving device such as a wireless LAN, Bluetooth (registered trademark), or infrared, and various connection terminals for printed circuit boards or flexible circuit boards.

[0053] Fig. 5 is a top view showing a configuration of an imaging device 200 according to an embodiment of the present disclosure. Specifically, Fig. 5 is a diagram showing the state of imaging device 200, including main body 201, grip 202, and display 203, as viewed from above from the side inserted into the oral cavity. According to Fig. 5, main body 201 is configured as a columnar body having a base end 225 and a tip end 222, and having a predetermined length in a direction substantially parallel to the direction in which light is irradiated from light source 212, i.e., direction H of insertion into the oral cavity. At least tip end 222 of main body 201 is inserted into the oral cavity.

[0054] The main body 201 is formed in the shape of a hollow cylindrical column with a perfectly circular cross section. The wall 224 may be made of any material capable of guiding light therein, and one example is a thermoplastic resin. Examples of thermoplastic resins include polyolefin resins such as linear polyolefin resins (e.g., polypropylene resins) and cyclic polyolefin resins (e.g., norbornene resins), cellulose ester resins such as triacetyl cellulose and diacetyl cellulose, polyester resins, polycarbonate resins, (meth)acrylic resins, polystyrene resins, or mixtures or copolymers thereof. In other words, the wall 224 of the main body 201 functions as a light guide for guiding light emitted from a light source into the oral cavity or toward a diffuser.

[0055] Since main body 201 is formed hollow, wall portion 224 forms storage space 223 on its inner surface. Camera 211 is stored in this storage space 223. Note that main body 201 only needs to be formed in a columnar shape having storage space 223. Therefore, storage space 223 does not need to have a cylindrical shape with a perfect circular cross section, and may have an elliptical or polygonal cross section. Furthermore, main body 201 does not necessarily need to be formed hollow inside.

[0056] The tip of the grip 202 is connected to the base end 225 of the main body 201. A user holds the grip 202 to perform operations such as inserting and removing the imaging device 200. The grip 202 is formed of a columnar body having a predetermined length in a direction substantially parallel to the direction H in which it is inserted into the oral cavity, i.e., along the longitudinal direction of the main body 201, and is arranged on the same straight line as the main body 201 in the direction H. In this embodiment, the grip 202 is formed so that its cross section in the vertical direction is substantially oval, but it does not have to be oval and may be a perfect circle, ellipse, or polygon.

[0057] The grip 202 has a connecting portion 230 formed at a position closest to the base end 225 of the main body 201, and is connected to the main body 201 via the connecting portion 230. The outer periphery of the connecting portion 230 has engagement protrusions 217 (217-1 to 217-4) for positioning the assisting tool 300 and a positioning protrusion 218. The engagement protrusions 217 engage with engagement protrusions 318 (318-1 to 318-4) provided on the assisting tool 300. The positioning protrusions 218 are inserted into insertion holes 321 provided on the assisting tool 300 to position the imaging device 200 and the assisting tool 300 relative to each other. In this embodiment, the engagement protrusions 217 of the main body 201 are a total of four engagement protrusions (engagement protrusions 217-1 to 217-4) that are arranged at equal intervals on the surface of the grip 202 in the vicinity of the base end 225 of the main body 201. Furthermore, one positioning protrusion 218 is disposed between the engaging protrusions 217 on the surface of the grip 202, in a position near the base end 225 of the main body 201. However, this is not limiting, and it is also possible to dispose only one of the engaging protrusion 217 and the positioning protrusion 218. Furthermore, the number of both the engaging protrusion 217 and the positioning protrusion 218 may be any number, as long as there is one or more.

[0058] The grip 202 also includes an imaging button 220 on its top surface near the base end 225 of the main body 201, i.e., near the tip of the grip 202 in the insertion direction H into the oral cavity. This allows the operator 600 to easily press the imaging button 220 with the index finger or the like when holding the grip 202. The grip 202 also has a power button 221 located on its top surface near the display 203, i.e., on the opposite side of the grip 202 from the imaging button 220. This makes it possible to prevent the operator 600 from accidentally pressing the power button 221 when holding the grip 202 and capturing an image.

[0059] The display 203 has a generally rectangular parallelepiped shape as a whole, and is disposed on the same straight line as the main body 201 in the direction H. The display 203 also includes a display panel 215 on the surface opposite to the direction H of insertion into the oral cavity (i.e., toward the user). Therefore, the display 203 is formed so that the surface including the display panel is generally perpendicular to the longitudinal direction of the main body 201 and the grip 202, which are formed generally parallel to the direction H of insertion into the oral cavity. The surface opposite to the surface including the display panel is connected to the grip 202 on the side opposite the oral cavity of the grip 202. The shape of the display is not limited to a generally rectangular parallelepiped shape, and may be any shape, such as a cylindrical shape.

[0060] The diffusion plate 219 is disposed at the tip 222 of the main body 201, and diffuses light that is emitted from the light source 212 and passes through the main body 201 toward the oral cavity. The diffusion plate 219 has a shape that corresponds to the cross-sectional shape of the portion of the main body 201 that is configured to be able to guide light. In this embodiment, the main body 201 is formed in a hollow cylindrical shape. Therefore, the cross section of the diffusion plate 219 is also formed in a hollow shape that corresponds to that shape.

[0061] The camera 211 is used to generate an image of a subject by detecting light that is diffused by the diffuser 219 and irradiated into the oral cavity and reflected by the subject. The camera 211 is disposed on the inner surface of the wall 224 of the main body 201, i.e., in a storage space 223 formed inside the main body 201, so as to be collinear with the main body 201 in the direction H. Although only one camera 211 is described in this embodiment, the photographing device 200 may include multiple cameras. By generating an image of a subject using multiple cameras, the image of the subject includes information about the three-dimensional shape. Furthermore, in this embodiment, the camera 211 is disposed in the storage space 223 of the main body 201, but it may also be disposed at the tip 222 of the main body 201 or on the main body 201 (either inside the main body 201 or on the outer periphery of the main body 201).

[0062] Here, the camera 211 can also function as the information detection sensor 232. For example, it is possible to detect reflected light of visible light irradiated from the light source 212 and obtain an image of a subject as living body detection information. The camera 211 can also detect reflected near-infrared light and obtain a near-infrared image as living body detection information. The camera 211 can also detect infrared light emitted from a living body and obtain an infrared image as living body detection information. The camera 211 can also detect reflected ultraviolet light and obtain an ultraviolet image as living body detection information.

[0063] The information detection sensor 232 is a sensor for acquiring information about the living body of the user. The camera 211 may function as such a sensor, or the information detection sensor 232 may be arranged separately from the camera 211. In the example of FIG. 5, the information detection sensor 232 is arranged near the tip 222 of the main body 201. Specifically, the information detection sensor 232 is arranged on the outer surface of the main body 201 so as to face upward during use. By arranging the information detection sensor 232 in this position, it is possible to reduce the adverse effects of the tongue. However, the information detection sensor 232 can be arranged in an optimal position depending on the type of sensor used.

[0064] FIG. 6 is a schematic diagram illustrating a cross-sectional configuration of an imaging device 200 according to an embodiment of the present disclosure. According to FIG. 6, the light sources 212 are arranged on a substrate 231 disposed on the distal end side of the grip 202, and a total of four light sources 212-1 to 212-4 are disposed on the substrate 231. The light sources 212 are, for example, each configured with an LED, and each LED emits light having a predetermined frequency band toward the oral cavity. Specifically, the light emitted from the light sources 212 enters the proximal end 225 of the main body 201 and is guided toward the diffuser 219 by the wall 224 of the main body 201. The light that reaches the diffuser 219 is diffused into the oral cavity by the diffuser 219. The light diffused by the diffuser 219 is then reflected by the pharynx 715, which is the subject. This reflected light reaches the camera 211, and an image of the subject is generated.

[0065] The light sources 212-1 to 212-4 may be configured to be independently controlled. For example, by illuminating some of the light sources 212-1 to 212-4, the shadow of a subject (such as an influenza follicle) having a three-dimensional shape can be included in the subject image. This allows the subject image to include information about the three-dimensional shape of the subject, making it possible to more clearly distinguish the subject and more accurately determine the possibility of influenza infection using a determination algorithm. Furthermore, the light sources 212-1 to 212-4 may irradiate light in bands such as the visible light band, near-infrared light band, infrared light band, and ultraviolet light band.

[0066] In addition, in this embodiment, the light sources 212-1 to 212-4 are arranged on the base end 225 side of the main body 201, but they may also be arranged at the tip 222 of the main body 201 or on the main body 201 (either inside the main body 201 or on the outer periphery of the main body 201).

[0067] In this embodiment, the diffuser 219 is used to prevent the light emitted from the light source 212 from illuminating only a portion of the oral cavity and to generate uniform light. Therefore, as an example, a lens-shaped diffuser having a desired diffusion angle is used, in which a fine lens array is formed on the surface of the diffuser 219. Alternatively, a diffuser that can diffuse light by other methods, such as a diffuser that achieves light diffusion function by randomly arranged fine irregularities on the surface, may be used. Furthermore, the diffuser 219 may be configured integrally with the main body 201. For example, this can be achieved by forming fine irregularities on the tip portion of the main body 201.

[0068] In this embodiment, the diffusion plate 219 is disposed on the tip 222 side of the main body 201. However, the present invention is not limited to this, and the diffusion plate 219 may be disposed anywhere between the light source 212 and the oral cavity to be irradiated therewith, and may be disposed, for example, at the tip 222 of the main body 201 or in the main body 201 (either inside the main body 201 or on the outer periphery of the main body 201).

[0069] 3. Information stored in memory 112 of processing device 100 7A is a diagram conceptually illustrating an image management table stored in the processing device 100 according to an embodiment of the present disclosure. Information stored in the image management table is updated and stored as needed in accordance with the progress of processing by the processor 111 of the processing device 100.

[0070] According to FIG. 7A, the image management table stores subject image information, candidate information, and judgment image information, etc., in association with user ID information. "User ID information" is information unique to each user and is used to identify each user. User ID information is generated each time a new user is registered by the operator. "Subject image information" is information used to identify a subject image captured by the operator for each user. The subject image is one or more images containing a subject captured by the camera of the image capture device 200, and is received from the image capture device 200 and stored in the memory 112. "Candidate information" is information used to identify a candidate image from one or more subject images for selecting a judgment image. "Judgment image information" is information used to identify a judgment image used to determine the possibility of influenza infection. Such a judgment image is selected based on similarity from the candidate images identified by the candidate information. As described above, information used to identify each image is stored as subject image information, candidate information, and judgment image information. In this way, the information for specifying each image is typically identification information for identifying each image, but it may also be information indicating the storage location of each image or the image data itself of each image.

[0071] 7B is a diagram conceptually illustrating a living body detection information table stored in the processing device 100 according to an embodiment of the present disclosure. The information stored in the living body detection information table is updated and stored as needed in accordance with the progress of processing by the processor 111 of the processing device 100.

[0072] 7B, the biological detection information table stores temperature information, exhalation information, respiratory sound information, heart rate information, near-infrared image information, ultraviolet image information, oxygen saturation information, and the like, in association with user ID information. The “temperature information” is information detected by a temperature sensor when the information detection sensor 232 is used. The temperature information is stored in the memory 112 by receiving, from the image capturing device 200, detection data detected by the temperature sensor or temperature information calculated based on the detection data. The “exhalation information” is information detected by a respiratory sensor when the information detection sensor 232 is used. The exhalation information is stored in the memory 112 by receiving, from the image capturing device 200, detection data detected by the respiratory sensor or disease information estimated based on the detection data. The “respiratory sound information” is information detected by an acoustic sensor when the information detection sensor 232 is used. For example, acoustic data of respiratory sounds or sounds in the oral cavity detected by the acoustic sensor is used as the information. "Heart rate information" is information detected by a heart rate sensor when the information detection sensor 232 is a heart rate sensor. The heart rate information is stored in the memory 112 by receiving the heart rate measured by the heart rate sensor from the image capture device 200. "Near-infrared image information" is information identifying a near-infrared image detected by a near-infrared sensor when the information detection sensor 232 is a near-infrared sensor. This information may be any of image data of the near-infrared image and information indicating a storage location of the image data. The near-infrared image information is received from the image capture device 200 and stored in the memory 112. "Ultraviolet image information" is information identifying an ultraviolet image detected by an ultraviolet sensor when the information detection sensor 232 is a UV sensor. This information may be any of image data of the ultraviolet image and information indicating a storage location of the image data. The ultraviolet image information is received from the image capture device 200 and stored in the memory 112. "Oxygen saturation information" is information indicating the blood oxygen saturation measured by a pulse oximeter sensor when the information detection sensor 232 is a pulse oximeter sensor.Heart rate information is stored in memory 112 by receiving blood oxygen saturation measured by a pulse oximeter sensor from imaging device 200 .

[0073] Each piece of information stored in the living body detection information table can be used as medical interview information for the user of each piece of user ID information.

[0074] 7C is a diagram conceptually illustrating a user table stored in the processing device 100 according to an embodiment of the present disclosure. The information stored in the user table is updated and stored as needed in accordance with the progress of processing by the processor 111 of the processing device 100.

[0075] According to FIG. 7C, the user table stores attribute information, medical interview information, two-dimensional code information, and determination result information in association with user ID information. "User ID information" is information unique to each user and is used to identify each user. User ID information is generated each time a new user is registered by an operator. "Attribute information" is information input by, for example, an operator or user, and is information related to the individual user, such as the user's name, gender, age, and address. "Medical interview information" is information input by, for example, an operator or user, and is information used as a reference for diagnosis by a doctor, such as the user's medical history and symptoms. Examples of such medical interview information include patient background information such as weight, allergies, and underlying diseases; temperature (body temperature), peak body temperature since onset, time elapsed since onset, heart rate, pulse rate, oxygen saturation, exhaled air, blood pressure, medication status, contact with other influenza patients, subjective symptoms and physical findings such as joint pain, muscle pain, headache, fatigue, loss of appetite, chills, sweating, cough, sore throat, runny nose / nasal congestion, tonsillitis, gastrointestinal symptoms, rash on the hands and feet, redness or white coating of the pharynx, swollen tonsils, history of tonsillectomy, strawberry tongue, and swelling of the anterior cervical lymph nodes with tenderness; influenza vaccination history and vaccination timing. This medical interview information is acquired as needed from information stored in the biometric information detection table shown in Figure 7B. "Two-dimensional code information" is information for identifying a recording medium on which user ID information, information for identifying the user ID, attribute information, medical interview information, and / or a combination thereof are recorded. Such a recording medium does not necessarily have to be a two-dimensional code. Instead of a two-dimensional code, various information can be used, such as a one-dimensional barcode, other multidimensional codes, text information such as specific numbers or letters, image information, etc. "Determination result information" is information that indicates the determination result of the possibility of influenza infection based on the determination image. An example of such determination result information is the positivity rate for influenza. However, it is not limited to the positivity rate, and any information that indicates the possibility, such as information specifying whether the result is positive or negative, may be used. Furthermore, the determination result does not need to be a specific numerical value, and may be in any format, such as a classification according to the level of the positivity rate, or a classification indicating whether the result is positive or negative.

[0076] The attribute information and medical interview information do not need to be input by the user or operator each time, but may be received from, for example, an electronic medical record device or other terminal device connected via a wired or wireless network. Alternatively, they may be acquired by analyzing the subject image captured by the imaging device 200. Furthermore, although not specifically shown in FIGS. 7A to 7C, it is also possible to store in the memory 112 information on the current epidemic of infectious diseases that are the subject of diagnosis or diagnostic support, such as influenza, as well as external factor information such as the results of assessments and the status of other users affected by these infectious diseases.

[0077] 4. Processing sequence executed by the processing device 100 and the imaging device 200 Fig. 8 is a diagram showing a processing sequence executed between the processing device 100 and the photographing device 200 according to an embodiment of the present disclosure. Specifically, Fig. 8 shows a processing sequence executed from when a photographing mode is selected in the processing device 100, when a subject image is photographed by the photographing device 200, until when a determination result is output from the processing device 100.

[0078] 8, the processing device 100 outputs a mode selection screen via the output interface 114 and accepts a mode selection by the operator via the input interface 113 (S11). Then, when the selection of the imaging mode is accepted, the processing device 100 outputs an input screen for attribute information via the output interface 114. The processing device 100 accepts input by the operator or user via the input interface 113, acquires attribute information, and stores the attribute information in a user table in association with user ID information (S12). Furthermore, when the attribute information is acquired, the processing device 100 outputs an input screen for medical interview information via the output interface 114. The processing device 100 accepts input by the operator or user via the input interface 113, acquires medical interview information, and stores the information in a user table in association with user ID information (S13). Note that the attribute information and medical interview information do not need to be acquired at this timing, and can also be acquired at another timing, such as before the determination process. Furthermore, this information may be acquired not only by receiving input via the input interface 113 but also by receiving it from an electronic medical record device or other terminal device connected via a wired or wireless network. Furthermore, this information may be acquired by inputting it into the electronic medical record device or other terminal device, recording it on a recording medium such as a two-dimensional code, and then photographing the recording medium with a camera or photographing device 200 connected to the processing device 100. Furthermore, this information may be acquired by having a user, operator, patient, medical professional, etc. fill it out on a paper medium such as a medical questionnaire, and then scanning the paper medium with a scanner or photographing device 200 connected to the processing device 100 and performing optical character recognition.

[0079] Under the control of the processor 111, the processing device 100 generates a two-dimensional code recording the previously generated user ID information and stores it in the memory 112 (S14). Then, the processing device 100 outputs the generated two-dimensional code via the output interface 114 (S15).

[0080] Next, the photographing device 200 starts up the camera 211 etc. by accepting an input from the operator to the input interface 210 (for example, a power button) (S21). Then, the photographing device 200 reads the user ID information recorded in the two-dimensional code by photographing the two-dimensional code output via the output interface 114 with the activated camera 211 (S22).

[0081] Next, the operator covers the tip of the photographing device 200 with the auxiliary tool 300 and inserts the photographing device 200 into the user's oral cavity to a predetermined position. When the photographing device 200 receives input from the operator via the input interface 210 (e.g., a photographing button), it starts photographing an image of the subject that includes at least a part of the oral cavity (S23). The photographing device 200 processes the photographed image of the subject, and when the camera 211 captures the subject (particularly the pharynx) within its angle of view, it starts detecting living body detection information using the information detection sensor 232 (S24). When photographing the subject image and detecting the living body detection information are completed, the photographing device 200 stores the photographed subject image and the corresponding detection information in the memory 214 in association with the user ID information read from the two-dimensional code, and outputs the photographed subject image on the display panel 215 of the display (S25). Then, when the photographing device 200 receives an input from the operator indicating the end of photographing via the input interface 210, it transmits the stored subject image and biometric detection information (T21) to the processing device 100 via the communication interface 216, in association with the user ID information.

[0082] Next, when the processing device 100 receives the subject image and the living body detection information via the communication interface 115, it stores them in the memory 112 and registers them in the image management table based on the user ID information. The processing device 100 selects a determination image from the stored subject images to be used to determine the possibility of influenza infection (S31). Although not specifically shown, the processing device 100 processes the received living body detection information to convert it into living body detection information that can be used in the determination process and stores it. When the determination image is selected, the processing device 100 executes the determination process for the possibility of influenza infection using the selected determination image and the stored living body detection information (S32). When the determination result is obtained, the processing device 100 stores the obtained determination result in association with the user ID information in the user table and outputs it via the output interface 114 (S33). This completes the processing sequence.

[0083] 5. Processing flow executed by the processing device 100 (mode selection processing, etc.) Fig. 9 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 9 is a diagram showing a processing flow executed at a predetermined cycle for the processes related to S11 to S15 in Fig. 8. The processing flow is mainly performed by the processor 111 of the processing device 100 reading and executing a program stored in the memory 112.

[0084] 9, the processor 111 outputs a mode selection screen via the output interface 114 (S111). Here, FIG. 18 is a diagram illustrating an example of a screen displayed on the processing device 100 according to an embodiment of the present disclosure. Specifically, FIG. 18 illustrates an example of a mode selection screen output in S111 and S112 of FIG. 9. According to FIG. 18, approximately at the center of the display functioning as the output interface 114, a shooting mode icon 11 for switching to a shooting mode for shooting an image of a subject, and a determination result confirmation mode icon 12 for switching to a determination result confirmation mode for outputting the results of an existing determination of the possibility of influenza on the display are displayed. The user can select which mode to switch to by operating the input interface 113.

[0085] 9 again, the processor 111 determines whether or not a mode selection by the operator has been accepted via the input interface 113 (S112). At this time, if the processor 111 determines that no input has been made to either the shooting mode icon 11 or the judgment result confirmation mode icon 12 shown in FIG. 18 and that no mode selection has been accepted, the processing flow ends.

[0086] On the other hand, when processor 111 determines that a mode has been selected by receiving input to either shooting mode icon 11 or determination result confirmation mode icon 12 shown in Fig. 18, processor 111 determines whether or not a shooting mode has been selected (S113). Then, when processor 111 determines that determination result confirmation mode icon 12 shown in Fig. 18 has been selected, it displays the desired determination result via output interface 114 (S118).

[0087] On the other hand, when the processor 111 determines that the shooting mode icon 11 shown in FIG. 18 has been selected, it displays a screen (not shown) on the output interface 114 for accepting input of the user's attribute information. The screen includes items such as the user's name, gender, age, and address that need to be input as attribute information, as well as input boxes for inputting answers to each item. The processor 111 then acquires the information input into each input box via the input interface 113 as attribute information (S114). The processor 111 then generates new user ID information corresponding to the user newly stored in the user table, and stores the attribute information in the user table in association with the user ID information. Note that if user ID information is selected in advance before inputting attribute information, the generation of new user ID information can be omitted.

[0088] Next, processor 111 displays a screen (not shown) on output interface 114 for accepting input of the user's medical interview information. The screen includes items that need to be input as medical interview information, such as the user's body temperature, heart rate, medication status, and the presence or absence of subjective symptoms, as well as input boxes for inputting answers to each item. Processor 111 then acquires the information input into each input box via input interface 113 as medical interview information, and stores it in a user table in association with user ID information (S115).

[0089] The attribute information and medical interview information have been described as being input by the processing device 100. However, this is not limiting, and the information may be acquired by receiving information input to an electronic medical record device or other terminal device connected via a wired or wireless network.

[0090] Next, processor 111 refers to the user table, reads out the user ID information corresponding to the user who entered this information, and generates a two-dimensional code that records this (S116). Processor 111 associates the generated two-dimensional code with the user ID information, stores it in the user table, and outputs it via output interface 114 (S117). This ends the processing flow.

[0091] Here, Fig. 19 is a diagram showing an example of a screen displayed on the processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 19 is a diagram showing an example of a display screen of the two-dimensional code output in S117 of Fig. 9. According to Fig. 19, the user ID information of the user to whom attribute information, etc. has been input is displayed above the display functioning as the output interface 114. In addition to this, the two-dimensional code generated in S116 of Fig. 19 is displayed approximately in the center of the display. By photographing the two-dimensional code with the photographing device 200, it is possible to read the user ID information recorded in the two-dimensional code.

[0092] 6. Processing flow executed by the photographing device 200 (photographing process, etc.) Fig. 10 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 10 is a diagram showing a processing flow executed at a predetermined cycle for the processes related to S21 to S25 in Fig. 8. The processing flow is mainly performed by the processor 213 of the imaging device 200 reading and executing a program stored in the memory 214.

[0093] 10, the processor 213 determines whether or not an input from the operator has been accepted via the input interface 210 (for example, a power button) (S211). At this time, if the processor 213 determines that an input from the operator has not been accepted, the processing flow ends.

[0094] On the other hand, when the processor 213 determines that an input by the operator has been accepted, it outputs a standby screen to the display panel 215 (S212). The standby screen (not shown) includes a through image captured by the camera 211. Then, when the operator moves the photographing device 200 so that the two-dimensional code output to the output interface 114 of the processing device 100 is included in the angle of view of the camera 211, the processor 213 photographs the two-dimensional code with the camera 211 (S213). When the two-dimensional code is photographed, the processor 213 reads the user ID information recorded in the two-dimensional code and stores the read user ID information in the memory 214 (S214). Then, the processor 213 outputs the standby screen again to the display panel 215 (S215).

[0095] Next, the operator covers the tip of the photographing device 200 with the auxiliary tool 300 and inserts the photographing device 200 into the oral cavity of the user to a predetermined position. Then, when the processor 213 receives an operation to start photographing by the operator via the input interface 210 (e.g., the photographing button), the processor 213 controls the camera 211 to start photographing subject images of the subject (S216). These subject images are photographed by pressing the photographing button to continuously photograph a fixed number of images (e.g., 30 images) at fixed intervals. When the processor 213 has finished photographing the subject images, it stores the photographed subject images in memory 214 in association with the read user ID information.

[0096] The photographing device 200 also processes the subject image captured as a through image, for example, to determine whether the camera 211 has captured the subject (particularly the pharynx) within its field of view. As an example, the photographed subject image is input to a trained judgment image selection model, and the processor 213 determines whether a candidate image has been obtained. Then, at the timing when it is determined that a candidate image has been obtained, the information detection sensor 232 is turned on to start detecting living body detection information (S217). This allows detection to begin when the photographing device 200 captures the subject, i.e., when the information detection sensor 232 is in an appropriate orientation and position. When the drive of each sensor arranged as the information detection sensor 232 begins, the processor 213 detects the desired living body detection information. When the detection of the living body detection information is completed, the processor 213 associates the detected living body detection information with user ID information and stores it in the memory 214. Note that the use of the trained judgment image selection model is merely an example. For example, the subject image may be segmented using semantic segmentation or the like to determine whether the pharynx or the like is included in the center of the image. Furthermore, the judgment using the learned judgment image selection model or the judgment by segmentation may be performed by the processor 213 of the imaging device 200, or may be performed by the processor 112 of the processing device 100 that receives the subject image, and the judgment result may be transmitted to the imaging device 200. Furthermore, the detection itself by the information detection sensor 232 may start when the imaging button is pressed or may be performed continuously, and only the living body detection information synchronized with the timing when the candidate image is obtained may be extracted.

[0097] Then, the processor 213 outputs the stored subject image and living body detection information to the display panel 215 (S218).

[0098] Here, the operator removes the photographing device 200 together with the auxiliary tool 300 from the oral cavity, checks the subject image and living body detection information output on the display panel 215, and can input an instruction to re-photograph if the desired image or desired living body detection information is not obtained. Therefore, the processor 213 determines whether or not input of an instruction to re-photograph has been received from the operator via the input interface 210 (S219). If input of an instruction to re-photograph has been received, the processor 213 displays the standby screen of S215 again, enabling photographing of the subject image and detection of living body detection information.

[0099] On the other hand, if the input of the instruction to re-photograph has not been accepted, and the operator returns the photographing device 200 to the mounting base 400 and receives an instruction to end photographing from the processing device 100, the processor 213 transmits the subject image, the living body detection information, and the user ID information associated with the subject image stored in the memory 214 to the processing device 100 via the communication interface 216 (S220). This ends the processing flow.

[0100] 7. Processing flow (determination processing, etc.) executed by the processing device 100 Fig. 11 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 11 is a diagram showing a processing flow executed for the processes related to S31 to S33 in Fig. 8. The processing flow is mainly performed by the processor 111 of the processing device 100 reading and executing a program stored in the memory 112.

[0101] 11, when the processor 111 receives a subject image and user ID information associated therewith from the photographing device 200, it stores the image in the memory 112 and registers the image in the image management table (S311). Then, the processor 111 outputs the received user ID information or corresponding attribute information (e.g., name) via the output interface 114, and accepts selection of a user to be judged for the possibility of influenza (S312). At this time, if multiple pieces of user ID information and corresponding subject images are received from the photographing device 200, it is possible to output multiple pieces of user ID information or corresponding attribute information to select one of the users.

[0102] When processor 111 receives the selection of the user to be determined via input interface 113, it reads out attribute information associated with the user ID information of that user from the user table in memory 112 (S313). Similarly, processor 111 reads out medical interview information associated with the user ID information of the user to be determined from the user table in memory 112 (S314). Note that it is also possible to read out information stored in the living body detection information table as this medical interview information. That is, it is possible to acquire information such as temperature (body temperature), exhaled air, respiratory sounds, heart rate, oxygen saturation, near-infrared images (or blood vessel conditions estimated from the images), and ultraviolet images (or structural information of the living body surface obtained from the images) as medical interview information from the living body detection information table without requiring input by the user, etc.

[0103] Next, the processor 111 reads out from the memory 112 the subject image associated with the user ID information of the selected user, and executes a process of selecting a determination image to be used in determining the possibility of influenza (S315: details of this selection process will be described later). Then, the processor 111 executes a process of determining the possibility of influenza based on the selected determination image and the biological detection information used as medical interview information (S316: details of this determination process will be described later). When a determination result is obtained by the determination process, the processor 111 stores the result in a user table in association with the user ID information, and outputs the determination result via the output interface 114 (S317). This ends the processing flow.

[0104] Fig. 12 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 12 is a diagram showing details of the determination image selection process executed in S315 of Fig. 11. This processing flow is mainly performed by the processor 111 of the processing device 100 reading and executing a program stored in the memory 112.

[0105] 12, the processor 111 reads out from the memory 112 the subject images associated with the user ID information of the selected user (S411). Next, the processor 412 selects images that will be candidates for the judgment image from the read out subject images (S412). This selection is performed, for example, using a trained judgment image selection model.

[0106] Here, Fig. 13 is a diagram showing a processing flow related to generation of a trained model according to an embodiment of the present disclosure. Specifically, Fig. 13 is a diagram showing a processing flow related to generation of a trained judgment image selection model used in S412 of Fig. 12. The processing flow may be executed by the processor 111 of the processing device 100, or may be executed by a processor of another processing device.

[0107] As shown in FIG. 13, the processor executes a step of acquiring a subject image of a subject including at least a portion of the pharynx as a training subject image (S511). Next, the processor executes a processing step of assigning label information to the acquired training subject image, indicating whether the image can be used as a judgment image (S512). The processor then executes a step of storing the assigned label information in association with the training subject image (S513). Note that the label assignment process and label information storage process may involve a human determining in advance whether the training subject image is a judgment image, and the processor storing the result in association with the training subject image. Alternatively, the processor may perform an analysis using a known image analysis process to determine whether the image is a judgment image, and store the result in association with the training subject image. The label information is assigned based on factors such as whether at least a portion of the oral cavity, which is the subject, is captured, and whether the image quality is good due to factors such as camera shake, defocus, and cloudiness.

[0108] Once the training object images and the associated label information are obtained, the processor executes a step of performing machine learning of a selection pattern of a judgment image using the training object images and the associated label information (S514). For example, the machine learning is performed by providing a set of training object images and label information to a neural network configured by combining neurons, and repeating learning while adjusting the parameters of each neuron so that the output of the neural network is the same as the label information. Then, a step of acquiring a trained judgment image selection model (e.g., neural network and parameters) is executed (S515). The acquired trained judgment image selection model may be stored in memory 112 of the processing device 100 or another processing device connected to the processing device 100 via a wired or wireless network.

[0109] Returning to FIG. 12 again, the processor 111 inputs the subject image read out in S411 into the learned judgment image selection model, thereby acquiring as output a candidate image that is a candidate for the judgment image. This makes it possible to select an image that captures at least a partial region of the oral cavity that is the subject, or an image with good image quality that is free from camera shake, defocus, subject motion blur, exposure, and cloudiness. Furthermore, it is possible to stably select images with good image quality regardless of the operator's skill in photographing. Then, the processor 111 registers the acquired candidate image that is a candidate for the judgment image in the image management table.

[0110] Next, processor 111 executes a process of selecting a judgment image from the selected candidate images based on similarity (S413). Specifically, processor 111 compares the obtained candidate images with each other and calculates the similarity between each candidate image. Then, processor 111 selects a candidate image that is determined to have low similarity to other candidate images as a judgment image. Such similarity between each candidate image is calculated by a method using local features in each candidate image (Bag-of-Keypoints method), a method using Earth Mover's Distance (EMD), a method using Support Vector Machine (SVM), a method using Hamming distance, a method using cosine similarity, or the like.

[0111] In this way, by calculating the similarity between the obtained candidate images and selecting a candidate image that is determined to have a low similarity to other candidate images, subject images with different fields of view are selected as judgment images. This makes it possible to perform judgment processing based on more diverse information than when subject images obtained with the same field of view are used as judgment images, thereby making it possible to further improve judgment accuracy. Specifically, even if a part of the pharynx is hidden by the uvula in one judgment image, the hidden part of the pharynx is visible in another judgment image with a different field of view, so it is possible to prevent important features such as influenza follicles from being overlooked.

[0112] Next, the processor 111 registers the candidate image selected based on the similarity as a determination image in the image management table (S414).

[0113] Here, the subject image, candidate image, and judgment image may each be one or more. However, as an example, it is preferable to select a group of candidate images from a group of, for example, 5 to 30 subject images, and ultimately obtain a group of about five judgment images. This is because selecting a judgment image from a large number of subject images increases the likelihood of obtaining a better judgment image. Furthermore, using multiple judgment image groups in the judgment process described below can improve judgment accuracy compared to using only one judgment image. As another example, each time a subject image is captured, the captured subject image may be transmitted to the processing device 100, and then candidate images and judgment images may be selected. Alternatively, the image capturing device 200 may select candidate images and judgment images, and the image capturing process may be terminated when a predetermined number of judgment images (e.g., about five) have been obtained. This minimizes the time required to capture subject images while maintaining the improved judgment accuracy as described above. In other words, discomfort to the user, such as vomiting reflex, may be reduced.

[0114] Fig. 14 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 14 is a diagram showing details of the process for determining the possibility of influenza, which is executed in S316 of Fig. 11. This processing flow is mainly performed by the processor 111 of the processing device 100 reading and executing a program stored in the memory 112.

[0115] According to FIG. 14, the processor 111 acquires the determination result by performing ensemble processing on the first positive rate, the second positive rate, and the third positive rate, each of which has been acquired by a different method.

[0116] First, the process of acquiring the first positive rate will be described. The processor 111 reads out from the memory 112 a determination image associated with the user ID information of the user to be determined (S611). The processor 111 then performs predetermined preprocessing on the read determination image. Examples of such preprocessing include filtering such as bandpass filters including high-pass filters and low-pass filters, averaging filters, Gaussian filters, Gabor filters, Canny filters, Sobel filters, Laplacian filters, median filters, and bilateral filters; blood vessel extraction processing using a Hessian matrix or the like; segmentation processing of specific regions (e.g., follicles) using machine learning; trimming processing of the segmented region; haze removal processing; super-resolution processing; and combinations thereof, selected according to the purpose, such as high definition, region extraction, noise removal, edge enhancement, image correction, and image conversion. By performing preprocessing in this way, regions of interest important in diagnosing diseases, such as influenza follicles, can be extracted and emphasized in advance, thereby improving the accuracy of diagnosis.

[0117] Here, specific examples of such preprocessing, including defogging, super-resolution, and segmentation, are described below. First, the defogging process, for example, uses a trained defogging image model obtained by machine learning, where a set of training subject images and training degraded images obtained by adding defogging to the training subject images using a defogging filter or the like is provided to a learning device. The processor 111 inputs the read-out judgment image as input to the trained defogging image model stored in memory 112, and obtains an judgment image from which the defogging has been removed as output. The super-resolution process uses a trained super-resolution image model obtained by machine learning, where a set of a high-resolution image of the subject and a low-resolution image obtained by performing degradation processing, such as scaling down or blurring, on the high-resolution image is provided to a learning device as training images. The processor 111 inputs the judgment image from which the defogging has been performed as input to the trained super-resolution image model stored in memory 112, and obtains an judgment image from which the super-resolution processing has been performed as output. The segmentation process uses a trained segmentation image model obtained by machine learning, which is obtained by providing a learning device with a set of training subject images and position information of the labels obtained by labeling regions of interest (e.g., follicles) based on operational input by doctors for the training subject images. The processor 111 inputs the super-resolution processed judgment image and the trained segmentation image model stored in the memory 112, and acquires a judgment image in which the regions of interest (e.g., follicles) are segmented. The processor 111 then stores the pre-processed judgment image in the memory 112. Here, the processes are described in the order of defogging, super-resolution, and segmentation, but any order is acceptable, or at least one of the processes may be performed. While the examples of all processes use trained models, other processes such as defogging filters, scaling, and sharpening may also be used.

[0118] The processor 111 then provides the preprocessed determination image as an input to a feature extractor (S613) and acquires the image feature of the determination image as an output (S614). Furthermore, the processor 111 provides the feature of the acquired determination image as an input to a classifier (S615) and acquires a first positive rate indicating a first possibility of influenza infection as an output (S616). The feature extractor can acquire a predetermined number of feature vectors, such as the presence or absence of follicles or redness in the determination image. As an example, 1024-dimensional feature vectors are extracted from the determination image, and these are stored as the feature of the determination image.

[0119] Here, Fig. 15 is a diagram showing a processing flow related to generation of a trained model according to an embodiment of the present disclosure. Specifically, Fig. 15 is a diagram showing a processing flow related to generation of a trained positive rate determination and selection model including the feature extractor of S613 and the classifier of S615 in Fig. 14. This processing flow may be executed by the processor 111 of the processing device 100, or may be executed by a processor of another processing device.

[0120] 15, the processor executes a step of acquiring an image as a judgment image for learning by performing preprocessing similar to that of S612 in Fig. 14 on an image of a subject including at least a part of the pharynx (S711). Next, the processor executes a processing step of assigning a correct answer label to the user who is the subject of the acquired judgment image for learning, the correct answer label being assigned in advance based on the results of a rapid influenza test by immunochromatography, a PCR test, a virus isolation and culture test, etc. (S712). Then, the processor executes a step of storing the assigned correct answer label information as judgment result information in association with the judgment image for learning (S713).

[0121] Once the training judgment images and the corresponding correct label information are obtained, the processor executes a step of performing machine learning of a positive rate judgment pattern using them (S714). As an example, this machine learning is performed by providing a pair of the training judgment images and the correct label information to a feature extractor formed from a convolutional neural network and a classifier formed from a neural network, and repeating learning while adjusting the parameters of each neuron so that the output from the classifier is the same as the correct label information. Then, a step of acquiring a trained positive rate judgment model is executed (S715). The acquired trained positive rate judgment model may be stored in the memory 112 of the processing device 100 or another processing device connected to the processing device 100 via a wired or wireless network.

[0122] Returning to Figure 14, the processor 111 inputs the judgment image preprocessed in S612 into the learned positive rate judgment model, thereby obtaining as output the features of the judgment image (S614) and a first positive rate (S616) indicating the first possibility of contracting influenza, and stores them in memory 112 in association with the user ID information.

[0123] Next, the process of acquiring the second positive rate will be described. The processor 111 reads out from the memory 112 the medical interview information including the living body detection information associated with the user ID information of the user to be judged, and attribute information as necessary (S617). The processor 111 also reads out from the memory 112 the feature amount of the judgment image calculated in S614 and stored in the memory 112 in association with the user ID information (S614). The processor 111 then provides the read out medical interview information including the living body detection information and the feature amount of the judgment image as inputs to the trained positive rate judgment model (S618), and acquires as output a second positive rate indicating a second possibility of contracting influenza (S619).

[0124] Here, Fig. 16 is a diagram showing a processing flow related to generation of a trained model according to an embodiment of the present disclosure. Specifically, Fig. 16 is a diagram showing a processing flow related to generation of a trained positive rate determination and selection model in S618 of Fig. 14. This processing flow may be executed by the processor 111 of the processing device 100, or may be executed by a processor of another processing device.

[0125] 16, the processor executes a step of acquiring learning features from a determination image obtained by performing preprocessing similar to S612 in FIG. 14 on an image of a subject including at least a portion of the pharynx (S721). The processor also executes a step of acquiring medical interview information and attribute information including living body detection information that have been previously stored in association with user ID information of the user who is the subject of the determination image (S721). Next, the processor executes a processing step of assigning a correct label to the user who is the subject of the determination image based on the results of a rapid influenza test by immunochromatography, a PCR test, a virus isolation and culture test, etc. (S722). The processor then executes a step of storing the assigned correct label information as determination result information in association with the learning features of the determination image and the medical interview information and attribute information including living body detection information (S723).

[0126] Once the learning features of the judgment image, the medical interview information and attribute information including the biological detection information, and the corresponding correct label information are obtained, the processor executes a step of performing machine learning of a positive rate judgment pattern using these (S724). As an example, this machine learning is performed by providing these sets of information to a neural network combining neurons, and repeating learning while adjusting the parameters of each neuron so that the output from the neural network is the same as the correct label information. Then, a step of acquiring a trained positive rate judgment model is executed (S725). The acquired trained positive rate judgment model may be stored in the memory 112 of the processing device 100 or another processing device connected to the processing device 100 via a wired or wireless network.

[0127] Returning to Figure 14, the processor 111 inputs the features of the judgment image read out in S614 and the medical interview information including the biometric detection information read out in S617 into the learned positive rate judgment model, thereby obtaining as output a second positive rate (S619) indicating a second possibility of influenza infection, and stores it in memory 112 in association with the user ID information.

[0128] Next, the process of acquiring the third positive rate will be described. Processor 111 reads out from memory 112 medical interview information including living body detection information associated with the user ID information of the user to be determined, and attribute information as necessary (S617). Processor 111 also reads out from memory 112 the first positive rate calculated in S616 and stored in memory 112 in association with the user ID information. Processor 111 then provides the read medical interview information including living body detection information and the first positive rate as inputs to the trained positive rate determination model (S620), and acquires as output a third positive rate indicating a third possibility of contracting influenza (S621).

[0129] Here, Fig. 17 is a diagram showing a processing flow related to generation of a trained model according to an embodiment of the present disclosure. Specifically, Fig. 17 is a diagram showing a processing flow related to generation of a trained positive rate determination and selection model in S620 of Fig. 14. This processing flow may be executed by the processor 111 of the processing device 100, or may be executed by a processor of another processing device.

[0130] 17, the processor executes a step of acquiring first positivity information by inputting a judgment image obtained by performing preprocessing similar to that of S612 of FIG. 14 on an image of a subject including at least a portion of the pharynx into a learned positivity rate judgment and selection model including a feature extractor (S613 of FIG. 14) and a classifier (S615 of FIG. 14) (S731). The processor also executes a step of acquiring medical interview information and attribute information including liveness detection information that have been stored in advance in association with user ID information of the user who is the subject of the judgment image (S731). Next, the processor executes a processing step of assigning a correct label to the user who is the subject of the judgment image based on the results of a rapid influenza test by immunochromatography, a PCR test, a virus isolation and culture test, etc. (S732). The processor then executes a step of storing the assigned correct label information as judgment result information in association with the first positivity rate information, and the medical interview information and attribute information including liveness detection information (S733).

[0131] Once the first positive rate information, medical interview information including biological detection information, attribute information, and corresponding correct label information are obtained, the processor executes a step of performing machine learning of a positive rate determination pattern using these information (S734). As an example, this machine learning is performed by providing these sets of information to a neural network combining neurons, and repeating learning while adjusting the parameters of each neuron so that the output from the neural network is the same as the correct label information. Then, a step of acquiring a trained positive rate determination model is executed (S735). The acquired trained positive rate determination model may be stored in the memory 112 of the processing device 100 or in another processing device connected to the processing device 100 via a wired or wireless network.

[0132] Returning to Figure 14, the processor 111 inputs the first positive rate information read out in S616 and the medical interview information including the biological detection information read out in S617 into the learned positive rate determination model, thereby obtaining as output a third positive rate (S621) indicating a third possibility of contracting influenza, and stores it in memory 112 in association with the user ID information.

[0133] Once the first positive rate, second positive rate, and third positive rate have been calculated in this manner, the processor 111 reads each positive rate from the memory 112 and performs ensemble processing (S622). As an example of this ensemble processing, the obtained first positive rate, second positive rate, and third positive rate are input to a ridge regression model, and the ensemble result of each positive rate is obtained as a determination result of the possibility of contracting influenza (S623).

[0134] The ridge regression model used in S622 is generated by machine learning using the processor 111 of the processing device 100 or a processor of another processing device. Specifically, the processor acquires the first positive rate, the second positive rate, and the third positive rate from the learning judgment image. The processor also assigns a correct label to the user who is the subject of the learning judgment image, based on the results of a rapid influenza test using immunochromatography, a PCR test, a virus isolation and culture test, or the like. The processor then provides a ridge regression model with a set of each positive rate and its corresponding correct label, and repeats learning while adjusting the parameters assigned to each positive rate so that the output is the same as the correct label information of the ridge regression model. This results in a ridge regression model used for ensemble processing, which is stored in the memory 112 of the processing device 100 or another processing device connected to the processing device 100 via a wired or wireless network.

[0135] Furthermore, although the use of a ridge regression model has been described as an example of ensemble processing, any method may be used, such as a process for obtaining the average value of each positive rate, a process for obtaining the maximum value, a process for obtaining the minimum value, a weighted addition process, or a process using other machine learning methods such as bagging, boosting, stacking, lasso regression, or linear regression.

[0136] Processor 111 stores the determination result thus obtained in the user table in memory 112 in association with the user ID information (S624), thereby ending this processing flow.

[0137] In Fig. 14, ensemble processing is performed on the first positive rate, the second positive rate, and the third positive rate to obtain a final determination result. However, this is not limiting, and each positive rate may be used as the final determination result as is, or ensemble processing may be performed using any two positive rates to obtain a final determination result. Furthermore, other positive rates obtained by other methods may be further added and ensemble processing may be performed to obtain a final determination result.

[0138] As shown in FIG. 11, the obtained determination results are output via the output interface 114, but it is also possible to output only the final determination results, or to output each positive rate together.

[0139] As described above, this embodiment provides a processing device, a processing program, a processing method, and a processing system suitable for processing images obtained by photographing the inside of the oral cavity for use in intraoral diagnosis. In particular, this embodiment uses the information detection sensor 232 disposed in the photographing device 200 to detect biometric information of the living body of the user while photographing the inside of the oral cavity, and determines the possibility of disease based on this information. Therefore, biometric information obtained at a position closer to the area photographed by the photographing device 200, i.e., the area of ​​interest, and at the same time, can be used for the determination, enabling more accurate determination.

[0140] 8. Variations 10, detection of living body detection information is started at the timing when it is detected in S216 that a subject (particularly the pharynx) has been captured in the angle of view of the camera 211 (S217). However, this is not limiting, and detection may be started simultaneously with pressing of the capture button to start capturing an image of the subject. Furthermore, detection of living body detection information, that is, start of detection by the information detection sensor 232, may be triggered by pressing of the capture button or the like again. Furthermore, the detection may be started when the power is turned on, and may be continuous.

[0141] In the example of FIG. 14, a case has been described in which information indicating the possibility of influenza is output using at least one of medical interview information and attribute information including biometric detection information. However, instead of or in addition to this information, external factor information related to influenza may be used to output information indicating the possibility of influenza. Such external factor information may include assessment results made on other users, diagnosis results by doctors, and influenza epidemic information in the user's area. The processor 111 acquires such external factor information from other processing devices via the communication interface 115 and provides the external factor information as input to the trained positive rate determination model, thereby making it possible to obtain a positive rate that takes the external factor information into account.

[0142] In the example of FIG. 14 , the medical interview information and attribute information are input in advance by an operator or user, or are received from an electronic medical record device or the like connected to a wired or wireless network. However, instead of or in addition to these, this information may be obtained from a captured subject image. The attribute information and medical interview information associated with the training subject image are assigned as correct labels to the training subject image, and a trained information estimation model is obtained by machine learning these pairs using a neural network. Then, the processor 111 inputs the subject image to the trained information estimation model, thereby obtaining the desired medical interview information and attribute information. Examples of such medical interview information and attribute information include gender, age, degree of pharyngeal redness, degree of tonsillar swelling, and the presence or absence of white foxing. This eliminates the need for the operator to input the medical interview information and attribute information.

[0143] The following describes a specific example of medical interview information, in which feature quantities of pharyngeal follicles are obtained. As described above, follicles appearing in the pharynx are a characteristic sign of influenza and are visually confirmed by doctors during diagnosis. Therefore, a labeling process is performed on regions of interest, such as follicles, based on operational input by doctors on the training subject images. Position information (shape information) of the labels in the training subject images is acquired as training position information, and a trained region extraction model is obtained by machine learning a set of the training subject images and the labeled training position information using a neural network. Processor 111 then provides the subject images as input to the trained region extraction model, outputting position information (shape information) of the regions of interest (i.e., follicles). Processor 111 then stores the acquired position information (shape information) of the follicles as medical interview information.

[0144] In the example of Fig. 14, a case has been described in which the determination image read from the memory 112 is preprocessed in S612 and then provided as an input to the feature extractor. However, preprocessing is not necessarily required. For example, the processor 111 may read the determination image from the memory 112 and provide the read determination image as an input to the feature extractor without preprocessing. Furthermore, even when preprocessing is performed, the processor 111 may provide both the preprocessed determination image and the non-preprocessed determination image as input to the feature extractor.

[0145] 15 to 17, judgment images that have been preprocessed in the same manner as in S612 in Fig. 14 are used as training data. However, in consideration of the cases where preprocessing is not performed in Fig. 14 as described above, or where both the preprocessed judgment image and the preprocessed judgment image are used as the judgment image, judgment images that have not been preprocessed may be used as training data.

[0146] The trained models described in Figures 13, 15 to 17, etc. were generated using neural networks or convolutional neural networks. However, they are not limited to these, and can also be generated using machine learning techniques such as nearest neighbor methods, decision trees, regression trees, and random forests.

[0147] In the example of FIG. 8 , the processing device 100 acquires attribute information and medical interview information, selects a judgment image, performs judgment processing, and outputs the judgment result, while the photographing device 200 photographs a subject image. However, these various processes can be distributed among the processing device 100, the photographing device 200, and other devices as appropriate. FIG. 20 is a schematic diagram of a processing system 1 according to an embodiment of the present disclosure. Specifically, FIG. 20 is a diagram showing an example of connections between various devices that may constitute the processing system 1. According to FIG. 20 , the processing system 1 includes the processing device 100, the photographing device 200, a terminal device 810 such as a smartphone, tablet, or laptop PC, an electronic medical record device 820, and a server device 830, all of which are connected to each other via a wired or wireless network. Note that each device listed in FIG. 20 does not necessarily need to be provided; it may be provided as appropriate according to the distribution example of processing described below.

[0148] Instead of the example of FIG. 8, in the processing system 1 illustrated in FIG. 20, various processes can be distributed as follows. (1) The imaging device 200 performs all processes such as capturing an image of a subject, acquiring attribute information and medical interview information, selecting an image for determination, performing determination processing, and outputting the determination results. (2) The photographing device 200 photographs the subject image and outputs the judgment result, and the server device 830 (cloud server device) executes the processes using machine learning, such as the selection of the judgment image and the judgment process. (3) The input of medical interview information and attribute information is performed by the terminal device 810, the selection of a judgment image, the judgment process, and the output of the judgment result are performed by the processing device 100, and the photographing device 200 photographs the subject image. (4) The photographing device 200 inputs the medical interview information and attribute information and photographs the subject image, and the processing device 100 selects the judgment image, performs judgment processing, and outputs the judgment result. (5) The input of medical interview information and attribute information and the output of the judgment results are performed by the terminal device 810, the selection of the judgment image and the judgment process are performed by the processing device 100, and the photographing device 200 photographs the subject image. (6) The input of medical interview information and attribute information is performed by the electronic medical record device 820, the selection of the judgment image and the judgment process are performed by the processing device 100, the photographing of the subject image is performed by the photographing device 200, and the output of the judgment result is performed by the terminal device 810. (7) The electronic medical record device 820 inputs the medical interview information and attribute information, and outputs the judgment results. The processing device 100 selects the judgment image and performs the judgment process. The imaging device 200 captures the subject image. (8) The input of medical interview information and attribute information and the output of the judgment results are performed by the terminal device 810, the selection of the judgment image and the judgment process are performed by the server device 830, and the photographing device 200 photographs the subject image. (9) The input of medical interview information and attribute information and the output of judgment results are performed by the terminal device 810 and the electronic medical record device 820, the selection of judgment images and judgment processing are performed by the server device 830, and the photographing device 200 photographs the subject image. (10) The electronic medical record device 820 inputs the medical interview information and attribute information, and outputs the judgment results. The server device 830 selects the judgment image and performs the judgment process. The imaging device 200 captures the subject image.

[0149] One example of the above-mentioned distributed processing will be specifically described. The processing relating to S11 to S15 shown in Fig. 8 is executed in a terminal device 810 such as a smartphone carried by a patient or the like, a tablet used in a medical institution or the like, or a laptop PC used by a doctor or the like. After that, when the processing relating to S21 to S24 is executed in the imaging device 200, the subject image is transmitted to the server device 830 via the terminal device 810 or directly. The server device 830 that has received the subject image executes the processing relating to S31 to S32, and the server device 830 outputs a determination result to the terminal device 810. The terminal device 810 that has received the output of the determination result stores the determination result in a memory and displays it on a display.

[0150] Note that the above is merely an example of distribution of processing. Also, in this disclosure, the processing device 100 is referred to as a processing device. However, this is merely because the processing device 100 executes various processes related to the determination process, etc. For example, when various processes are executed by the imaging device 200, the terminal device 810, the electronic medical record device 820, the server device 830, etc., these also function as processing devices and may be referred to as processing devices.

[0151] 3, the subject image is captured using a substantially cylindrical imaging device 200. However, the present invention is not limited to this, and it is also possible to use, for example, a terminal device 810 as the imaging device and capture the subject image using a camera provided in the terminal device 810. In such a case, the camera is not inserted into the oral cavity near the pharynx, but is placed outside the incisors (outside the body) to capture images of the oral cavity.

[0152] These modified examples have the same configurations, processes, and procedures as the embodiment described in Figures 1 to 19, except for the points specifically described above. Therefore, detailed descriptions of these matters will be omitted. Furthermore, it is also possible to configure a system by appropriately combining or replacing the elements described in each modified example or each embodiment.

[0153] The processes and procedures described herein can be realized not only by those explicitly described in the embodiments, but also by software, hardware, or a combination thereof. Specifically, the processes and procedures described herein can be realized by implementing logic corresponding to the processes in media such as integrated circuits, volatile memory, nonvolatile memory, magnetic disks, and optical storage. Furthermore, the processes and procedures described herein can be implemented as computer programs and executed by various computers, including processing devices and server devices.

[0154] Although processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, multiple software, multiple components, and / or multiple modules. Furthermore, although various information described herein is described as being stored in a single memory or storage unit, such information may be stored in multiple memories within a single device or multiple memories distributed across multiple devices. Furthermore, software and hardware elements described herein may be realized by integrating them into fewer components or by decomposing them into more components. [Explanation of symbols]

[0155] 1 Processing System 100 Processing equipment 200 Imaging Device 300 Assistive Devices 400 Mounting Table 600 Operator 700 users 810 Portable terminal device 820 Electronic medical record device 830 Server equipment

Claims

[Claim 1] at least one processor; the at least one processor: Acquire one or more determination images of the subject photographed by a photographing device including a camera for photographing an image of the subject including at least a part of the oral cavity of the user, acquiring, from the imaging device, living body detection information of the subject that is different from the one or more determination images; determining a possibility of a patient suffering from a predetermined disease based on a trained determination model stored in a memory for determining a possibility of the patient suffering from the predetermined disease, the one or more determination images obtained, and the biological detection information; outputting information indicating the determined possibility of the disease; a processing device configured to process the

Images