Processing device, processing program, processing method, and processing system

Abstract

[Problem] To use a determination image of a subject to determine the possibility that the subject is suffering from a prescribed disease. [Solution] The present invention: acquires one or a plurality of determination images of a subject via a user camera for taking images of the subject; acquires at least one among medical interview information and attribute information about the user and determines the possibility that the subject is suffering from the prescribed disease on the basis of at least one among a trained determination model for determining the possibility that the subject is suffering from the predetermined disease, the determination images, the medical interview information, and the attribute information; and determines the reliability of the possibility that the subject is suffering on the basis of at least one among the determination images, the medical interview information, and the attribute information.

Description

Processing device, processing program, processing method, and processing system

[0001] The present disclosure relates to a processing device, a processing program, a processing method, and a processing system for processing an image of a subject captured by a camera.

[0002]

[0003] Conventionally, systems have been known in which, for example, a captured image is transmitted to a server device and the server device processes the image using a trained analysis model to obtain a diagnosis result for a predetermined disease. Patent Document 1 (JP-A-2005-102626) describes an example of such a system: "In a diagnosis prediction system, a control unit of a support server divides an original image to generate multiple segmented images, generates a segmented prediction model that predicts a diagnosis result from the segmented images using first teacher information that associates and records diagnosis results for the original image with the multiple segmented images, and records the generated segmented prediction model in a learning result storage unit. Second teacher information is generated by inputting feature information generated by each segmented image of the first teacher information into the segmented prediction model, based on the position of the segmented image, and generates an integrated prediction model that predicts a diagnosis result using the second teacher information. When an image to be evaluated is acquired, the control unit predicts a diagnosis result using the segmented prediction model and the integrated prediction model recorded in the learning result storage unit."

[0003] Japanese Patent Application Laid-Open No. 2022-50089

[0004] Therefore, in light of the above-described technology, the present disclosure aims to provide, in various embodiments, a processing device, a processing program, a processing method, or a processing system for determining the possibility of contracting a specified disease using a judgment image of a subject.

[0005] According to one aspect of the present disclosure, there is provided a processing device including at least one processor configured to perform processing to acquire one or more judgment images of a subject via a camera for capturing an image of the subject of a user, acquire at least one of medical interview information and attribute information of the user, judge the possibility of the subject having a predetermined disease based on a trained judgment model for judging the possibility of the subject having the predetermined disease and at least one of the judgment image, the medical interview information, and the attribute information, and determine the reliability of the possibility of the subject having the disease based on at least one of the judgment image, the medical interview information, and the attribute information.

[0006] According to one aspect of the present disclosure, there is provided a processing program that, when executed by at least one processor, causes the at least one processor to function in the following manner: acquire one or more judgment images of a subject via a camera for capturing an image of the subject of a user; acquire at least one of medical interview information and attribute information of the user; determine the possibility of the subject having a specified disease based on a trained judgment model for determining the possibility of the subject having the specified disease, and at least one of the judgment image, the medical interview information, and the attribute information; and determine the reliability of the possibility of the subject having the disease based on at least one of the judgment image, the medical interview information, and the attribute information.

[0007] According to one aspect of the present disclosure, there is provided a processing method executed by at least one processor, the processing method including: a step of acquiring one or more judgment images of a subject via a camera for capturing an image of the subject of a user; a step of acquiring at least one of medical interview information and attribute information of the user; a step of judging the possibility of the subject having a predetermined disease based on a trained judgment model for judging the possibility of the subject having the predetermined disease and at least one of the judgment image, the medical interview information, and the attribute information; and a step of determining the reliability of the possibility of the subject having the disease based on at least one of the judgment image, the medical interview information, and the attribute information.

[0008] According to one aspect of the present disclosure, there is provided a processing system including: an imaging device equipped with a camera for capturing an image of the user's subject; and a processing device connected to the imaging device via a wired or wireless network.

[0009] According to the present disclosure, it is possible to provide a processing device, a processing program, a processing method, and a processing system suitable for processing images obtained by photographing the inside of the oral cavity for use in intraoral diagnosis.

[0010] It should be noted that the above effects are merely illustrative for the sake of convenience and are not limiting. In addition to or instead of the above effects, any effect described in this disclosure or an effect obvious to a person skilled in the art may be achieved.

[0011] FIG. 1 is a diagram illustrating a usage state of a processing system 1 according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating a usage state of an imaging device 200 according to an embodiment of the present disclosure. FIG. 3 is a schematic diagram of a processing system 1 according to an embodiment of the present disclosure. FIG. 4 is a block diagram illustrating a configuration of a processing device 100 according to an embodiment of the present disclosure. FIG. 5 is a block diagram illustrating configurations of an imaging device 200 and a terminal device 300 according to an embodiment of the present disclosure. FIG. 6A is a diagram conceptually illustrating an image management table stored in a processing device 100 according to an embodiment of the present disclosure. FIG. 6B is a diagram conceptually illustrating a user table stored in a processing device 100 according to an embodiment of the present disclosure. FIG. 7A is a schematic diagram of distribution information of accumulated data stored in a processing device 100 according to an embodiment of the present disclosure. FIG. 7B is a schematic diagram of distribution information of accumulated data stored in a processing device 100 according to an embodiment of the present disclosure. FIG. 7C is a schematic diagram of distribution information of accumulated data stored in a processing device 100 according to an embodiment of the present disclosure. FIG. 8 is a diagram illustrating a processing sequence executed between a processing device 100, an imaging device 200, and a terminal device 300 according to an embodiment of the present disclosure. FIG. 9 is a diagram showing a processing flow executed in the imaging device 200 according to an embodiment of the present disclosure. FIG. 10 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. FIG. 11 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. FIG. 12 is a diagram showing a processing flow related to generation of a trained model according to an embodiment of the present disclosure. FIG. 13 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. FIG. 14 is a diagram showing a processing flow related to generation of a trained model according to an embodiment of the present disclosure. FIG. 15 is a diagram showing a processing flow related to generation of a trained model according to an embodiment of the present disclosure. FIG. 16 is a diagram showing a processing flow related to generation of a trained model according to an embodiment of the present disclosure. FIG. 17 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. FIG. 18 is a schematic diagram of a reliability determination process according to an embodiment of the present disclosure. FIG. 19 is a schematic diagram of a reliability determination process according to an embodiment of the present disclosure.

[0012] Various embodiments of the present disclosure will be described with reference to the accompanying drawings, in which common elements are designated by the same reference numerals.

[0013] 1. Overview of Processing System 1 The processing system 1 according to the present disclosure is primarily used to capture images of the inside of a user's oral cavity to obtain a subject image. In particular, the processing system 1 is used to capture images of the back of the throat and surrounding area of ​​the oral cavity, specifically the pharynx. Therefore, the following description will primarily focus on the case where the processing system 1 according to the present disclosure is used to capture images of the pharynx. However, the pharynx is only one example of a region to be captured, and the processing system 1 according to the present disclosure can also be suitably used for other regions within the oral cavity, such as the tonsils. Furthermore, the captured subject image is not limited to an image of the inside of the oral cavity; various regions of the user may be captured depending on the disease being diagnosed.

[0014] The processing system 1 according to the present disclosure is used to determine the possibility of a predetermined disease from an image of a subject obtained by photographing the subject, the subject including at least the pharyngeal region of the oral cavity of a user, and to diagnose or assist in the diagnosis of the predetermined disease. An example of a disease that can be determined by the processing system 1 is influenza. Typically, the possibility of influenza is diagnosed by examining the user's pharynx and tonsils and determining the presence or absence of findings such as follicles in the pharyngeal region. However, by using the processing system 1 to determine the possibility of influenza and output the results, it is possible to diagnose or assist in the diagnosis. Note that determining the possibility of influenza is just one example. The processing system 1 can be suitably used to determine any disease that results in intraoral findings. Note that intraoral findings are not limited to those discovered by a doctor or the like and whose existence is medically known. For example, differences that can be recognized by persons other than doctors or that can be detected using artificial intelligence or image recognition technology can be suitably applied to the processing system 1. Examples of such diseases include influenza, as well as infectious diseases such as streptococcal infection, adenovirus infection, EB virus infection, mycoplasma infection, hand, foot and mouth disease, herpangina, and candidiasis; diseases presenting with vascular or mucosal disorders such as arteriosclerosis, diabetes, and hypertension; tumors such as tongue cancer and pharyngeal cancer; and periodontal diseases such as dental caries, gingivitis, and periodontal disease.

[0015] The processing system 1 according to the present disclosure determines the reliability of the determined possibility of disease. Here, the reliability may be indicated by a numerical value or may be indicated by words divided into multiple levels such as high, medium, and low.

[0016] In the present disclosure, terms such as "determination," "assessment," and "diagnosis" are used in relation to diseases, but these do not necessarily mean a definitive determination, judgment, or diagnosis by a doctor. For example, it can also include the case where the processing system 1 of the present disclosure is used by the user himself or herself, or by an operator other than a doctor, and the processing device 100 included in the processing system 1 performs the determination, judgment, or diagnosis.

[0017] In addition, in the present disclosure, the user who is the subject of image capture by the image capture device 200 may include any person, such as a patient, a test subject, a diagnostic user, or a healthy individual. In addition, in the present disclosure, the operator who holds the image capture device 200 and performs image capture operations is not limited to medical professionals such as doctors, nurses, and laboratory technicians, but may also include any person, such as the user. The processing system 1 according to the present disclosure is typically expected to be used in a medical institution. However, this is not limited to this case, and the location of use may be any, such as the user's home, school, or workplace.

[0018] In the present disclosure, as described above, examples of the subject include at least a portion of the user's oral cavity. The disease to be diagnosed may also be any disease that manifests in the oral cavity. However, the following description will be given of a case in which the subject includes the pharynx or the area around the pharynx, and the disease is influenza.

[0019] Furthermore, in the present disclosure, the subject image and the assessment image may be one or more videos or one or more still images. As an example of operation, when the power button is pressed, a through image is captured by the camera, and the captured through image is displayed on the display 203. Then, when the operator presses the capture button, one or more still images are captured by the camera, and the captured image is displayed on the display 203. Alternatively, when the user presses the capture button, video capture begins, and images captured by the camera during that time are displayed on the display 203. Then, when the capture button is pressed again, video capture ends. Thus, in this series of operations, various images, such as through images, still images, and videos, are captured by the camera and displayed on the display. However, the subject image does not refer to only a specific image among these, but may include all images captured by the camera. Furthermore, the assessment image simply refers to an image used to determine the possibility of a specific disease, and does not necessarily have to undergo the specific processing described below. For example, the subject image captured by the imaging device 200 can be used as the assessment image as is.

[0020] FIG. 1 is a diagram illustrating a state in which a processing system 1 according to an embodiment of the present disclosure is in use. According to FIG. 1 , an operator attaches an auxiliary tool 500 to the tip of an imaging device 200 so as to cover the tip, and inserts the imaging device 200 together with the auxiliary tool 500 into the user's oral cavity 710. Specifically, first, an operator (which may be the user 700 or may be a different person from the user 700) attaches the auxiliary tool 500 to the tip of the imaging device 200 so as to cover the tip. Then, the operator inserts the imaging device 200 with the auxiliary tool 500 attached into the oral cavity 710. At this time, the tip of the auxiliary tool 500 passes through the incisors 711 and is inserted to the vicinity of the soft palate 713. That is, the imaging device 200 is similarly inserted to the vicinity of the soft palate 713. At this time, the auxiliary tool 500 (which functions as a tongue depressor) pushes the tongue 714 downward, restricting the movement of the tongue 714. This allows the operator to ensure a good field of view of the imaging device 200 and to take a good image of the pharynx 715 located in front of the imaging device 200.

[0021] The captured subject image (typically, an image including the pharynx 715) is transmitted from the imaging device 200 to the processing device 100, which is communicatively connected via a wired or wireless network. The processor of the processing device 100, which has received the subject image, processes a program stored in its memory, thereby selecting a determination image to be used for determination from the subject image and determining the possibility of the subject having a predetermined disease. The result is then transmitted to the terminal device 300 and output to a display or the like via the output interface of the terminal device 300.

[0022] FIG. 2 is a diagram illustrating a usage state of the imaging device 200 according to an embodiment of the present disclosure. Specifically, FIG. 2 is a diagram illustrating a state in which the imaging device 200 is held by an operator 600. As illustrated in FIG. 2 , the imaging device 200 is composed of a main body 201, a grip 202, and a display 203, from the side inserted into the oral cavity. The main body 201 and the grip 202 are formed in a substantially columnar shape of a predetermined length along the insertion direction H into the oral cavity. The display 203 is disposed on the opposite side of the grip 202 from the main body 201 side. Therefore, the imaging device 200 is formed in a substantially columnar shape as a whole, and is held by the operator 600 by holding it in a manner similar to holding a pencil. In other words, since the display panel of the display 203 faces the operator 600 in usage, the operator 600 can easily handle the imaging device 200 while checking the subject image captured by the imaging device 200 in real time.

[0023] Furthermore, when the operator 600 holds the grip 202 in an orientation in which the subject image is displayed in the normal orientation on the display 203, the photographing button 220 is configured to be located on the upper surface of the grip, so that when the operator 600 holds the grip, the operator 600 can easily press the photographing button 220 with the index finger or the like.

[0024] 2. Configuration of Processing System 1 Fig. 3 is a schematic diagram of a processing system 1 according to an embodiment of the present disclosure. According to Fig. 3, the processing system 1 includes a processing device 100, an imaging device 200 communicatively connected to the processing device 100 via a wired or wireless network, and a terminal device 300 communicatively connected to the processing device 100 via a wired or wireless network. The processing device 100 receives and processes subject images captured by the imaging device 200 based on an operation input by an operator accepted by the imaging device 200 or the terminal device 300. The processing device 100 also determines the possibility of the subject having a predetermined disease based on the received subject images, medical interview information, and findings information, and transmits the result to the terminal device 300.

[0025] The tip of the imaging device 200 is inserted into the oral cavity of a user to capture images of the oral cavity, particularly the pharynx. Specific imaging processing will be described later. The captured subject image is transmitted to the processing device 100 via a wired or wireless network.

[0026] The terminal device 300 inputs subject information, interview information, diagnostic information, etc. required for processing in the processing device 100, and receives and outputs related information related to the imaging device 200 and the results of determining the possibility of contracting a specified disease from the processing device 100.

[0027] The processing system 1 may further include a mounting table 400 as necessary. The mounting table 400 is capable of stably mounting the imaging device 200. The mounting table 400 may be connected to a power source via a wired cable, thereby supplying power to the imaging device 200 from a power supply terminal of the mounting table 400 through a power supply port of the imaging device 200.

[0028] FIG. 4 is a block diagram showing the configuration of a processing device 100 according to an embodiment of the present disclosure. According to FIG. 4, the processing device 100 includes a memory 112, a processor 111, and a communication interface 113. These components are electrically connected to one another via control lines and data lines. It is not necessary for the processing device 100 to include all of the components shown in FIG. 4; some components may be omitted, or other components may be added. A typical example of such a processing device 100 is a server device. The processing device 100 may also be connected to another server device to form an integrated processing device 100. The processing device 100 may also be connected to another database device to form an integrated processing device 100.

[0029] First, the processor 111 of the processing device 100 functions as a control unit that controls other components of the processing system 1 based on a program stored in the memory 112. Based on the program stored in the memory 112, the processor 111 stores the subject image received from the imaging device 200 in the memory 112 and processes the stored subject image. Specifically, the processor 111 executes the following processes based on the program stored in the memory 112: "acquiring one or more determination images of the subject via the camera 211 for capturing an image of the subject of the user," "acquiring at least one of the user's medical interview information and attribute information," "determining the possibility of a predetermined disease based on a trained determination model for determining the possibility of the disease and at least one of the determination image, the medical interview information, and the attribute information," and "determining the reliability of the possibility of the disease based on at least one of the determination image, the medical interview information, and the attribute information." The processor 111 is mainly composed of one or more CPUs, but may also be combined with a GPU, an FPGA, or the like as appropriate.

[0030] The memory 112 is composed of RAM, ROM, non-volatile memory, HDD, etc., and functions as a storage unit. The memory 112 stores instructions and commands for various control operations of the processing system 1 according to this embodiment as programs. Specifically, the memory 112 stores programs executed by the processor 111, such as "a process of acquiring one or more determination images of a subject via the camera 211 for capturing an image of the subject of the user," "a process of acquiring at least one of the user's medical interview information and attribute information," "a process of determining the possibility of a predetermined disease based on a trained determination model for determining the possibility of the disease and at least one of the determination image, medical interview information, and attribute information," and "a process of determining the reliability of the possibility of the disease based on at least one of the determination image, medical interview information, and attribute information." In addition to the programs, the memory 112 also stores an image management table for managing subject images captured by the camera 211 of the imaging device 200 and the images, a user table for storing the user's attribute information, medical interview information, and determination results, and distribution information of learning data.

[0031] The memory 112 also stores various trained models, such as a trained judgment image selection model used to select a judgment image from the subject image and a trained judgment model for determining the possibility of disease from the judgment image. In particular, in this embodiment, a first trained judgment model (hereinafter also referred to as a trained positive rate judgment model) that determines a first possibility of the disease (first positive rate), a second trained judgment model (hereinafter also referred to as a trained positive rate judgment model) that determines a second possibility of the disease (second positive rate), and a third trained judgment model that determines a third possibility of the disease (third positive rate) are stored. Note that the types and quantities of the trained judgment models are not limited to those described above and can be adjusted as appropriate as long as they can determine the possibility of the disease.

[0032] The communication interface 113 functions as a communication unit for transmitting and receiving information to and from the terminal device 300, the image capturing device 200, and / or other devices. Examples of the communication interface 113 include a wired communication connector such as a USB or SCSI, a wireless communication transmitting and receiving device such as a wireless LAN, Bluetooth (registered trademark), infrared, or LTE, and various connection terminals for printed circuit boards or flexible circuit boards.

[0033] FIG. 5 is a block diagram showing the configurations of an image capturing device 200 and a terminal device 300 according to an embodiment of the present disclosure. According to FIG. 5 , the image capturing device 200 includes a camera 211, a light source 212, a processor 213, a memory 214, a display panel 215, an input interface 210, and a communication interface 216. The terminal device 300 includes a processor 311, a memory 312, an input interface 313, an output interface 314, and a communication interface 315. These components are electrically connected to each other via control lines and data lines. The processing system 1 does not need to include all of the components shown in FIG. 5 ; some components may be omitted, or other components may be added. For example, the processing system 1 may include a battery for driving each component. As described with reference to FIG. 3 , the image capturing device 200 and the terminal device 300 only need to be connected to each other via a wired or wireless network so as to be able to communicate with each other; they do not need to be configured to be able to communicate directly with each other.

[0034] First, the processor 311 of the terminal device 300 functions as a control unit that controls the other components of the processing system 1 based on a processing program stored in the memory 312. Based on the processing program stored in the memory 312, the processor 311 inputs user information, medical interview information, findings information, etc., and outputs the determination result of the possibility of disease received from the processing device 100, etc. Specifically, the processor 111 executes the following processes based on the processing programs stored in the memory 312: "accepting input of user information related to the user by an operator or user himself / herself via the input interface 313," "transmitting the accepted user information to the processing device 100 via the communication interface 315," "accepting input of medical interview information and finding information of a subject by an operator or user via the input interface 313," "transmitting the accepted medical interview information and finding information together with the user information to the processing device 100 via the communication interface 315," "selecting a subject via the input interface 313 and transmitting a request for determining the possibility of the selected subject having a predetermined disease to the processing device 100 via the communication interface 315," and "receiving, from the processing device 100, via the communication interface 315, a determination result indicating the possibility of the user having a predetermined disease determined based on the subject image, etc., and the subject image, and outputting the determination result and the subject image together via the output interface 314." The processor 311 is mainly composed of one or more CPUs, but may be combined with a GPU, an FPGA, etc. as appropriate.

[0035] The memory 312 is composed of RAM, ROM, non-volatile memory, HDD, etc., and functions as a storage unit. The memory 312 stores instructions and commands for various controls of the processing system 1 according to this embodiment as programs. Specifically, the memory 312 stores programs to be executed by the processor 311, such as "a process of accepting input of user information related to the user by the operator or user himself / herself via the input interface 313," "a process of transmitting accepted user information to the processing device 100 via the communication interface 315," "a process of accepting input of medical interview information and finding information of the subject by the operator or user via the input interface 313," "a process of transmitting the accepted medical interview information and finding information together with user information to the processing device 100 via the communication interface 315," "a process of selecting a subject via the input interface 313 and transmitting a request for determining the possibility of the selected subject having a specified disease to the processing device 100 via the communication interface 315," and "a process of receiving, from the processing device 100, via the communication interface 315, a determination result indicating the possibility of the user having a specified disease determined based on the subject image, etc., and the subject image, and outputting the determination result and the subject image together via the output interface 314."

[0036] The input interface 313 functions as an input unit that accepts instructions input by the operator to the terminal device 300. Examples of the input interface 313 include physical key buttons such as a "confirmation button" for making various selections, a "back / cancel button" for returning to the previous screen or canceling an input confirmation operation, a cross key button for moving a pointer or the like output to the output interface 314, an on / off key for turning the power of the terminal device 300 on and off, and character input key buttons for inputting various characters. The input interface 313 may also be a touch panel that is superimposed on the display functioning as the output interface 314 and has an input coordinate system corresponding to the display coordinate system of the display. In this case, icons corresponding to the physical keys are displayed on the display, and the operator selects each icon by inputting instructions via the touch panel. The touch panel may be any type, such as a capacitive type or a resistive type, to detect user instructions. The input interface 313 does not always need to be physically provided on the terminal device 300 and may be connected as needed via a wired or wireless network.

[0037] The output interface 314 functions as an output unit for outputting information such as the determination result received from the processing device 100. An example of the output interface 314 is a display configured with a liquid crystal panel, an organic EL display, a plasma display, or the like. However, the terminal device 300 itself does not necessarily need to be equipped with a display. For example, an interface for connecting to a display or the like connectable to the terminal device 300 via a wired or wireless network can also function as the output interface 314 for outputting display data to the display or the like.

[0038] The communication interface 315 functions as a communication unit for transmitting and receiving subject information, medical interview information, subject images, finding information, etc. to and from the processing device 100 connected via a wired or wireless network. Examples of the communication interface 315 include various types, such as a wired communication connector such as USB or SCSI, a wireless communication transmitting / receiving device such as a wireless LAN, Bluetooth (registered trademark), infrared, or LTE, and various connection terminals for printed circuit boards or flexible circuit boards.

[0039] Next, the camera 211 of the imaging device 200 functions as an imaging unit that detects light reflected from the oral cavity, which is the subject, and generates a subject image. To detect the light, the camera 211 includes, for example, a CMOS image sensor, a lens system, and a drive system for achieving the desired functions. The image sensor is not limited to a CMOS image sensor, and other sensors, such as a CCD image sensor, can also be used. Although not specifically shown, the camera 211 may have an autofocus function, and is preferably set, for example, so that the focus is set on a specific area, such as the front of the lens. Furthermore, the camera 211 may have a zoom function and is preferably set to capture an image at an appropriate magnification depending on the size of the pharynx or influenza follicles.

[0040] The sensors constituting the camera 211 are subjected to predetermined calibration before shipping. Specifically, the calibration method involves attaching the imaging device to a calibration jig incorporating a color chart for calibration imaging, and performing predetermined imaging. The average pixel value of each color panel of the color chart in the captured image is then measured. Color correction is performed so that the average pixel value matches a reference value. Here, the reference value may be the average value of the camera used during training of the trained judgment model or performance evaluation (clinical trial). The color correction may also be performed by algorithmic processing. More specifically, image quality correction parameters that minimize the mean squared error of the average pixel values ​​of each color panel may be calculated by optimization. Information related to these calibrations may then be stored in the memory 214. By performing such calibration, changes in image color and brightness due to manufacturing variations in the sensor, etc., can be corrected, enabling inspections to be performed with the same image quality as that during training of the trained judgment model, etc., described below. This ensures the judgment accuracy of the trained judgment model, described below.

[0041] In addition to the above-described color chart correction, the following correction process may be added or performed instead of the color chart correction. Specifically, an imaging device is attached to a pharyngeal manifold for correction imaging, and a predetermined image is taken. After that, the output value of the trained determination model in the captured image is calculated. Next, color correction is performed so that the output value becomes a reference value. Here, the reference value may be the average value of the output values ​​of the pharyngeal manifold captured by the camera used during training of the trained determination model or during performance evaluation (clinical trial). Furthermore, the color correction may be performed by algorithmic processing. More specifically, image quality correction parameters that minimize the mean squared error of the average pixel value of each color panel and the squared error of the output value may be calculated by optimization.

[0042] The predetermined correction may be performed again depending on the timing of use of the image capturing device 200 after shipment. For example, the correction may be performed automatically every time the image capturing device 200 is used, or may be performed automatically or manually during regular inspection (calibration). This makes it possible to correct changes in the color tone and brightness of the image due to aging and deterioration of the sensor, etc., and to perform inspections with the same image quality as that during AI learning of the trained determination model, etc., described below.

[0043] The light source 212 is driven by instructions from the processor 213 and functions as a light source unit for irradiating light into the oral cavity. The light source 212 includes one or more light sources. In this embodiment, the light source 212 is composed of one or more LEDs, and light having a predetermined frequency band is irradiated from each LED toward the oral cavity. The light source 212 uses light having a desired band selected from the ultraviolet light band, the visible light band, and the infrared light band, or a combination thereof. Note that when the processing device 100 determines the possibility of influenza, it is preferable to use white light in the visible light band.

[0044] The light source 212 is subjected to a predetermined correction before shipping. Specifically, the imaging device 200 is placed relative to an integrating sphere, light emitted from the light guide tube reaches an illuminance meter, and the total luminous flux (brightness) of the light source 212 is measured. The current value of the light source 212 is then adjusted so that the brightness value becomes a reference value. Here, the reference value may be the average value of the camera used during training of the trained judgment model or performance evaluation (clinical trial). This adjustment may be performed by algorithmic processing, or may be performed to obtain a predetermined value calculated from the relationship between brightness and current value. Information related to these corrections may be stored in the memory 214. This correction allows for correction of changes in image color and brightness due to manufacturing variations in LEDs, etc., making it possible to perform an inspection with the same image quality as that during training of the trained judgment model, etc., described below. This ensures the judgment accuracy of the trained judgment model, described below.

[0045] The predetermined correction may be performed again depending on the timing of use of the imaging device 200 after shipment. For example, the correction may be performed automatically every time the imaging device 200 is used, or may be performed automatically or manually during periodic inspection (calibration). This makes it possible to correct changes in the color tone and brightness of the image due to aging and deterioration of the LEDs, etc., and enables inspection to be performed with the same image quality as the image quality at the time of learning the trained determination model, etc., described later.

[0046] The processor 213 functions as a control unit that controls other components of the imaging device 200 based on a program stored in the memory 214. The processor 213 controls the driving of the camera 211 and the driving of the light source 212 based on the program stored in the memory 214, and also controls the storage of the subject image captured by the camera 211 in the memory 214. The processor 213 also controls the output of the subject image and user information stored in the memory 214 to the display 203 and the transmission to the processing device 100. The processor 213 also controls the output of information related to the correction of the camera 211 and the light source 212 stored in the memory 214 to the display 203 and the transmission to the processing device 100. The processor 213 is mainly composed of one or more CPUs, but may be combined with other processors as appropriate.

[0047] The memory 214 is composed of RAM, ROM, non-volatile memory, HDD, etc., and functions as a storage unit. The memory 214 stores instructions and commands for various controls of the image capturing device 200 as programs. In addition to the programs, the memory 214 also stores subject images captured by the camera 211, various pieces of user information, etc. The memory 214 also stores information related to the correction of the camera 211 and the light source 212 described above.

[0048] The display panel 215 is provided on the display 203 and functions as a display unit for displaying the subject image captured by the imaging device 200. The display panel 215 is configured by a liquid crystal panel, but is not limited to a liquid crystal panel and may be configured by an organic EL display, a plasma display, or the like.

[0049] The input interface 210 functions as an input unit that accepts user input instructions for the processing device 100 and the image capture device 200. Examples of the input interface 210 include a "recording button" for instructing the image capture device 200 to start or stop recording, a "power button" for turning the image capture device 200 on or off, a "confirmation button" for making various selections, a "back / cancel button" for returning to the previous screen or canceling an input confirmation operation, and physical key buttons such as a cross key button for moving icons displayed on the display panel 215. Note that these various buttons and keys may be physically provided, or may be displayed as icons on the display panel 215 and be selectable using a touch panel or the like superimposed on the display panel 215 and arranged as the input interface 210. The method for detecting user input instructions via the touch panel may be any method, such as a capacitive type or a resistive type.

[0050] The communication interface 216 functions as a communication unit for transmitting and receiving information to and from the image capturing device 200 and / or other devices. Examples of the communication interface 216 include a wired communication connector such as a USB or SCSI, a wireless communication transmitting and receiving device such as a wireless LAN, Bluetooth (registered trademark), infrared, or LTE, and various connection terminals for printed circuit boards or flexible circuit boards.

[0051] 6A is a diagram conceptually illustrating an image management table stored in the processing device 100 according to an embodiment of the present disclosure. The information stored in the image management table is updated and stored as needed in accordance with the progress of processing by the processor 111 of the processing device 100.

[0052] According to FIG. 6A , the image management table stores, in association with user ID information, subject image information, candidate information, judgment image information, feature information, score information, and the like. "User ID information" is information unique to each user for identifying each user. User ID information is generated each time a new user is registered by the operator. "Subject image information" is information identifying a subject image captured by the operator for each user. The subject image is one or more images including a subject captured by the camera of the image capture device 200, and is received from the image capture device 200 and stored in the memory 112. "Candidate information" is information identifying a candidate image from one or more subject images for selecting a judgment image. "Judgment image information" is information for identifying a judgment image used to judge the possibility of influenza infection. Such a judgment image is selected from the candidate images identified by the candidate information based on similarity. "Feature information" is information obtained by inputting the judgment image into a feature extractor (FIG. 13 , S613) described below. For example, it is information related to the average color tone and average brightness of the judgment image. "Score information" is numerical information assigned by a learned judgment image selection model, which will be described later, during the screening process when selecting a judgment image from the subject images. For example, it is a numerical value assigned to determine whether an image is appropriate as a judgment image, and the higher the numerical value, the more appropriate the image is as a judgment image (the more likely it is to be selected). As described above, information for identifying each image is stored as subject image information, candidate information judgment image information, feature information, and score information. In this way, the information for identifying each image is typically identification information for identifying each image, but it may also be information indicating the storage location of each image or the image data of each image itself.

[0053] 6B is a diagram conceptually illustrating a user table stored in the processing device 100 according to an embodiment of the present disclosure. The information stored in the user table is updated and stored as needed in accordance with the progress of processing by the processor 111 of the processing device 100.

[0054] According to FIG. 6B , the user table stores attribute information, medical interview information, two-dimensional code information, judgment result information, first judgment information, second judgment information, third judgment information, reliability, tag information, etc., in association with user ID information. "User ID information" is information unique to each user and is used to identify each user. User ID information is generated each time a new user is registered by an operator. "Attribute information" is information input by, for example, an operator or user, etc., and is information related to the individual user, such as the user's name, gender, age, address, etc. "Medical interview information" is information input by, for example, an operator or user, etc., and is information used as a reference for diagnosis by a doctor, etc., such as the user's medical history and symptoms. Examples of such medical interview information include patient background information such as weight, allergies, and underlying diseases; body temperature, peak body temperature since onset, time elapsed since onset; heart rate; pulse rate; oxygen saturation; blood pressure; medication status; contact with other influenza patients; subjective symptoms and physical findings such as joint pain, muscle pain, headache, fatigue, loss of appetite, chills, sweating, cough, sore throat, runny nose / nasal congestion, tonsillitis, gastrointestinal symptoms, rash on the hands and feet, redness or white coating of the pharynx, swollen tonsils, history of tonsillectomy, strawberry tongue, and swollen anterior cervical lymph nodes with tenderness; influenza vaccination history; and timing of vaccination. "Two-dimensional code information" is information for identifying a recording medium on which user ID information, information for identifying the user ID, attribute information, medical interview information, and / or combinations thereof are recorded. Such a recording medium does not need to be a two-dimensional code. Instead of a two-dimensional code, various other things can be used, such as a one-dimensional barcode, other multidimensional codes, text information such as specific numbers or letters, image information, etc.

[0055] The "determination result information" is information indicating the determination result of the possibility of influenza infection based on the determination image. In particular, in this embodiment, the determination result information is the determination result obtained by the ensemble processing (S622 in FIG. 13 ), which will be described later, and is an overall result of determining the possibility of infection using the results of multiple determination processes. An example of such determination result information is the positivity rate for influenza. However, it is not limited to the positivity rate, and any information indicating a possibility, such as information specifying whether the result is positive or negative, may be used. Furthermore, the determination result does not need to be a specific numerical value, and may be in any form, such as a classification according to the level of the positivity rate or a classification indicating whether the result is positive or negative.

[0056] The "first determination information" is information indicating the determination result of the possibility of influenza based on the determination image. In particular, in this embodiment, the first determination information is the output result from the classifier (S616 in FIG. 13 ), which will be described later. The first determination information is also information used in the ensemble processing, which will be described later. That is, the first determination information is information indicating a first positivity rate (first possibility). The "second determination information" is information indicating the determination result of the possibility of influenza based on the feature amounts of the determination image, the medical interview information, and the attribute information. In particular, in this embodiment, the second determination information is the output result (S619 in FIG. 13 ) of the trained positivity rate determination model (S618 in FIG. 13 ), which will be described later. The second determination information is also information used in the ensemble processing, which will be described later. That is, the second determination information is information indicating a second positivity rate (second possibility). The "third determination information" is information indicating the determination result of the possibility of influenza based on the first positivity rate, the medical interview information, and the attribute information. In particular, in this embodiment, the third determination information is the output result (S621 in FIG. 13 ) of the learned positive rate determination model (S620 in FIG. 13 ) described later. The third determination information is also information used in the ensemble processing described later. In other words, the third determination information is information indicating the third positive rate.

[0057] "Reliability information" is information indicating the reliability of the judgment result. In particular, in this embodiment, the reliability information is the results of the first judgment and the second judgment (S813 and S815 in FIG. 17) described below. The reliability serves as a guideline for determining whether or not various pieces of information used to output the judgment result will be used in the re-learning of each trained judgment model. In other words, the reliability is an index indicating the likelihood of the judgment result.

[0058] "Tag information" is information indicating that the reliability of the judgment result is low and that the information will not be used as retraining data for the trained judgment model. That is, in the case shown in FIG. 6B, a check mark is added to the user ID "U1" as a tag, and the information linked by "U1" will not be used as the retraining data. That is, with regard to the information in the image management table shown in FIG. 6A and the information in the user table shown in FIG. 6B, the information in the top row linked by the user ID "U1" will not be used as the retraining data.

[0059] The attribute information and medical interview information do not need to be input by the user or operator each time, but may be received from, for example, an electronic medical record device or another terminal device connected via a wired or wireless network. Alternatively, they may be acquired by analyzing the subject image captured by the imaging device 200. Furthermore, although not specifically shown in Figures 6A and 6B, it is also possible to store in the memory 112 information on the current prevalence of infectious diseases that are the subject of diagnosis or diagnostic support, such as influenza, as well as information on external factors, such as the diagnosis results and morbidity status of other users regarding these infectious diseases.

[0060] Furthermore, each piece of information stored in Figures 6A and 6B does not necessarily have to be stored in the memory 112 of the processing device 100, but may be stored in a database device connected to the processing device 100 via a wired or wireless communication network, and may be read from the database device as the processing progresses.

[0061] 7A to 7C are schematic diagrams illustrating distribution information of accumulated data stored in the processing device 100 according to an embodiment of the present disclosure. Specifically, FIG. 7A to FIG. 7C illustrate a convex hull generated to encompass a plurality of accumulated data.

[0062] In Figure 7A, the horizontal and vertical axes represent the training data (accumulated data) used to train the second trained judgment model, which is the second trained judgment model (the trained positive rate judgment model of S618). Specifically, in Figure 7A, the horizontal axis represents the medical interview information of the training data, and the vertical axis represents the image features of the training data. Furthermore, each point plotted in Figure 7A is determined from the medical interview information and image features associated with one user ID. A convex hull C1 is then formed to include all of the multiple points plotted based on the medical interview information and image features. In Figure 7A, the shape of the convex hull C1 is a triangle as an example, but other shapes may also be used depending on the number of points to be plotted and the training data used for the vertical and horizontal axes.

[0063] In Figure 7B, the horizontal axis represents the input of the second trained judgment model, which is the second trained judgment model, and the vertical axis represents the output. Each point plotted in Figure 7B is determined from input / output data associated with one user ID. A convex hull C2 is then formed to include all of the plotted points based on the input / output data of the second trained judgment model. In Figure 7B, the shape of the convex hull C2 is pentagonal as an example, but it can also have other shapes depending on the number of points to be plotted and the data used for the vertical and horizontal axes.

[0064] In Figure 7C, the output (second positive rate) of the second trained judgment model, which is the second trained judgment model, is plotted on the first axis (X axis), the output (first positive rate) of the first trained judgment model (the feature extractor related to S613 and the classifier related to S615) is plotted on the second axis (Y axis), and the input (features of the medical interview information and images) of the second trained judgment model is plotted on the third axis (Z axis). In particular, in Figure 7C, the third axis represents data (e.g., numerical values) determined from the features of the medical interview information and images. Furthermore, each point plotted in Figure 7C is determined from input / output data associated with one user ID. A convex hull C3 is then formed to include all of the multiple points plotted based on the input / output data. In Figure 7C, the shape of the convex hull C3 is a triangular pyramid as an example, but other shapes may also be used depending on the number of points to be plotted and the learning data used on the vertical and horizontal axes.

[0065] The memory 112 may store not only the distribution information related to the accumulated data described above (FIGS. 7A to 7C), but also the training data of other trained models, distribution information related to input / output data, and data from performance evaluations (clinical trials). For example, in FIG. 7C, the output of the first trained judgment model may be the output of the convolutional layer (feature vector) rather than the positive rate. Furthermore, distribution information of the input (image features, average color tone, luminance, etc.) of the first trained judgment model and distribution information related to the image score, which is the output of the trained judgment image selection model, may be stored. Even in these cases, a convex hull of a predetermined shape is formed from the plotted points. In other words, in the present disclosure, the distribution information of the accumulated data includes information formed by appropriately selecting the training data and input / output data of each learning model, and convex hulls of various shapes are formed in each distribution information.

[0066] 8 is a diagram showing a processing sequence executed between the processing device 100, the imaging device 200, and the terminal device 300 according to an embodiment of the present disclosure. Specifically, Fig. 8 shows a processing sequence executed from when information related to the user of the imaging device 200, who is the subject of assessment, is input to the terminal device 300, when the imaging device 200 captures an image of the subject, until when the assessment result (possibility of having the disease) and the judgment result (reliability) are output from the processing device 100.

[0067] First, as shown in FIG. 8 , the terminal device 300 stores subject information about a subject for whom a possibility of suffering from a predetermined disease is being determined based on operator input received via the input interface 313 (S11). Here, the subject information may include attribute information about the subject. Next, the terminal device 300 stores interview information about the subject based on the operator input received via the input interface 313 (S12). The terminal device 300 also stores diagnostic information about the subject based on the operator input received via the input interface 313 (S12). The terminal device 300 then transmits the stored information, including the subject information, interview information, and diagnostic information, to the processing device 100 via the communication interface 315 as subject-related information (T11). These processes may be performed after or simultaneously with the processes in the imaging device 200, which will be described later; the order in which they are performed is not limited. Furthermore, information may be input from any of S11 to S13.

[0068] Next, as shown in FIG. 8 , the image capture device 200 activates the camera 211 and other components upon receiving an input from the operator via the input interface 210 (e.g., a power button) (S21). The image capture device 200 then selects user information for the user to be captured from the subject information for uncaptured subjects received from the processing device 100, based on the operator's instruction input received via the input interface 210 (S22). Next, the image capture device 200 determines whether the assistive device 500 is being worn, and if not, outputs a display indicating that the assistive device 500 is being worn via the display panel 215 to encourage the user to wear it (S23). Note that this display is merely an example, and other methods, such as sound, flashing light, or vibration, may also be used to encourage the user to wear the assistive device 500. When it is detected that the assistive device 500 has been worn by the image capture device 200 (S24), the image capture device 200 captures an image of the subject based on the operator's instruction input received via the input interface 210 (S25). When the photographing of the subject image is completed, the photographing device 200 stores the photographed subject image in the memory 214 in association with the user ID information, and outputs the photographed subject image on the display panel 215 of the display (S26). Then, when the photographing device 200 receives an input from the operator indicating that photographing is complete via the input interface 210, it transmits the stored subject image (T21) to the processing device 100 via the communication interface 216 in association with the user ID information.

[0069] Next, the processing device 100 receives the subject-related information and the subject image via the communication interface 113, stores them in the memory 112, and registers them in the image management table based on the user ID information. The processing device 100 selects a judgment image from the stored subject images to be used to determine the possibility of influenza infection (S31). After the judgment image is selected, the processing device 100 executes a process to determine the possibility of influenza infection using the selected judgment image and the subject-related information (S32). When a judgment result is obtained, the processing device 100 stores the obtained judgment result in association with the user ID information in the user table, and also stores other information related to the judgment result (feature information, score information, first judgment information, second judgment information, third judgment information, etc.) in the memory 112 and outputs them via the communication interface 113 (S33). Thereafter, the processing device 100 executes a process to determine the reliability of the obtained judgment result (S34). When the reliability judgment result is obtained, the processing device 100 stores it in the user table and outputs it via the communication interface 113 (S35). This completes the processing sequence.

[0070] 5. Processing flow (photography processing, etc.) executed by the photographing device 200 Fig. 9 is a diagram showing a processing flow executed by the processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 9 is a diagram showing a processing flow executed at a predetermined cycle for the processing related to S21 to S26 in Fig. 8. This processing flow is mainly performed by the processor 213 of the photographing device 200 reading and executing a program stored in the memory 214.

[0071] 9, the processor 213 determines whether or not an input from the operator has been received via the input interface 210 (e.g., a power button) (S211). At this time, if the processor 213 determines that an input from the operator has not been received, the processing flow ends.

[0072] On the other hand, when the processor 213 determines that input by the operator has been accepted, it starts up the photographing device 200 (S211). Then, when the processor 213 receives subject information of subjects whose subject images have not yet been photographed from the processing device 100 via the communication interface 216, it outputs the subject information of the subjects whose subject images have not yet been photographed as a list to the display via the display panel 215. The processor 213 accepts selection of subject information of subjects to be photographed from the list via the input interface 210 (S212).

[0073] Next, the processor 213 activates the camera 211 and determines whether the assistive device 500 is properly attached to the photographing device 200. If it is determined that the assistive device 500 is not properly attached, it outputs a display indicating that the assistive device 500 is being attached to the photographing device 200 (S213). The processor 213 then periodically determines whether the assistive device 500 is being attached while the display screen for the assistive device 500 is being output. If it determines that the assistive device 500 is properly attached, the processor 213 proceeds to the next step. After determining that the assistive device 500 is properly attached, the processor 213 receives a photographing start operation from the operator via the input interface 210 (e.g., the photographing button), and controls the camera 211 to start photographing subject images of the subject (S214). These subject images are photographed by pressing the photographing button to continuously photograph a fixed number of images (e.g., 30 images) at fixed intervals. When the processor 213 has finished photographing the subject images, it stores the photographed subject images in the memory 214 in association with the read user ID information. Then, the processor 213 outputs the stored subject image to the display panel 215 (S215).

[0074] Here, the operator can remove the photographing device 200 together with the assisting tool 500 from the oral cavity, check the subject image output on the display panel 215, and input a command to re-photograph if the desired image is not obtained. Therefore, the processor 213 determines whether or not input of a command to re-photograph has been received from the operator via the input interface 210 (S216). If input of a command to re-photograph has been received, the processor 213 displays the standby screen of S215 again, enabling photographing of the subject image.

[0075] On the other hand, if the input of the instruction to re-photograph has not been accepted, and the operator returns the photographing device 200 to the mounting base 400 and receives an instruction to end photographing from the processing device 100, the processor 213 transmits the subject image stored in the memory 214 and the user ID information associated with the subject image to the processing device 100 via the communication interface 216 (S217).

[0076] 6. Processing flow (determination processing, etc.) executed by the processing device 100 Fig. 10 is a diagram showing a processing flow executed by the processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 10 is a diagram showing a processing flow executed for the processes related to S31 to S33 in Fig. 8. This processing flow is mainly performed by the processor 111 of the processing device 100 reading and executing a program stored in the memory 112.

[0077] 10 , when the processor 111 receives a subject image and associated user ID information from the image capture device 200, it stores the image in the memory 112 and registers the image in the image management table (S311). Then, when the processor 111 receives the user ID information or corresponding attribute information (e.g., name) accepted by the terminal device 300 via the communication interface 113, it selects a user to be assessed for the possibility of influenza (S312). At this time, if multiple pieces of user ID information and associated subject images are received from the image capture device 200, it is possible to output the multiple pieces of user ID information or the corresponding attribute information to select one of the users.

[0078] When the processor 111 selects a user, the processor 111 reads attribute information associated with the user ID information of the selected user from the user table in the memory 112 (S313). Similarly, the processor 111 reads medical interview information associated with the user ID information of the user to be determined from the user table in the memory 112 (S314).

[0079] Next, the processor 111 reads out from the memory 112 the subject image associated with the user ID information of the selected user, and executes a process of selecting a determination image to be used in determining the possibility of influenza (S315: details of this selection process will be described later). Then, the processor 111 executes a process of determining the possibility of influenza based on the selected determination image (S316: details of this determination process will be described later). When a determination result is obtained by the determination process, the processor 111 stores the result in a user table in association with the user ID information, and outputs the determination result via the communication interface 113 (S317). This ends the processing flow.

[0080] Fig. 11 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 11 is a diagram showing details of the determination image selection process executed in S315 of Fig. 10. This processing flow is mainly performed by the processor 111 of the processing device 100 reading and executing a program stored in the memory 112.

[0081] 11 , the processor 111 reads out from the memory 112 the subject images associated with the user ID information of the selected user (S411). Next, the processor 111 selects, from the read out subject images, images that will be used as candidates for the judgment image (S412). This selection is performed, for example, using a trained judgment image selection model.

[0082] Here, Fig. 12 is a diagram showing a processing flow related to generation of a trained model according to an embodiment of the present disclosure. Specifically, Fig. 12 is a diagram showing a processing flow related to generation of a trained judgment image selection model used in S412 of Fig. 11. This processing flow may be executed by the processor 111 of the processing device 100, or may be executed by a processor of another processing device.

[0083] As shown in FIG. 2 , the processor executes a step of acquiring a subject image of a subject including at least a portion of the pharynx as a training subject image (S511). Next, the processor executes a processing step of assigning label information to the acquired training subject image, indicating whether the image can be used as a judgment image (S512). The processor then executes a step of storing the assigned label information in association with the training subject image (S513). Note that the label assignment process and label information storage process may involve a human pre-determining whether the training subject image is a judgment image, and the processor storing the result in association with the training subject image. Alternatively, the processor may perform a known image analysis process to determine whether the image is a judgment image, and store the result in association with the training subject image. The label information is assigned based on factors such as whether at least a portion of the oral cavity is captured, and whether the image quality is good due to factors such as camera shake, out-of-focus, and cloudiness.

[0084] Once the training subject images and the associated label information are obtained, the processor 111 executes a step of performing machine learning of a selection pattern of a judgment image using the training subject images and the associated label information (S514). For example, the machine learning is performed by providing a neural network formed by combining neurons with the sets of training subject images and the label information, and repeating learning while adjusting the parameters of each neuron so that the output of the neural network is the same as the label information. Then, a step of acquiring a trained judgment image selection model (e.g., a neural network and parameters) is executed (S515). The acquired trained judgment image selection model may be stored in the memory 112 of the processing device 100 or in another processing device connected to the processing device 100 via a wired or wireless network.

[0085] Returning to FIG. 11 , the processor 111 inputs the subject image read in S411 into the learned judgment image selection model to obtain, as an output, a candidate image that serves as a judgment image. This allows for the selection of an image that captures at least a portion of the subject oral cavity, or an image with good image quality, such as camera shake, defocus, subject motion blur, exposure, or cloudiness. Furthermore, images with good image quality can be reliably selected regardless of the operator's photographing skill. The processor 111 then registers the obtained candidate image that serves as a judgment image in an image management table. At this time, the processor 111 registers not only the image but also the feature values ​​and score of the judgment image in the image management table.

[0086] Next, the processor 111 executes a process of selecting a judgment image from the selected candidate images based on similarity (S413). Specifically, the processor 111 compares the obtained candidate images to calculate the similarity between each candidate image. The processor 111 then selects a candidate image that is determined to have low similarity to other candidate images as the judgment image. The similarity between each candidate image is calculated using a method that uses local features in each candidate image (Bag-of-Keypoints method), a method that uses Earth Mover's Distance (EMD), a method that uses Support Vector Machine (SVM), a method that uses Hamming distance, a method that uses cosine similarity, or the like.

[0087] In this way, by calculating the similarity between the obtained candidate images and selecting a candidate image that is determined to have a low similarity to other candidate images, subject images with different fields of view are selected as judgment images. This makes it possible to perform judgment processing based on more diverse information than when subject images obtained with the same field of view are used as judgment images, thereby making it possible to further improve judgment accuracy. Specifically, even if a part of the pharynx is hidden by the uvula in one judgment image, the hidden part of the pharynx is visible in another judgment image with a different field of view, so it is possible to prevent important features such as influenza follicles from being overlooked.

[0088] Next, the processor 111 registers the candidate image selected based on the similarity as a determination image in the image management table (S414).

[0089] Here, the subject image, candidate image, and judgment image may each be one or more. However, as an example, it is preferable to select a group of candidate images from a group of, for example, 5 to 30 subject images, and ultimately obtain a group of about five judgment images. This is because selecting a judgment image from a large number of subject images increases the likelihood of obtaining a better judgment image. Furthermore, using multiple judgment image groups in the judgment process described below can improve judgment accuracy compared to using only one judgment image. As another example, each time a subject image is captured, the captured subject image may be sent to the processing device 100 and then selected as a candidate image and a judgment image, or the image capture device 200 may select candidate images and a judgment image, and then end capture once a predetermined number of judgment images (e.g., about five) have been obtained. This can minimize the time required to capture subject images while maintaining the improved judgment accuracy as described above. In other words, it can reduce discomfort to the user, such as vomiting reflexes.

[0090] Fig. 13 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 13 is a diagram showing details of the process for determining the possibility of influenza, which is executed in S316 of Fig. 10. This processing flow is mainly performed by the processor 111 of the processing device 100 reading and executing a program stored in the memory 112.

[0091] According to Figure 13, the processor 111 obtains a judgment result by performing ensemble processing of the first positive rate (first possibility of infection), the second positive rate (second possibility of infection), and the third positive rate, each obtained using different methods.

[0092] First, the process of acquiring the first positive rate will be described. The processor 111 reads out from the memory 112 a determination image associated with the user ID information of the user to be determined (S611). The determination image is the image selected in S315 of FIG. 10 . The processor 111 then performs predetermined preprocessing (S612) on the read determination image. Examples of such preprocessing include filtering such as bandpass filters including high-pass filters and low-pass filters, averaging filters, Gaussian filters, Gabor filters, Canny filters, Sobel filters, Laplacian filters, median filters, and bilateral filters; vascular extraction processing using Hessian matrices; segmentation processing of specific regions (e.g., follicles) using machine learning; trimming processing of the segmented region; haze removal processing; super-resolution processing; and combinations thereof, selected according to the purpose, such as high definition, region extraction, noise removal, edge enhancement, image correction, and image conversion. In this way, by performing preprocessing, it is possible to improve the accuracy of diagnosis by extracting and emphasizing in advance areas of interest that are important in diagnosing diseases, such as follicles in influenza.

[0093] In addition to the above-described preprocessing, the processor 111 may also perform color conversion processing of each subject image based on information related to correction of the camera 211 and the light source 212 transmitted from the image capturing device 200. This makes it possible to adjust the color tone, etc., between the judgment image and the image of the training data of the trained model, thereby further improving the judgment accuracy of each trained model.

[0094] The processor 111 then provides the preprocessed determination image as an input to a feature extractor (S613) and acquires image features of the determination image as an output (S614). Furthermore, the processor 111 provides the acquired features of the determination image as an input to a classifier (S615) and acquires a first positivity rate indicating a first possibility of influenza infection as an output (S616). The feature extractor can acquire a predetermined number of features, such as the presence or absence of follicles or redness in the determination image, as vectors. For example, feature vectors are extracted from the determination image, and these are stored as features of the determination image.

[0095] Here, Fig. 14 is a diagram showing a processing flow related to generation of a trained model according to an embodiment of the present disclosure. Specifically, Fig. 14 is a diagram showing a processing flow related to generation of a trained positive rate determination and selection model including the feature extractor of S613 and the classifier of S615 in Fig. 13. This processing flow may be executed by the processor 111 of the processing device 100, or may be executed by a processor of another processing device.

[0096] 14, the processor executes a step of acquiring an image as a training judgment image by performing preprocessing similar to that of S612 of Fig. 13 on an image of a subject including at least a portion of the pharynx (S711). Next, the processor executes a processing step of assigning a correct answer label to the user who is the subject of the acquired training judgment image, the correct answer label being assigned in advance based on the results of a rapid influenza test by immunochromatography, a PCR test, a virus isolation and culture test, etc. (S712). Then, the processor executes a step of storing the assigned correct answer label information as judgment result information in association with the training judgment image (S713).

[0097] Once the training judgment images and the corresponding correct label information are obtained, the processor executes a step of performing machine learning of a positive rate judgment pattern using them (S714). For example, this machine learning is performed by providing a pair of the training judgment images and the correct label information to a feature extractor configured from a convolutional neural network and a classifier configured from a neural network, and repeatedly learning while adjusting the parameters of each neuron so that the output from the classifier is the same as the correct label information. Then, a step of acquiring a trained positive rate judgment model is executed (S715). The acquired trained positive rate judgment model may be stored in the memory 112 of the processing device 100 or in another processing device connected to the processing device 100 via a wired or wireless network.

[0098] Returning to Figure 13, the processor 111 inputs the judgment image preprocessed in S612 into the learned positive rate judgment model, thereby obtaining as output the features of the judgment image (S614) and a first positive rate (S616) indicating the first possibility of contracting influenza, and stores them in memory 112 in association with the user ID information.

[0099] Next, the process of acquiring the second positive rate will be described. The processor 111 reads from the memory 112 at least one of the medical interview information and attribute information associated with the user ID information of the user to be assessed (S617). The processor 111 also reads from the memory 112 the feature amounts of the determination image calculated in S614 and stored in the memory 112 in association with the user ID information (S614). The processor 111 then provides the read at least one of the medical interview information and attribute information and the feature amounts of the determination image as inputs to the trained positive rate assessment model (S618), and acquires as output a second positive rate indicating a second possibility of influenza infection (S619).

[0100] Here, Fig. 15 is a diagram showing a processing flow related to generation of a trained model according to an embodiment of the present disclosure. Specifically, Fig. 15 is a diagram showing a processing flow related to generation of a trained positive rate determination and selection model in S618 of Fig. 13. This processing flow may be executed by the processor 111 of the processing device 100, or may be executed by a processor of another processing device.

[0101] 15, the processor executes a step of acquiring learning features from a determination image obtained by performing preprocessing similar to that of S612 in FIG. 13 on an image of a subject including at least a portion of the pharynx (S721). The processor also executes a step of acquiring medical interview information and attribute information previously stored in association with user ID information of the user who is the subject of the determination image (S721). Next, the processor executes a processing step of assigning a correct label to the user who is the subject of the determination image, based on the results of a rapid influenza test using immunochromatography, a PCR test, a virus isolation and culture test, etc. (S722). The processor then executes a step of storing the assigned correct label information as determination result information in association with the learning features of the determination image, the medical interview information, and the attribute information (S723).

[0102] Once the learning features of the judgment image, as well as the medical interview information, attribute information, and the corresponding correct label information are obtained, the processor executes a step of performing machine learning of a positive rate judgment pattern using these (S724). As an example, this machine learning is performed by providing these sets of information to a neural network combining neurons, and repeatedly learning while adjusting the parameters of each neuron so that the output from the neural network is the same as the correct label information. Then, a step of acquiring a trained positive rate judgment model is executed (S725). The acquired trained positive rate judgment model may be stored in the memory 112 of the processing device 100 or in another processing device connected to the processing device 100 via a wired or wireless network.

[0103] Returning to Figure 13, the processor 111 inputs the features of the judgment image read out in S614 and at least one of the medical interview information and attribute information read out in S617 into the learned positivity rate judgment model, thereby obtaining as output a second positivity rate (S619) indicating a second possibility of contracting influenza, and stores it in memory 112 in association with the user ID information.

[0104] Next, the process of acquiring the third positivity rate will be described. The processor 111 reads from the memory 112 at least one of the medical interview information and attribute information associated with the user ID information of the user to be determined (S617). The processor 111 also reads from the memory 112 the first positivity rate calculated in S616 and stored in the memory 112 in association with the user ID information. The processor 111 then provides the read-out medical interview information and / or attribute information and the first positivity rate as inputs to the trained positivity rate determination model (S620), and acquires as output a third positivity rate indicating a third possibility of contracting influenza (S621).

[0105] Here, Fig. 16 is a diagram showing a processing flow related to generation of a trained model according to an embodiment of the present disclosure. Specifically, Fig. 16 is a diagram showing a processing flow related to generation of a trained positive rate determination and selection model in S620 of Fig. 13. This processing flow may be executed by the processor 111 of the processing device 100, or may be executed by a processor of another processing device.

[0106] 16 , the processor executes a step of acquiring first positivity information by inputting a judgment image obtained by performing preprocessing similar to that of S612 of FIG. 13 on an image of a subject including at least a portion of the pharynx into a trained positivity rate judgment and selection model including a feature extractor (S613 of FIG. 13 ) and a classifier (S615 of FIG. 13 ) (S731). The processor also executes a step of acquiring medical interview information and attribute information previously stored in association with user ID information of the user who is the subject of the judgment image (S731). Next, the processor executes a processing step of assigning a correct label to the user who is the subject of the judgment image based on the results of a rapid influenza test using immunochromatography, a PCR test, a virus isolation and culture test, etc. (S732). The processor then executes a step of storing the assigned correct label information as judgment result information in association with the first positivity rate information, the medical interview information, and the attribute information (S733).

[0107] Once the first positive rate information, the medical interview information, the attribute information, and the corresponding correct label information are obtained, the processor executes a step of performing machine learning of a positive rate determination pattern using the information (S734). As an example, this machine learning is performed by providing a set of information to a neural network combining neurons, and repeatedly learning while adjusting the parameters of each neuron so that the output from the neural network is the same as the correct label information. Then, a step of acquiring a trained positive rate determination model is executed (S735). The acquired trained positive rate determination model may be stored in the memory 112 of the processing device 100 or in another processing device connected to the processing device 100 via a wired or wireless network.

[0108] Returning to Figure 13, the processor 111 inputs the first positive rate information read out in S616 and at least one of the medical interview information and attribute information read out in S617 into the learned positive rate determination model, thereby obtaining as output a third positive rate (S621) indicating a third possibility of contracting influenza, and stores it in memory 112 in association with the user ID information.

[0109] Once the first positive rate, second positive rate, and third positive rate have been calculated in this manner, the processor 111 reads each positive rate from the memory 112 and performs ensemble processing (S622). As an example of the ensemble processing, the obtained first positive rate, second positive rate, and third positive rate are input to a ridge regression model, and the ensemble result of each positive rate is obtained as a determination result of the possibility of contracting influenza (S623).

[0110] The ridge regression model used in S622 is generated by machine learning using the processor 111 of the processing device 100 or a processor of another processing device. Specifically, the processor acquires the first, second, and third positive rates from the learning judgment image. The processor also assigns correct labels to the user who is the subject of the learning judgment image, based on the results of a rapid influenza test using immunochromatography, a PCR test, a virus isolation and culture test, or the like. The processor then provides the ridge regression model with a set of each positive rate and its corresponding correct label, and repeats learning while adjusting the parameters assigned to each positive rate so that the output is the same as the correct label information of the ridge regression model. This results in a ridge regression model used for ensemble processing, which is stored in the memory 112 of the processing device 100 or another processing device connected to the processing device 100 via a wired or wireless network.

[0111] Furthermore, although the use of a ridge regression model has been described as an example of ensemble processing, any method may be used, such as a process for obtaining the average value of each positive rate, a process for obtaining the maximum value, a process for obtaining the minimum value, a weighted addition process, or a process using other machine learning methods such as bagging, boosting, stacking, lasso regression, or linear regression.

[0112] The processor 111 stores the determination result thus obtained in the user table in the memory 112 in association with the user ID information (S624), thereby ending the processing flow.

[0113] 13, ensemble processing is performed on the first positive rate, the second positive rate, and the third positive rate to obtain a final determination result. However, this is not limited to this, and each positive rate may be used as the final determination result as is, or ensemble processing may be performed using any two positive rates to obtain the final determination result. Furthermore, other positive rates obtained by other methods may be further added and ensemble processing may be performed to obtain the final determination result.

[0114] As shown in FIG. 10, the obtained determination results are output via the communication interface 113, but it is also possible to output only the final determination results, or to output each positive rate together.

[0115] In this way, in this embodiment, it is possible to provide a processing device, processing program, processing method, and processing system suitable for processing images obtained by photographing the inside of the oral cavity for use in intraoral diagnosis.

[0116] 7. Processing flow (determination processing, etc.) executed by processing device 100 Fig. 17 is a diagram showing a processing flow executed by processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 17 is a diagram showing a processing flow executed for the processing related to S34 to S35 in Fig. 8. This processing flow is mainly performed by processor 111 of processing device 100 reading and executing a program stored in memory 112.

[0117] First, the processor 111 reads from the memory 112 data that will serve as material for determining the reliability of the determination result, which is the possibility of contracting a predetermined disease (S811). For example, the processor 111 reads from the memory 112 the first positive rate obtained in S616 of FIG. 13 and the second positive rate obtained in S618 as data to be used in the first determination, which will be described later. More specifically, the processor 111 reads the first positive rate stored as the first determination information stored in the user table in S32 of FIG. 8 and the second positive rate stored as the second determination information. The processor 111 also reads the interview information read in S617 of FIG. 13, the image feature amount in S614, and the distribution information of the accumulated data shown in FIG. 7A as data to be used in the second determination, which will be described later.

[0118] Next, the processor 111 performs a first determination process to determine the reliability of the determination results (positive rate) in S32 and S33 of FIG. 8 (S812). Specifically, the processor 111 compares the first positive rate, which is the output of the first trained determination model, with the second positive rate, which is the output of the second trained determination model, and calculates the difference, which is the comparison result. The processor 111 then determines the reliability based on the calculated difference. Here, the reliability may be a numerical value determined based on the calculated difference.

[0119] Next, the processor 111 determines whether the determination result (reliability) from the first determination process in S812 is lower than a predetermined standard (S813). For example, the processor 111 determines whether the reliability calculated as a numerical value is lower than a predetermined threshold value. The threshold value may be determined based on past determination results, a doctor's opinion on the past determination results, or the like. If the determination result from the first determination process is lower than the predetermined standard (S813: Yes), the determination result of the positive rate is deemed suspicious, and the second determination process and related processes are skipped.

[0120] Next, if the determination result of the first determination process is higher than the predetermined standard (S813: No), the processor 111 performs a second determination process to determine the reliability of the determination results (positive rate) in S32 and S33 of FIG. 8 (S814). Specifically, the processor 111 determines where the point determined by the read-out medical interview information and image feature amounts is located in the distribution information of the accumulated data shown in FIG. 7A. That is, as shown in FIG. 18, the point determined by the read-out medical interview information and image feature amounts is determined as point α1 or point α2. Here, FIG. 18 is a schematic diagram of the reliability determination process according to an embodiment of the present disclosure.

[0121] Next, the processor 111 determines whether the determination result (position of the point) by the second determination process in S814 satisfies a predetermined criterion (S815). The predetermined criterion is, for example, whether the point determined in S814 is located inside the convex hull C1 shown in FIG. 18 . If the determination result by the second determination process satisfies the predetermined criterion (S815: Yes), the information input to the second trained determination model is deemed appropriate, the reliability of the positive rate determination result is deemed high, and the process proceeds to S818. In FIG. 18 , if point α1 is determined as the determination result in S814, the reliability of the positive rate determination result is deemed high, and the process proceeds to S818.

[0122] Next, if the judgment result of the first judgment process is lower than the predetermined standard (S813: Yes), or if the judgment result of the second judgment process does not satisfy the predetermined standard (S815: No), the processor 111 determines that the judgment result of the positive rate is suspicious, decides not to use the information related to the judgment result for additional learning of the trained judgment model, and attaches tag information indicating that it will not be used (S816). Therefore, if the judgment result of the second judgment process satisfies the predetermined standard (S815: Yes), it determines that the information input to the second trained judgment model is appropriate, and the judgment result of the positive rate is also appropriate, and the tag information is not attached.

[0123] Next, the processor 111 outputs the determination result regarding the reliability via the communication interface 113 (S817). For example, the processor 111 determines that the determination result of the positivity rate is inappropriate and displays on the screen that the reliability of the determination result is low, thereby notifying the user, operator, doctor, etc. More specifically, the processor 111 may display an input error in the medical interview information and prompt the user, operator, doctor, etc. to check the medical interview information. Furthermore, the processor 111 may not only simply display that the reliability is low, but also display the difference in the first determination process and information related to the second determination process as additional information. Furthermore, the processor 111 may display that a retest using the same method or an additional test using another method (a rapid influenza test using immunochromatography or a PCR test) is recommended, thereby prompting the user, operator, doctor, etc. to take additional measures.

[0124] Next, the processor 111 stores the determination result of the first determination process in the memory 112, and also stores the determination result of the second determination process if it has been performed. Specifically, the processor 111 stores these determination results in the reliability information shown in FIG. 6B .

[0125] 8. Modifications In the above-described embodiment, the processor 111 performs a first determination process to determine the reliability of the determination result, and then performs a second determination process depending on the determination result of the first determination process. However, this is not limited to this. For example, the processor 111 may perform the second determination process regardless of the determination result of the first determination process. That is, the processor 111 may comprehensively determine the reliability of the determination result based on the determination results of the first determination process and the second determination process. In this case, the processor 111 may score each of the determination results of the first determination process and the second determination process, and determine the reliability depending on the calculated score.

[0126] In the above-described embodiment, the processor 111 determines whether the determination result (point position) of the second determination process is located inside the convex hull C1 shown in FIG. 18 , but this is not limited to this. For example, the process of S815 may be performed using a method such as that shown in FIG. 19 . Here, FIG. 19 is a schematic diagram of a reliability determination process according to an embodiment of the present disclosure. As shown in FIG. 19 , the distance L1 between the determined point β1 and the convex hull C1 may be calculated, and if the distance L1 is equal to or greater than a predetermined value, the reliability of the determination result may be determined to be low. In other words, even if the point β1 is outside the convex hull C1, if it can be positioned in a manner that is almost identical to the accumulated data (learning data) within the convex hull C1, the processor 111 does not determine that the reliability of the determination result is low and instead enables the use of related information as data for further learning. The calculated distance from the convex hull may be stored in the memory 112 together with the determination result of the second determination process.

[0127] Furthermore, although the processor 111 used the distribution information of the training data as shown in Figures 7A and 18 as the distribution information of the accumulated data used in the second judgment process, this is not limited to this. For example, the processor 111 may use the distribution information of the accumulated data as shown in Figures 7B or 7C instead of the distribution information of the training data as shown in Figures 7A and 18. When the distribution information of the accumulated data shown in Figure 7B is used, the processor 111 selects the input (medical interview information, attribute information) of the second trained judgment model and the output (second judgment information) of the second trained judgment model as the data to be read in S811. When the distribution information of the accumulated data shown in Figure 7C is used, the processor 111 selects the output (first judgment information) of the first trained judgment model as the data to be read in S811, in addition to the input and output of the second trained judgment model.

[0128] In the above-described embodiment and modified example, the reliability determination process is performed using the input / output data of the first trained judgment model and the second trained judgment model and the accumulated data. However, the input / output data of the third trained judgment model may also be used. In this case, the input / output data of the first, second, and third trained judgment models and the accumulated data may be appropriately selected to determine the combination used to determine the reliability. Furthermore, when training the trained judgment models, if K-fold cross-validation (e.g., 5-fold cross-validation: K = 5) is used, K trained judgment models will be created. In such a case, the reliability determination process may be performed by comparing the output data between the K models.

[0129] Furthermore, instead of the distribution information related to the accumulated data shown in Figures 7A to 7C, processor 111 may read and use the training data of other trained models and distribution information related to input / output data from memory 112. For example, processor 111 may perform the processes of S814 and S815 using distribution information of the output (feature vector) of the convolutional layer, distribution information of the input (image features, average color tone, luminance, etc.) of the first trained judgment model, and distribution information related to the image score that is the output of the trained judgment image selection model. Naturally, the predetermined criterion in S815 may differ depending on the distribution information of the accumulated data used.

[0130] The processor 111 may use a plurality of pieces of distribution information of the accumulated data in S814 and S816. For example, the processor 111 may use the two pieces of distribution information of the accumulated data shown in Figures 7A and 7B, execute the process of S814 for each piece of distribution information, and determine whether the position of a point in each piece of distribution information is within the convex hull or whether the distance from the convex hull is equal to or less than a predetermined value. In such a case, the processor 111 may determine that the reliability is low if the judgment result in any one of the judgments does not satisfy a predetermined criterion, or may determine that the reliability is low if the judgment result in multiple judgments does not satisfy the predetermined criterion.

[0131] Naturally, the processor 111 may perform either the first determination process or the second determination process and determine the reliability of the determination result. That is, the required reliability of the determination result, the load for determining the reliability, and the like may be taken into consideration, and an optimal determination process or combination of determination processes may be performed. The processor 111 may also perform the second determination process using reference data other than the accumulated data described above.

[0132] In the above embodiment and modified examples, even when the reliability of the judgment result is low, information related to the judgment result is stored in the memory 112. However, when the reliability is determined to be low, the processor 111 may process to delete the information related to the judgment result or not store it in the memory 112. Furthermore, when the reliability is determined to be low, the processor 111 may store the information related to the judgment result in a storage device such as a memory separate from the re-learning data. This prevents the information from being stored as re-learning data, making it possible to prevent a decrease in the accuracy of re-learning and reduce the amount of stored data.

[0133] In the above-described embodiment, the processor 111 merely determines whether the reliability of the assessment result is high or low. However, the processor 111 may also determine the possibility of an input error in the information used in the process of assessing the likelihood of morbidity and notify the user of the determination result. For example, the processor 111 may determine the possibility of an input error in the medical interview information. Here, an input error in the medical interview information refers to an input error in each element constituting the medical interview information (e.g., body temperature, headache, etc.), not an input error in the medical interview information as a whole. That is, the processor 111 can determine the input error in the information used in the process of assessing the likelihood of morbidity for each input element. In this case, for example, in the multidimensional space of the medical interview information, the processor 111 may define point a as representing the input medical interview information, and point b as the point with the shortest distance from point a on the distribution convex hull of the training data, and set a difference vector a-b based on points a and b. The processor 111 may then determine that a vector element (medical interview information) with a large difference is likely to be an input error. Here, the processor 111 may take into consideration the difference in units between the medical interview information (i.e., the units between the medical interview items) in the difference.

[0134] In the example of Figure 13, a case has been described in which information indicating the possibility of influenza is output using at least one of medical interview information and attribute information. However, instead of or in addition to this information, external factor information related to influenza may be used to output information indicating the possibility of influenza. Such external factor information may include assessment results made on other users, diagnosis results by doctors, and influenza epidemic information in the user's area, etc. The processor 111 acquires such external factor information from other processing devices, etc. via the communication interface 113, and provides the external factor information as input to the trained positive rate determination model, thereby making it possible to obtain a positive rate that takes the external factor information into account.

[0135] In the example of Figure 13, the medical interview information and attribute information are described as being input in advance by an operator or user, or received from an electronic medical record device or the like connected to a wired or wireless network. However, instead of or in addition to these, this information may be obtained from a captured subject image. The attribute information and medical interview information associated with the training subject image are assigned as correct answer labels to the training subject image, and a trained information estimation model is obtained by machine learning these pairs using a neural network. The processor 111 then inputs the subject image to the trained information estimation model, thereby obtaining the desired medical interview information and attribute information. Examples of such medical interview information and attribute information include gender, age, degree of pharyngeal redness, degree of tonsillar swelling, and the presence or absence of white foxing. This eliminates the need for the operator to input the medical interview information and attribute information.

[0136] The following describes, as a specific example of medical interview information, the case of obtaining feature quantities of pharyngeal follicles. As described above, follicles appearing in the pharynx are a characteristic sign of influenza and are visually confirmed by doctors during diagnosis. Therefore, a labeling process is performed on regions of interest, such as follicles, through operational input by doctors on the training subject images. Position information (shape information) of the labels in the training subject images is then acquired as training position information, and a trained region extraction model is obtained by machine learning the set of the training subject images and the labeled training position information using a neural network. The processor 111 then provides the subject images as input to the trained region extraction model, outputting position information (shape information) of the regions of interest (i.e., follicles). The processor 111 then stores the obtained position information (shape information) of the follicles as medical interview information.

[0137] Next, as another specific example of medical interview information, a case where a heart rate is obtained will be described. First, the processor 111 receives a predetermined period of shooting time as a subject image. Then, the processor 111 extracts each of the RGB color components from each frame constituting the received video to obtain the luminance of the G (green) component. The processor 111 generates a luminance waveform of the G component in the video from the luminance of the G component of each obtained frame, and estimates the heart rate from the peak value. Note that this method utilizes the fact that hemoglobin in blood absorbs green light, but it goes without saying that the heart rate may be estimated by other methods. The processor 111 then stores the estimated heart rate as medical interview information.

[0138] In the example of Figure 13, a case has been described in which the determination image read from the memory 112 is preprocessed in step S612 and then provided as input to the feature extractor. However, preprocessing is not necessarily required. For example, the processor 111 may read the determination image from the memory 112 and provide the read determination image as input to the feature extractor without preprocessing. Furthermore, even when preprocessing is performed, the processor 111 may provide both the preprocessed determination image and the non-preprocessed determination image as input to the feature extractor.

[0139] 14 to 16, the training data used were judgment images that had been preprocessed in the same manner as in S612 in Fig. 13. However, in consideration of the cases where preprocessing is not performed in Fig. 13 as described above or where both the preprocessed and unpreprocessed judgment images are used as the training data, judgment images that have not been preprocessed may be used as the training data.

[0140] The trained models described in Figures 12, 14 to 16, etc. use neural networks such as neural networks, convolutional neural networks, multi-layer Hercepton (MLP), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), GNN (Graph Neural Network), and Transformers; gradient boosting decision trees (GBDTs) such as LightGBM (Light Gradient Boosting Machine), XGBoost, and CatBoost; and learners such as ridge regression, logistic regression, support vector regression (SVR), nearest neighbor methods, decision trees, regression trees, and random forests.

[0141] In the example of FIG. 8 , the processing device 100 selects a judgment image, performs judgment processing, and outputs the judgment result, and the imaging device 200 captures a subject image. However, these various processes can be appropriately distributed and processed by the processing device 100, the imaging device 200, etc. Furthermore, in the processing system 1, a server device is given as an example of the processing device 100, but this is not the only example. Other examples of the processing device 100 include various terminal devices such as smartphones, tablets, and laptop PCs, electronic medical record devices, and devices for controlling other medical devices. Furthermore, while a tablet is typically given as an example of the terminal device 300, other examples include smartphones, laptop PCs, electronic medical record devices, and devices for controlling other medical devices.

[0142] Furthermore, when a subject image captured by the imaging device 200 is transmitted to the processing device 100 and subjected to a determination process in the processing device 100, information related to correction of the camera 211 and the light source 212 may be transmitted together with the subject image. Then, the processing device 100 may use the information related to the correction to perform a color conversion process on the subject image. This makes it possible to adjust the color tone, etc., between the determination image and the image of the training data of the trained model, thereby further improving the determination accuracy of each trained model.

[0143] 3, the subject image is captured using a substantially cylindrical imaging device 200. However, the present invention is not limited to this, and it is also possible to use, for example, a terminal device 300 as the imaging device and capture the subject image using a camera provided in the terminal device 300. In such a case, the camera is not inserted into the oral cavity near the pharynx, but is placed outside the incisors (outside the body) to capture images of the oral cavity.

[0144] These modified examples have the same configurations, processes, and procedures as the embodiment described in Figures 1 to 19, except for the points specifically described above. Therefore, detailed descriptions of these matters will be omitted. Furthermore, it is also possible to configure a system by appropriately combining or replacing the elements described in each modified example or embodiment.

[0145] The processes and procedures described herein can be realized not only by those explicitly described in the embodiments, but also by software, hardware, or a combination thereof. Specifically, the processes and procedures described herein can be realized by implementing logic corresponding to the processes in a medium such as an integrated circuit, volatile memory, non-volatile memory, magnetic disk, or optical storage. Furthermore, the processes and procedures described herein can be implemented as computer programs and executed by various computers, including processing devices and server devices.

[0146] Although processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, multiple software, multiple components, and / or multiple modules. Furthermore, although various information described herein is described as being stored in a single memory or storage unit, such information may be stored in multiple memories within a single device or multiple memories distributed across multiple devices. Furthermore, software and hardware elements described herein may be realized by integrating them into fewer components or by decomposing them into more components.

[0147] REFERENCE SIGNS LIST 1 Processing system 100 Processing device 200 Imaging device 300 Terminal device 400 Placement table 500 Auxiliary tool 600 Operator 700 User

Claims

1. A processing device including at least one processor, the at least one processor being configured to perform the following processing: acquiring one or more judgment images of a subject via a camera for capturing an image of the subject of a user; acquiring at least one of medical interview information and attribute information of the user; determining the possibility of a user having a specified disease based on a learned judgment model for determining the possibility of the user having the specified disease and the judgment image, the medical interview information, and at least one of the attribute information; and judging the reliability of the possibility of the user having the disease based on the judgment image, the medical interview information, and at least one of the attribute information.

2. The processing device described in claim 1, wherein the trained judgment model includes: a first trained judgment model to which the judgment image is input and which judges a first possibility of infection with the specified disease; and a second trained judgment model to which at least one of the user's medical interview information and attribute information is input and which judges a second possibility of infection with the specified disease.

3. The processing device according to claim 2, wherein the at least one processor determines the reliability of the possibility of having the disease based on a comparison result between the first possibility and the second possibility.

4. The processing device described in claim 2, wherein the at least one processor determines the reliability of the possibility of the disease based on at least one of the user's medical questionnaire information and attribute information and distribution information of the learning data of the second trained judgment model.

5. The processing device according to claim 4, wherein the at least one processor determines the reliability of the possibility of the disease based on whether or not at least one of the user's medical interview information and attribute information is located inside a convex hull generated from distribution information of the learning data.

6. The processing device according to claim 4, wherein the at least one processor determines the reliability of the possibility of the disease based on the distance between at least one of the user's medical interview information and attribute information and a convex hull generated from distribution information of the learning data.

7. The processing device according to claim 4, wherein the at least one processor determines the reliability of at least one of the user's medical questionnaire information and attribute information based on a distance between the at least one of the user's medical questionnaire information and attribute information and a convex hull generated from distribution information of the learning data.

8. The processing device according to claim 2, wherein the at least one processor determines the reliability of the possibility of the disease based on a convex hull generated from the first possibility and the second possibility and at least one of the user's medical interview information and attribute information.

9. The processing device according to claim 1, wherein the at least one processor determines the reliability of the possibility of the morbidity based on distribution information of accumulated data relating to at least one of the judgment image, the medical interview information, the attribute information, and the possibility of the morbidity.

10. The processing device described in claim 9, wherein the at least one processor inputs the one or more judgment images into a feature extractor for extracting predetermined features from the one or more judgment images, calculates the predetermined features, includes the features in distribution information of the accumulated data, and judges the reliability of the possibility of the disease.

11. The processing device described in claim 9, wherein the at least one processor inputs the one or more subject images into a trained judgment image selection model for selecting the one or more judgment images from the one or more subject images captured by the camera, includes the output result of the trained judgment image selection model in distribution information of the accumulated data, and determines the reliability of the possibility of disease.

12. The processing device according to claim 9, wherein the at least one processor includes an output of a convolutional layer of the trained judgment model in distribution information of the accumulated data and determines the reliability of the possibility of disease.

13. The processing device described in claim 1, wherein the at least one processor determines whether or not to use information regarding the possibility of being affected by the specified disease for additional learning of the trained judgment model based on the result of judging the reliability of the possibility of being affected.

14. The processing device according to claim 13, wherein the at least one processor attaches tag information to information relating to the possibility of suffering from the specified disease that has been determined not to be used for further learning of the trained judgment model.

15. The processing device according to claim 1, wherein the at least one processor adds reliability information relating to the reliability of the possibility of the disease to the information relating to the possibility of the disease for the specified disease and stores the information in the memory.

16. A processing program that, when executed by at least one processor, causes the at least one processor to function as follows: acquire one or more judgment images of the subject via a camera for capturing images of the subject of a user; acquire at least one of medical interview information and attribute information of the user; judge the possibility of the subject having a specified disease based on a learned judgment model for judging the possibility of the subject having a specified disease and at least one of the judgment image, the medical interview information, and the attribute information; and determine the reliability of the possibility of the subject having a specified disease based on at least one of the judgment image, the medical interview information, and the attribute information.

17. A processing method executed by at least one processor, comprising: a step of acquiring one or more judgment images of a subject via a camera for capturing an image of the subject of a user; a step of acquiring at least one of medical history information and attribute information of the user; a step of judging the possibility of a specified disease based on a learned judgment model for judging the possibility of the disease and the judgment image, the medical history information, and at least one of the attribute information; and a step of determining the reliability of the possibility of the disease based on the judgment image, the medical history information, and at least one of the attribute information.

18. A processing system comprising: an imaging device equipped with a camera for capturing an image of a subject of the user; and a processing device according to any one of claims 1 to 14, connected to the imaging device via a wired or wireless network.