Medical image creation device

Abstract

To create an avatar image of a medical examinee on the basis of a determination and a diagnosis carried out by a doctor.SOLUTION: A medical image creation device according to an embodiment comprises a detection unit, a creation unit, and a correction unit. The detection unit detects feature information indicating the feature of the face of a medical examinee from a picked-up image including the face. The creation unit creates an avatar image of the medical examinee from the feature information. The correction unit corrects the avatar image on the basis of disease information of the medical examinee.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a medical image generating apparatus. [Background technology]

[0002] "Online medical care" has become widespread, where patients visiting a hospital or clinic can connect to a network using their smartphone or PC (Personal Computer) and make hospital (clinic) reservations, undergo medical interviews, receive prescriptions or medicines, and make payments all over the Internet.

[0003] Recently, a technology called "metaverse" that allows people to act through avatars in a virtual space has been evolving, and it is becoming possible to lead a social life in the metaverse. Along with the development of metaverse technology, a healthcare community service called "metaverse clinic" has been developed where people can participate as anonymous avatars in the metaverse and receive medical consultations, counseling, and roundtable discussions from medically qualified people such as doctors. In light of the above, it can be expected that there will be even more opportunities to visit hospitals online (in the metaverse) in the future.

[0004] For example, when a patient has a difficult disease diagnosis, when multiple diseases are suspected, or when detailed patient information is to be stored regardless of the above, it is desirable to generate an avatar that retains as many facial features as possible. Therefore, it is necessary to generate a mask by selecting the features that should be retained. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-42812 A Summary of the Invention [Problem to be solved by the invention]

[0006] One of the problems to be solved by the embodiments disclosed in this specification is to provide a medical image generating device capable of generating an avatar image of a patient based on a doctor's judgment or diagnosis. However, the problems to be solved by the embodiments disclosed in this specification and drawings are not limited to the above problem. Problems corresponding to each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0007] A medical image generating apparatus according to an embodiment includes a detection unit, a generation unit, and a correction unit. The detection unit detects feature information indicating facial features from a captured image including the face of a patient. The generation unit generates an avatar image of the patient from the feature information. The correction unit corrects the avatar image based on disease information of the patient. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a medical information processing system according to an embodiment. [Diagram 2] FIG. 2 is an example of a captured image acquired by the medical image generating apparatus according to the embodiment. [Diagram 3] FIG. 3 is an example of feature information detected by the medical image generating apparatus according to the embodiment. [Figure 4] FIG. 4 is an example of a mask generated by the medical image generating apparatus according to the embodiment. [Diagram 5] FIG. 5 is an example of feature information detected by the medical image generating apparatus according to the embodiment. [Figure 6] FIG. 6 is an example of a mask generated by the medical image generating apparatus according to the embodiment. [Figure 7] FIG. 7 is a block diagram showing a flow of processing performed by the medical image generating apparatus according to the embodiment. [Figure 8] FIG. 8 is a table showing an example for explaining the relationship between the number of reference detection items and the importance level corresponding to the disease information. [Figure 9]FIG. 9 is a table showing an example for explaining the relationship between the detection position and the number of detections corresponding to the number of detection items. [Figure 10] FIG. 10 is a flowchart for explaining a series of processing steps performed by the medical image generating apparatus according to this embodiment. [Figure 11] FIG. 11 is a block diagram showing an example of the configuration of a medical information processing system according to the first modification. [Figure 12] FIG. 12 is a flowchart for explaining a series of processing steps performed by the medical image generating apparatus according to the first modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a medical information processing system according to an embodiment will be described with reference to the drawings. Note that the embodiment is not limited to the following embodiment. In addition, the contents described in one embodiment are similarly applied to other embodiments in principle. For example, when a PACS (Picture Archiving and Communication System) is introduced in the medical information processing system 100, each device transmits and receives medical image data in a DICOM (Digital Imaging and Communications in Medicine) format in which supplementary information is added to the medical image data. Here, the supplementary information includes, for example, a patient ID (identifier) ​​that identifies a patient, an examination ID that identifies an examination, an ID of a My Number card, an apparatus ID that identifies each apparatus, and a series ID that identifies one imaging by each apparatus.

[0010] (Embodiment) Fig. 1 is a block diagram showing an example of the configuration of a medical information processing system according to an embodiment. As shown in Fig. 1, the medical information processing system 100 includes an information processing device 10, a medical image generating device 20, and a medical examination management terminal 30. In the medical information processing system 100, the information processing device 10, the medical image generating device 20, and the medical examination management terminal 30 can communicate with each other via a network such as the Internet.

[0011] Here, in the medical information processing system 100, the information processing device 10 is used by a patient. This patient is a person who is a target of online medical care. For example, the patient receives online medical care at a medical institution existing on the metaverse. This patient is also a user of the service provided by the medical image generating device 20. Note that in the medical information processing system 100, the number of patients can be changed arbitrarily.

[0012] The information processing device 10 is a terminal device operated by a patient. The information processing device 10 is, for example, a smartphone. Note that the information processing device 10 is not limited to a smartphone, and may be a desktop PC (Personal Computer), a notebook PC, a tablet, etc. The information processing device 10 has an input interface 11, a display 12, a camera 13, a communication control circuit 14, a memory circuit 15, and a processing circuit 16.

[0013] The input interface 11 corresponds to, for example, a mouse, a keyboard, a button, a touch panel, etc. The input interface 11 accepts various setting requests from an operator of the information processing device 10, and transfers the accepted various setting requests to the processing circuit 16 as appropriate.

[0014] The display 12 displays a GUI (Graphical User Interface) for an operator to input various setting requests using the input interface 11, and displays the processing results of the information processing device 10, etc.

[0015] The camera 13 is provided in the information processing device 10. The camera 13 is configured with a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. For example, the camera 13 captures an image of the operator's face, imports the captured image, and outputs the imported image to the memory circuitry 15.

[0016] The communication control circuit 14 controls communication with other devices via a network such as the Internet. For example, the communication control circuit 14 reads out a captured image from the storage circuit 25 in response to an instruction from an operator, and transmits the read captured image to the medical image generating device 20 via the network. Here, the communication control circuit 14 associates the captured image with a patient ID (Identifier) ​​that identifies the patient, and transmits the captured image to the medical image generating device 20. Also, for example, the communication control circuit 14 receives various information generated by the medical image generating device 20 via the network.

[0017] The memory circuitry 25 is, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk, and stores control programs and various data for performing various processes by the information processing device 10. The memory circuitry 25 also stores captured images transmitted from the camera 13.

[0018] The processing circuitry 16 controls the overall processing of the information processing device 10. Specifically, the processing circuitry 16 executes various processes based on various setting requests input by an operator via the input interface 11 and various control programs and various data read from the memory circuitry 15, and controls the processing of the camera 13, the communication control circuitry 14, etc.

[0019] Furthermore, the processing circuit 16 cooperates with the camera 13 via an imaging control function 161 to perform control so as to capture an image of the operator's face.

[0020] In the medical information processing system 100, the medical examination management terminal 30 is a terminal device installed in a medical institution such as a hospital and operated by a medical professional such as a doctor or a technician. This medical institution is also a user of the service provided by the medical image generating device 20. In the example shown in Fig. 1, only one medical institution is shown for the sake of convenience, but the number of medical institutions in the medical information processing system 100 can be changed as desired.

[0021] The medical examination management terminal 30 includes, for example, an input interface 31, a display 32, a communication control circuit 33, a memory circuit 34, and a processing circuit 35.

[0022] The input interface 31 corresponds to, for example, a mouse, a keyboard, a button, a touch panel, etc. The input interface 31 accepts various setting requests from the operator of the medical examination management terminal 30, and transfers the accepted various setting requests to the processing circuit 35 as appropriate.

[0023] The display 32 displays a GUI for the operator to input various setting requests using the input interface 31, and displays the processing results in the medical examination management terminal 30, etc.

[0024] The communication control circuit 33 controls communication with other devices via a network such as the Internet. For example, the communication control circuit 33 transfers facility information stored in the memory circuit 34 to the medical image generating device 20 via the network. Here, the communication control circuit 33 associates the facility information with a medical institution ID that identifies the medical institution, and transfers the facility information to the medical image generating device 20. Also, for example, the communication control circuit 33 receives various information generated by the medical image generating device 20 via the network.

[0025] The memory circuitry 34 is, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk, and stores control programs and various data for performing various processes by the medical examination management terminal 30. The memory circuitry 34 also stores facility information that records examination schedules for each medical image diagnostic device. This facility information includes information indicating the location of the medical institution.

[0026] The processing circuitry 35 controls the overall processing of the medical examination management terminal 30. Specifically, the processing circuitry 35 executes various processes and controls the processing of the communication control circuitry 33 based on various setting requests input by the operator via the input interface 31 and various control programs and various data read from the memory circuitry 34.

[0027] In the medical information processing system 100, the medical image generating device 20 is installed in, for example, a data center. As described later, the medical image generating device 20 acquires data from the information processing device 10 and the medical examination management terminal 30 to generate various information, and outputs the generated various information to the information processing device 10 and the medical examination management terminal 30. The medical image generating device 20 includes, for example, a communication control circuit 21, a memory circuit 22, and a processing circuit 23.

[0028] The communication control circuit 21 controls communication with other devices via a network such as the Internet. For example, the communication control circuit 21 receives images captured by the camera 13 from the information processing device 10 via the network. Also, for example, the communication control circuit 21 receives various information including facility information from the medical examination management terminal 30 via the network. Furthermore, for example, the communication control circuit 21 transmits various generated data to the information processing device 10 and the medical examination management terminal 30.

[0029] The memory circuit 22 is, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk.

[0030] The processing circuitry 23 controls the operation of the medical image generating device 20. As shown in Fig. 1, the processing circuitry 23 executes a first acquisition function 231, a detection function 232, and a generation function 233. Here, for example, each processing function executed by the first acquisition function 231, the detection function 232, and the generation function 233, which are components of the processing circuitry 103 shown in Fig. 1, is recorded in the storage circuitry 22 in the form of a program executable by a computer.

[0031] The processing circuitry 23 is a processor that realizes a function corresponding to each program by reading and executing each program from the storage circuitry 22. In other words, the processing circuitry 23 in a state in which each program has been read has each function shown in the processing circuitry 23 in Fig. 1. Note that the functions of the processing circuitry 23 are not limited to these.

[0032] The first acquisition function 231 acquires a captured image including the face of the examinee. The first acquisition function 231 is an example of a first acquisition unit. Specifically, the first acquisition function 231 acquires a captured image including the face of the examinee captured by the camera 13 and transmitted by the information processing device 10. Here, an example of the captured image is shown in FIG. 2. FIG. 2 is an example of a captured image acquired by the medical image generating device 20 according to the embodiment. The captured image 41 shown in FIG. 2 is a captured image including the face of the examinee.

[0033] The detection function 232 detects feature information indicating facial features. The detection function 232 is an example of a detection unit. Specifically, the detection function 232 detects feature information indicating facial features from a captured image including the examinee's face acquired by the first acquisition function 231. For example, the detection function 232 detects feature information indicating facial features using a trained feature information detection model. Here, an example of feature information is shown in FIG. 3. FIG. 3 is an example of feature information detected by the medical image generating apparatus 20 according to the embodiment. The feature information 42 shown in FIG. 3 is both eyebrows, both eyes, both ears, a nose including nostrils, a mouth including lips, a chin, and the like included in the examinee's face.

[0034] The generation function 233 generates an avatar image of the examinee from the feature information. The generation function 233 is an example of a generation unit. Specifically, the generation function 233 uses the feature information detected by the detection function 232 to generate an avatar image of the examinee by estimating the rotation, translation, shape, movement, etc. of the face using face reconstruction technology.

[0035] For example, the generation function 233 uses a model that reconstructs an avatar image of a patient from learned feature information. At this time, the more feature information detected, the more precise the face mesh model that can be generated, and the more accurately an avatar image of a patient that reflects the features can be generated. An example of an avatar image of a patient is shown in FIG. 4. FIG. 4 is an example of an avatar image of a patient generated by the medical image generating apparatus 20 according to the embodiment. An avatar image 43 of a patient shown in FIG. 4 is an avatar image of a patient corresponding to a captured image 41.

[0036] Furthermore, for example, as shown in FIG. 5, the detection function 232 additionally detects feature information for a part for which more detailed features are desired to be retained. Then, the generation function 233 reflects the detailed features of the part in the examinee's avatar image as shown in FIG. 6. FIG. 5 is an example of feature information detected by the medical image generation apparatus 20 according to the embodiment. Feature information 421 shown in FIG. 5 is both eyes included in the examinee's face. FIG. 6 is an example of an examinee's avatar image generated by the medical image generation apparatus 20 according to the embodiment. The examinee's avatar image 431 shown in FIG. 6 is an avatar image of the examinee corresponding to the captured image 41.

[0037] It is also possible to generate an avatar that reflects some detailed features from the beginning, as shown in Figure 5. Additional processing, such as feature information detection and regeneration, can be performed using deep learning techniques such as neural networks and mathematical models. The generated avatar image of the patient can be saved as a video or image.

[0038] In addition, since it is practically difficult to generate an avatar image of a patient that reflects all of the facial features using a single learning model, as the model becomes complex and the capacity becomes large, it is necessary to perform feature scanning by dividing the area into small parts. Here, the flow of the process of generating an avatar image of a patient will be explained using Fig. 7. Fig. 7 is a block diagram showing the flow of processing by the medical image generating device 20 according to the embodiment.

[0039] In step S1, the detection function 232 detects general facial feature information (step S1). In step S2, the generation function 233 generates an avatar image of the examinee that reflects the general facial features (step S2).

[0040] In step S3, the detection function 232 finely divides the captured image acquired by the first acquisition function 231 into parts, performs feature scanning, and detects feature information. In step S3, an example of additional scanning of the areas around the eyes, mouth, nose, cheeks, and chin is shown as the feature scanning (step S3). In step S4, the generation function 233 generates an avatar image of the examinee that reflects the fine features of the face (step S4). Then, the generation function 233 stores the generated avatar image in the memory circuitry 22. As a result, the medical image generation device 20 can generate an avatar image of the examinee that reflects all of the facial features by performing additional scanning.

[0041] Above, we have described the configuration of each device in the medical information processing system 100. Meanwhile, during online medical examinations, symptoms of illness are predicted mainly from the patient's face, so there are cases where it is necessary to record an image of the patient's face. However, from the viewpoint of privacy, storing the face image as is carries a high risk of personal identification, so the image needs to be processed before storing.

[0042] As a result, research is being conducted on a technology called “digital masking,” which erases personally identifiable features while preserving disease-related information from facial images (Reference 1).

[0043] Reference 1: Yang Y, Lyu J, Wang R, Wen Q, Zhao L, Chen W, Bi S, Meng J, Mao K, Xiao Y, Liang Y, Zeng D, Du Z, Wu Y, Cui T, Liu L, Iao WC, Li X, Cheung CY, Zhou J, Hu Y, Wei L, Lai IF, Yu X, Chen J, Wang Z, Mao Z, Ye H, Xiao W, Yang H, Huang D, Lin X, Zheng WS, Wang R, Yu-Wai-Man P, Xu F, Dai Q, Lin H. A digital mask to safeguard patient privacy. Nat Med. 2022 Sep;28(9):1883-1892. doi: 10.1038 / s41591-022-01966-1. Epub 2022 Sep 15. PMID: 36109638; PMCID: PMC9499857.

[0044] In the future, it is expected that image data will be recorded in the form of a digital mask (avatar) during online medical consultations using the technology disclosed in Reference 1. Digital masks can be generated, for example, by extracting features from various parts of the face using deep learning technology, and then digitizing the shape and movement of the face through 3D reconstruction based on the extracted features.

[0045] For example, when it is difficult to diagnose a disease, when multiple diseases are suspected, or when detailed patient information is to be stored regardless of the above, there is a desire to generate an avatar that retains as many facial features as possible. Therefore, it is necessary to generate an avatar image of the patient by selecting the features that should be retained.

[0046] Therefore, the medical image generating apparatus 20 according to this embodiment corrects the avatar image of the examinee based on the doctor's judgment and diagnosis. The processing circuitry 23 included in the medical image generating apparatus 20 further executes a first judgment function 234, a second acquisition function 235, a decision function 236, a correction function 237, and a display control function 238.

[0047] The first determination function 234 determines whether an additional scan is required for the captured image acquired by the first acquisition function 231. The first determination function 234 is an example of a first determination unit. For example, the first determination function 234 determines whether it is necessary to acquire more facial feature information of the examinee for the captured image acquired by the first acquisition function 231. The additional scan refers to detecting a predetermined number of detection items again from the captured image. Details regarding the additional scan will be described later.

[0048] The second acquisition function 235 acquires the disease information of the examinee. The second acquisition function 235 is an example of a second acquisition unit. Specifically, the second acquisition function 235 acquires the disease information of the examinee, including at least one of the doctor's judgment result, the genetic diagnosis result, and the surgical history including the surgical site. The disease information of the examinee is, for example, information linked to the examinee's ID such as My Number card linkage, information stored by each medical institution where the examinee is examined, or information such as the medical examination result that the examinee has personally. The doctor's judgment result includes disease candidate information corresponding to at least one disease candidate that appears on the examinee's face and corresponds to the examinee's disease among multiple disease candidates. Note that the disease information of the examinee is not limited to this.

[0049] A doctor may, for example, determine the disease of a patient from the following signs that appear on the patient's face. The signs may be, for example, glaucoma, kidney disease, stroke, liver disease, gout, anemia, gastrointestinal disease, etc. A sign of glaucoma is, for example, increased intraocular pressure and hardened eyelids. A sign of kidney disease is, for example, darkening of the area under the eyes (appearance of dark circles) due to decreased kidney function. A sign of stroke is a symptom called unilateral ptosis, in which only one side of the eyelid droops. A sign of liver disease is a symptom of brown spots or spider angiomas that appear on the cheek of the face. A sign of gout is a lump on the ear called a gouty tophus. A sign of anemia is a swelling of the face. A sign of gastrointestinal disease is a crack at the corner of the mouth.

[0050] The determination function 236 determines the number of detection items corresponding to the characteristic information based on the importance indicating the occurrence probability of the disease corresponding to the disease information. The determination function 236 is an example of a determination unit. The detection items are items corresponding to the disease determined by a doctor. The determination function 236 determines the number of detection items corresponding to the above-mentioned additional scan. Here, the number of detection items and the importance will be explained using FIG. 8. FIG. 8 is a table showing an example for explaining the relationship between the reference number of detection items and the importance corresponding to the disease information.

[0051] For example, the importance of the doctor's diagnosis is the probability that the patient has a particular disease. As shown in Fig. 8, when the doctor determines that the patient has a 100% probability of having stroke as disease information No. 1 (importance 100), the determination function 236 multiplies the standard number of detection items by the importance and determines the number of detection items to be additionally scanned, with only the area around the eyes, where stroke symptoms are likely to appear, as the detection items (see Table T1).

[0052] Also, as shown in FIG. 8, if the doctor determines that the patient may have both stroke and liver disease as disease information No. 2 (for example, both have importance of 50), the decision function 236 determines the number of detection items to be subjected to additional scanning by multiplying the standard number of detection items by the importance, with the eyes and mouth and cheeks being detection items of equal importance (see Table T1).

[0053] For example, the importance of the genetic test results is the probability of developing a certain disease based on the genetic test results. When the risk of liver disease is 100% (importance 100), the decision function 236 multiplies the reference number of detection items by the importance and determines the number of detection items to be additionally scanned, with the cheeks and the areas around the eyes being detection items since symptoms are likely to appear in these areas even if symptoms are not currently present.

[0054] Furthermore, for example, when following up on the healing of a surgical scar, the surgical scar area is set to 100% (importance level 100). In this case, it is possible to keep only the surgical scar as it is and make the rest an avatar image.

[0055] The detection items are, for example, the eyes, mouth, nose, head, ears, and the entire face. The eyes are, for example, the general shape of the eyes, eyelids, eyeballs, irises, pupils, eyebrows, eyelashes, conjunctiva, etc. The mouth is, for example, the general shape of the mouth, lips, gums, teeth, tongue, uvula, etc. The nose is, for example, the general shape of the nose, the root of the nose, the nasal bridge, the tip of the nose, the nasal wings, the external nostrils, etc. The head is, for example, the general shape of the head, the forehead, the hairline, etc. The entire face is, for example, the general shape of the face, wrinkles, swelling, skin (texture), etc. The texture is obtained from a texture scan separate from the scan when the mask is generated, or from a captured image of the examinee obtained from the camera 13, etc.

[0056] The detection function 232 redetects the feature information based on the detection number and detection position of the feature information, corresponding to the number of detection items determined by the determination function 236. Here, the content of redetecting the feature information will be described with reference to Fig. 9. Fig. 9 is a table showing an example for describing the relationship between the detection position and the reference detection number, corresponding to the number of detection items.

[0057] For example, the number of detections changes based on the importance level described above. The more the number of detections, the more detailed the 3D face mesh model becomes, and the more the avatar image can reflect the real features. The method for changing the number of detections is to vary the maximum number of detections. For example, as shown in FIG. 9, for disease information No. 1, when the detection position is the eyes and the detection number is 100, the detection function 232 multiplies the reference number of detections by the importance level to determine the number of detections for additional scanning (see Table T2). Also, as shown in FIG. 9, for disease information No. 2, when the detection position is the eyes and cheeks, the detection function 232 multiplies the reference number of detections by the importance level to determine the number of detections for additional scanning (see Table T2).

[0058] The detection position detected by the detection function 232 can be detected, for example, by using deep learning models such as those described in the following documents 2 and 3.

[0059] Document 2: F. Schroff, D. Kalenichenko and J. Philbin, "FaceNet: A unified embedding for face recognition and clustering", 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 815-823, doi: 10.1109 / CVPR.2015.7298682.

[0060] Document 3: H. Wang et al., "CosFace: Large Margin Cosine Loss for Deep Face Recognition", 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 5265-5274, doi: 10.1109 / CVPR.2018.00552.

[0061] The correction function 237 corrects the avatar image based on the disease information of the examinee. The correction function 237 is an example of a correction unit. Specifically, the correction function 237 corrects the avatar image generated by the generation function 233 based on the disease information of the examinee acquired by the second acquisition function 235. For example, the correction function 237 corrects the avatar image based on the feature information redetected by the detection function 232. Then, the correction function 237 stores the corrected avatar image in the memory circuitry 22. Thereby, when the image of the examinee in the online medical treatment is saved, an avatar image that is not personally identifiable can be generated, corrected, and saved in a state where necessary disease information is selected based on the doctor's judgment.

[0062] The display control function 238 performs control so as to display an avatar image. The display control function 238 is an example of a display control unit. Specifically, the display control function 238 performs control over the information processing device 10 and the medical examination management terminal 30 via the network so as to display the avatar image generated by the generation function 233. In addition, the display control function 238 performs control over the information processing device 10 and the medical examination management terminal 30 via the network so as to display the avatar image corrected by the correction function 237.

[0063] Next, an example of a procedure for generating an avatar image of a patient will be described with reference to Fig. 10. Fig. 10 is a flowchart for explaining a series of processing steps of the medical image generating apparatus 20 according to this embodiment. Note that this processing is started after a captured image of the patient is acquired.

[0064] First, the detection function 232 detects feature information indicating facial features (step S21). Next, the generation function 233 generates an avatar image of the examinee from the feature information (step S22). Next, the first determination function 234 determines whether an additional scan is required for the captured image acquired by the first acquisition function 231 (step S23). Here, if the first determination function 234 determines that an additional scan is not required (step S23: No), the processing circuit 23 proceeds to step S28. On the other hand, if the first determination function 234 determines that an additional scan is required (step S23: Yes), the processing circuit 23 proceeds to step S24.

[0065] In step S24, the second acquisition function 235 acquires the disease information of the examinee (step S24). Then, the determination function 236 determines the number of detection items corresponding to the characteristic information based on the importance indicating the occurrence probability of the disease corresponding to the disease information (step S25). Then, the detection function 232 redetects the characteristic information based on the detection number and detection position of the characteristic information corresponding to the number of detection items determined by the determination function 236 (step S26).

[0066] Next, correction function 237 corrects the avatar image based on the feature information redetected by detection function 232 (step S27). Then, correction function 237 stores the corrected avatar image in storage circuitry 22 (step S28). When step S28 ends, this process ends.

[0067] As described above, according to the embodiment, the medical image generating apparatus 20 detects feature information indicating facial features from a captured image including the patient's face, generates an avatar image of the patient from the feature information, and corrects the avatar image based on the patient's disease information.

[0068] As a result, for example, the medical image generating device 20 can generate an avatar image of a patient undergoing online medical treatment based on disease information such as a doctor's judgment or diagnosis. The avatar image of the patient is corrected based on the disease information, and therefore becomes an avatar image that is not personally identifiable. Therefore, the medical image generating device 20 can store the avatar image that is not personally identifiable in the memory circuitry 22.

[0069] The above-described embodiment can be modified as appropriate by changing a part of the configuration or function of each device. Therefore, some modified examples of the above-described embodiment will be described below as other embodiments. The following mainly describes the differences from the above-described embodiment, and the same reference numerals will be given to the points that are common to the contents already described, and detailed description will be omitted. The other embodiments described below may be implemented individually or in appropriate combination.

[0070] (Variation 1) In the above-described embodiment, the processing circuitry 23 determines the number of detection items corresponding to the feature information based on the importance indicating the occurrence probability of the disease corresponding to the disease information by the determination function 236 of the medical image generating device 20. In contrast, in the first modification, a form will be described in which the number of detection items is selected in advance by a user such as a doctor.

[0071] Fig. 11 is a block diagram showing a configuration example of a medical information processing system according to Modification 1. As shown in Fig. 11, the medical information processing system 100 according to Modification 1 includes an information processing device 10, a medical image generating device 20, and a medical examination management terminal 30, similar to the medical information processing system 100 according to the embodiment shown in Fig. 1. In addition, the processing circuitry 23 of the medical image generating device 20 according to Modification 1 executes a first acquisition function 231, a detection function 232, a generation function 233, a first determination function 234, a second acquisition function 235, a decision function 236, a correction function 237, a display control function 238, a selection function 239, and a second determination function 240.

[0072] The selection function 239 selects detection items based on a user's operation. The selection function 239 is an example of a selection unit. For example, the medical examination management terminal 30 transfers detection items inputted using the input interface 31 operated by the user via a network. Then, the communication control circuit 21 of the medical image generating device 20 receives the detection items inputted from the medical examination management terminal 30 via the network. Then, the selection function 239 selects the received detection items.

[0073] The second determination function 240 determines whether the avatar image generated by the generation function 233 has a high degree of personal recognition. The personal recognition degree is, for example, the proportion of the recognition degree that identifies the examinee. The personal recognition degree can be calculated, for example, by using a face recognition model such as those described in the following documents 4 and 5.

[0074] Document 4: F. Schroff, D. Kalenichenko and J. Philbin, "FaceNet: A unified embedding for face recognition and clustering", 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 815-823, doi: 10.1109 / CVPR.2015.7298682.

[0075] Reference 5: H. Wang et al., "CosFace: Large Margin Cosine Loss for Deep Face Recognition", 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 5265-5274, doi: 10.1109 / CVPR.2018.00552.

[0076] When a captured image and an avatar image are input as models, the degree of match between the two images is calculated. If the degree of personal recognition exceeds a predetermined threshold, the second determination function 240 estimates that the degree of personal recognition is high. Note that the degree of personal recognition may also be estimated based on a visual evaluation by the user.

[0077] Hereinafter, a description will be given of the flow of processing performed by the medical image generating apparatus 20 according to the modified example 1. Fig. 12 is a flowchart for explaining a series of processing flows of the medical image generating apparatus 20 according to the modified example. Also, a description of processing common to the processing executed by the medical image generating apparatus 20 according to the embodiment will be omitted.

[0078] In step S31, selection function 239 selects a detection item based on a user's operation (step S31). In step S32, second determination function 240 determines whether the personal recognition degree is high for the avatar image generated by generation function 233 (step S32). Here, if second determination function 240 determines that the personal recognition degree is low for the avatar image (step S32: No), processing circuit 23 proceeds to step S28. On the other hand, if second determination function 240 determines that the personal recognition degree is high for the avatar image (step S32: Yes), processing circuit 23 proceeds to step S24.

[0079] For example, the medical image generating device 20 selects a detection item based on a user's operation, and determines whether the avatar image generated corresponding to the detection item has a high degree of personal recognition. In this way, the medical image generating device 20 can generate an avatar image of a patient who will receive online medical treatment based on the detection items selected by a user such as a doctor.

[0080] (Variation 2) For example, the display control function 238 of the medical image generating device 20 performs display control to display or hide an avatar image. For example, when hiding a part of the feature information included in the avatar image, the display control function 238 displays part information indicating the part of the face of the feature information corresponding to the part that is hidden. The part information is, for example, the surgical site on the face of the examinee.

[0081] For example, if a patient has a surgical site on his / her face, the patient may receive online medical care for the treatment of the surgical site. In this case, the doctor only needs to identify the surgical site, so the doctor may display only the surgical site, which is part of the feature information included in the avatar image, and hide part of the feature information other than the surgical site. In this case, the display control function 238 displays part information indicating the facial part of the feature information corresponding to the hidden part.

[0082] (Variation 3) The importance level described above is determined from the doctor's judgment result, the genetic diagnosis result, and the surgical history including the surgical site as disease information, but is not limited thereto. For example, the left-right difference of the generated avatar image may be used as the importance level. For the left-right difference of the avatar image, for example, the difference in position from the center of the detection position of the same part on the left and right of the face is used. It is used when multiple symptoms are present but the disease is unknown, or when no particular symptoms are observed but the disease seems to be present. All the left-right differences of the selected detection items may be calculated, and the sum of the importance levels is set to 100, and each left-right difference may be used as the importance level.

[0083] According to at least one of the embodiments described above, an avatar image of a patient can be generated based on a doctor's judgment or diagnosis.

[0084] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0085] 10. Information processing device 20 Medical image generating device 23 Processing circuit 30 Medical examination management terminal 100 Medical Information Processing System 231 First Acquisition Function 232 Detection Function 233 Generation function 234 1st judgment function 235 Second Acquisition Function 236 Decision Function 237 Correction Function 238 Display Control Function 239 Selection Function 240 Second judgment function

Claims

1. a detection unit that detects feature information indicating features of a face of a patient from a captured image including the face of the patient; a generation unit that generates an avatar image of the examinee from the feature information; a correction unit that corrects the avatar image based on disease information of the examinee; A medical image generating apparatus comprising:

2. The disease information includes at least one of a doctor's diagnosis, a genetic diagnosis, and a surgical history including a surgical site. The medical image generating apparatus according to claim 1 .

3. The doctor's judgment result includes disease candidate information corresponding to at least one disease candidate that appears on the examinee's face and corresponds to the examinee's disease among a plurality of disease candidates. The medical image generating apparatus according to claim 2 .

4. a determination unit that determines the number of detection items corresponding to the characteristic information based on a degree of importance indicating an occurrence probability of a disease corresponding to the disease information, The detection unit redetects the feature information based on a detection position and a detection number of the feature information corresponding to the number of detection items, The correction unit corrects the avatar image based on the re-detected feature information. The medical image generating apparatus according to claim 2 .

5. a display control unit that displays or hides the avatar image, and when a part of the feature information included in the avatar is hidden, the display control unit displays part information indicating a facial part of the feature information corresponding to the part that is hidden. The medical image generating apparatus according to any one of claims 1 to 4.

Images