Computer program, information processing method, and information processing device

Abstract

Provided are a computer program, an information processing method, and an information processing device with which it is possible to expect accurate assessment of the presence or absence of the neuropathy in a subject on the basis of facial information acquired from a direction other than a direction of the view from the front.　A computer program according to this embodiment causes a computer to execute processing of: acquiring facial information on a subject detected by a sensor; generating facial structure information on the subject on the basis of the acquired facial information; complementing facial information on the subject that is missing in the acquired facial information on the basis of the generated facial structure information; and executing processing to estimate the presence or absence of the neuropathy of the subject on the basis of the complemented facial information. The computer program may: acquire, from a storage unit, previously generated reference facial structure information on the subject generated in advance; and generate facial structure information on the subject on the basis of the facial information detected by the sensor and the reference facial structure information stored in the storage unit.

Description

Computer program, information processing method, and information processing device

[0001] The present disclosure relates to a computer program, an information processing method, and an information processing device for estimating the presence or absence of a neurological disorder in a subject.

[0002] Patent Document 1 proposes a stroke assessment device that acquires a facial image including the face of a subject, determines whether the subject has facial nerve paralysis based on the facial image using a pre-trained model that has undergone deep learning, presents the subject with a questionnaire regarding stroke, acquires responses to the questionnaire, and assesses the possibility of the subject having a stroke based on the presence or absence of facial nerve paralysis and the responses.

[0003] Japanese Patent Application Laid-Open No. 2020-199072

[0004] In order to determine whether or not a subject has facial paralysis based on a facial image of the subject taken with a camera, as in the stroke detection device described in Patent Document 1, it is difficult to accurately determine whether or not a subject has facial paralysis unless a facial image of the subject's face taken from the front is used.

[0005] The present disclosure has been made in consideration of the above circumstances, and its purpose is to provide a computer program, an information processing method, and an information processing device that can be expected to accurately estimate the presence or absence of a neurological disorder in a subject based on facial information acquired from a direction other than frontal viewing.

[0006] A computer program according to one embodiment causes a computer to acquire facial information of a subject detected by a sensor, generate facial structure information of the subject based on the acquired facial information, complement missing facial information of the subject in the acquired facial information based on the generated facial structure information, and execute a process of estimating whether or not the subject has a neurological disorder based on the complemented facial information.

[0007] According to one embodiment, it is expected that the presence or absence of a neurological disorder in a subject can be accurately estimated based on face information acquired from a direction other than the frontal view.

[0008] FIG. 1 is a schematic diagram for explaining an overview of an information processing system according to the present embodiment. FIG. 2 is a block diagram showing the configuration of a server device according to embodiment 1. FIG. 3 is a block diagram showing the configuration of a terminal device according to the present embodiment. FIG. 4 is a flowchart for explaining the procedure of pre-processing performed in the information processing system according to embodiment 1. FIG. 5 is a schematic diagram for explaining the procedure of pre-processing performed in the information processing system according to embodiment 1. FIG. 6 is a flowchart showing the procedure of a process for estimating the presence or absence of a nerve disorder performed by the server device according to the present embodiment. FIG. 7 is a flowchart showing the procedure of a facial image complementation process performed by the server device according to embodiment 1. FIG. 8 is a schematic diagram for explaining a method for estimating the presence or absence of a nerve disorder performed by the server device according to the present embodiment. FIG. 9 is a schematic diagram showing an example of a notification screen for the estimation result of a nerve disorder. FIG. 10 is a schematic diagram for explaining the configuration of a learning model provided in the server device according to embodiment 2. FIG. 11 is a schematic diagram showing an example of shape information output by a shape estimation model. FIG. 12 is a flowchart showing the procedure of a facial image complementation process performed by the server device according to embodiment 2.

[0009] Specific examples of information processing systems according to embodiments of the present disclosure will be described below with reference to the drawings. Note that the present disclosure is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope of the claims.

[0010] <System Configuration> Fig. 1 is a schematic diagram for explaining an overview of an information processing system according to this embodiment. The information processing system according to this embodiment is configured to include a server device 1 that performs processing to estimate the presence or absence of a neurological disorder in a subject, and one or more terminal devices 3 that take facial images of the subject required for the estimation processing performed by the server device 1. In the information processing system according to this embodiment, the terminal device 3 is installed in a home where the subject lives (e.g., a living room or dining room), a nursing home, a medical facility, or the like. The terminal device 3 is equipped with a camera and periodically takes images of the subject within the home and transmits the taken images to the server device 1.

[0011] The server device 1 performs a process of estimating the presence or absence of a neurological disorder, such as a stroke (cerebrovascular disorder), cerebral infarction, or facial nerve paralysis, for the subject based on a photographed image of the subject acquired from the terminal device 3. The server device 1 according to this embodiment estimates the presence or absence of a neurological disorder based on the subject's face included in the photographed image. For example, if the server device 1 estimates that the subject has a neurological disorder, it transmits this estimation result to the terminal device 3. The terminal device 3, upon receiving this estimation result, can notify the subject or other users related to the subject (such as family members or medical professionals) of the risk of a neurological disorder by, for example, displaying a warning message on a display unit or outputting it as audio. Note that even if the server device 1 estimates that the subject does not have a neurological disorder, it may transmit this information to the terminal device 3 as an estimation result.

[0012] Furthermore, in the information processing system according to this embodiment, the terminal device 3 may estimate the presence or absence of a neurological disorder based on an image of the subject captured by a camera. If the terminal device 3 estimates the presence of a neurological disorder, the terminal device 3 may detect information about the subject using a sensor other than the camera mounted on the terminal device 3, such as a distance sensor or a microphone, and further estimate the presence or absence of a neurological disorder based on the detected information. For example, upon receiving from the server device 1 an estimate of the presence of a neurological disorder based on the camera image, the terminal device 3 may detect information using a sensor other than the camera and transmit the detected information to the server device 1. The server device 1 may further perform processing to estimate the presence or absence of a neurological disorder of the subject based on the information received from the terminal device 3. Examples of information detected about the subject include abnormalities in biological information, facial paralysis, mental abnormalities, falls, agitation or tremors, weakness, or abnormal speech. Specific symptoms of abnormal biological information include abnormalities in pulse, heart rate variability, breathing, blood oxygen concentration, or blood pressure fluctuations.

[0013] For example, when the server device 1 determines that a nerve disorder is present based on an image captured by a camera and also determines that a nerve disorder is present based on information detected by another sensor, the server device 1 finally transmits an inferred result that the subject has a nerve disorder to the terminal device 3. For example, when the server device 1 determines that a nerve disorder is present based on an image captured by a camera but determines that no nerve disorder is present based on information detected by another sensor, the server device 1 finally transmits an inferred result that the subject does not have a nerve disorder to the terminal device 3.

[0014] Note that this additional estimation of the presence or absence of a nerve disorder may be performed based on an image of the subject captured by a camera, rather than on information detected by a sensor other than the camera. In this case, the additional estimation is performed using a method different from determining the presence or absence of a nerve disorder based on the subject's face, such as determining the presence or absence of a nerve disorder based on the subject's entire body. Furthermore, the additional estimation is not necessarily performed, and the server device 1 may only estimate the presence or absence of a nerve disorder based on the subject's face captured by the camera of the terminal device 3.

[0015] Furthermore, when the information processing system according to this embodiment estimates that a subject has a neuropathy, it may conduct a diagnostic test on the subject to determine, for example, the severity of the neuropathy symptoms. For example, the server device 1 transmits information to the terminal device 3 of the subject determined to have a neuropathy for conducting a diagnostic test for neuropathy based on a determination index such as the Cincinnati Prehospital Stroke Scale (CPSS), the National Institute of Health Stroke Scale (NIHSS), or the Kurashiki Prehospital Scale (KPSS). The terminal device 3 conducts the diagnostic test based on the information received from the server device 1 and transmits information obtained from the subject to the server device 1. The server device 1 determines one or more test items included in the diagnostic test based on the information received from the terminal device 3 and transmits the results of the diagnostic test to the terminal device 3. The terminal device 3 displays the results of the diagnostic test received from the server device 1 and notifies the subject. Note that the diagnostic test is not necessarily conducted.

[0016] In the information processing system according to this embodiment, the server device 1 performs various processes to accurately perform the initial estimation process regarding the presence or absence of a neuropathy, i.e., to estimate the presence or absence of a neuropathy based on an image of the subject's face captured by the camera of the terminal device 3. The server device 1 estimates the presence or absence of a neuropathy by examining the bilateral symmetry of the subject's face based on the image of the subject's face captured by the camera of the terminal device 3. The server device 1 also compares a pre-registered image of the subject's face when normal with the image of the subject's face captured by the camera of the terminal device 3, and estimates the presence or absence of a neuropathy based on the difference.

[0017] However, the above estimation of the presence or absence of a neuropathy is premised on the premise that the subject's face captured by the camera of the terminal device 3 is a frontal view. The information processing system according to this embodiment captures the subject's face using the camera of the terminal device 3 installed in the subject's home, and the subject's face cannot necessarily be captured from the front. Therefore, in the information processing system according to this embodiment, if the image of the subject's face captured by the camera of the terminal device 3 is captured from a direction other than the front (for example, diagonally forward to the right or diagonally forward to the left, etc.) and the captured image does not capture the entire face of the subject, the server device 1 complements the missing portion (missing portion) to generate an image of the subject's face as viewed from the front, and the presence or absence of a neuropathy is estimated using the above method.

[0018] 2 is a block diagram showing the configuration of the server device 1 according to embodiment 1. The server device 1 according to this embodiment is configured to include a processing unit 11, a memory unit (storage) 12, and a communication unit (transceiver) 13. Note that in this embodiment, the processing will be described as being performed by one server device, but the processing may be performed in a distributed manner by a plurality of server devices.

[0019] The processing unit 11 is configured using an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit) or a quantum processor, a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processing unit 11 reads and executes a server program 12a stored in the storage unit 12, thereby performing various processes such as a process of estimating the presence or absence of a nerve disorder in a subject based on a photographed image acquired from the terminal device 3 and a process of conducting a diagnostic test on a subject determined to have a nerve disorder.

[0020] The storage unit 12 is configured using a large-capacity storage device such as a hard disk, etc. The storage unit 12 stores various programs executed by the processing unit 11 and various data required for the processing of the processing unit 11. In this embodiment, the storage unit 12 stores a server program 12a executed by the processing unit 11, and is provided with a reference information storage unit 12b that stores information used in the process of estimating the presence or absence of a nerve disorder.

[0021] In this embodiment, the server program (computer program, program product) 12a is provided in a form recorded on a recording medium 99 such as a memory card or an optical disc, and the server device 1 reads the server program 12a from the recording medium 99 and stores it in the storage unit 12. However, the server program 12a may also be written to the storage unit 12, for example, during the manufacturing stage of the server device 1. Alternatively, the server program 12a may be distributed by another remote server device or the like and acquired by the server device 1 via communication. For example, the server program 12a may be read from the recording medium 99 by a writing device and written to the storage unit 12 of the server device 1. The server program 12a may be provided in a form distributed via a network or in a form recorded on the recording medium 99.

[0022] The reference information storage unit 12b stores, as reference information, information about the subject's face (normal face) that has been acquired and / or generated in advance. The information stored in the reference information storage unit 12b may include, for example, a photographed image of the subject's face taken from the front, a three-dimensional model of the subject's face, and the like.

[0023] The communication unit 13 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN (Local Area Network), the Internet, etc. In this embodiment, the communication unit 13 communicates with one or more terminal devices 3 via the network N. The communication unit 13 transmits data provided by the processing unit 11 to other devices, and provides data received from other devices to the processing unit 11.

[0024] The storage unit 12 may be an external storage device connected to the server device 1. The server device 1 may be a multi-computer including multiple computers, or may be a virtual machine virtually constructed by software. The server device 1 is not limited to the above configuration, and may include, for example, a reading unit that reads information stored in a portable storage medium, an input unit that accepts operation input, or a display unit that displays images.

[0025] In addition, in the server device 1 according to this embodiment, the processing unit 11 reads and executes the server program 12a stored in the storage unit 12, whereby a facial image acquisition unit 11a, a three-dimensional model generation unit 11b, a facial image completion unit 11c, a nerve damage estimation unit 11d, a second estimation unit 11e, a diagnostic test processing unit 11f, a notification processing unit 11g, etc. are realized as software functional units in the processing unit 11. Note that in the figure, functional units related to the processing of estimating the presence or absence of a nerve damage in a subject are illustrated as functional units of the processing unit 11, and functional units related to other processing are not illustrated.

[0026] The facial image acquisition unit 11a performs a process of acquiring an image of the subject's face photographed by the terminal device 3 by communicating with the terminal device 3 via the communication unit 13. Furthermore, for example, if the subject's entire body is captured in an image photographed by the terminal device 3, the facial image acquisition unit 11a may perform a process of detecting the subject's face from the photographed image and extracting a partial image of the detected face. Note that the process of extracting the image of the subject's face from the photographed image may be performed by the terminal device 3 rather than by the server device 1, and in this case, the server device 1 can acquire the image of the subject's face extracted from the photographed image from the terminal device 3.

[0027] The three-dimensional model generation unit 11b performs processing to generate a (latest) three-dimensional model of the subject's face based on the image of the subject's face acquired by the face image acquisition unit 11a and the three-dimensional model of the subject stored in the reference information storage unit 12b.

[0028] In the information processing system according to the present embodiment, photographs and measurements are taken of the subject's face in advance, and a three-dimensional model of the subject's face is created in advance and stored in the reference information storage unit 12b of the server device 1. For example, photographs of the subject's face are taken from multiple angles, and two-dimensional images of the subject's face from various angles are collected. The surface shape of the subject's face is measured using a distance sensor that uses infrared or ultrasonic waves, and a shape model that reproduces the shape of the subject's face in a three-dimensional virtual space is generated. A three-dimensional model that serves as a reference for the subject's face is generated by pasting the collected two-dimensional images onto this shape model. The server device 1 stores the generated three-dimensional model and two-dimensional images of the subject's face taken from multiple angles when creating this three-dimensional model in the reference information storage unit 12b in advance. In the following description, the three-dimensional model of the subject's face stored in the reference information storage unit 12b is referred to as the reference three-dimensional model, and the two-dimensional images are referred to as reference face images. Instead of generating a shape model based on the measurement results of a ranging sensor, the shape model may be generated based on two-dimensional images of the subject's face captured from one or more angles. A trained face mesh learning model, for example, may be used to generate a three-dimensional shape model from a two-dimensional image. The face mesh learning model is a machine learning model that detects facial key points (feature points) from an image and can output hundreds of feature points from a human face in three-dimensional coordinates.

[0029] It is preferable that the reference 3D model of the subject's face is generated as a model that can change facial expressions, or as a model that can change facial expressions. Therefore, it is preferable that the subject's face is photographed and the surface shape is measured with the subject in various facial expressions. In this embodiment, it is also preferable that the various facial expressions include asymmetrical facial expressions.

[0030] The 3D model generation unit 11b generates a 3D model of the subject's face by pasting the image of the subject's face acquired by the facial image acquisition unit 11a onto a reference 3D model of the subject's face stored in the reference information storage unit 12b. At this time, the 3D model generation unit 11b compares, for example, the facial image to be pasted with the facial image pasted on the surface of the reference 3D model, selects the reference 3D model with the most similar expression, and pastes the facial image onto this reference 3D model. The 3D model generated in this way is a model onto which the image of the subject's face captured by the terminal device 3 is pasted, and the portion of the face missing from the image captured by the terminal device 3 is modeled using the facial image pasted on the reference 3D model.

[0031] The facial image completion unit 11c performs processing to complete missing portions of an image of the subject's face based on the three-dimensional model of the subject's face generated by the three-dimensional model generation unit 11b. The facial image completion unit 11c converts the generated three-dimensional model of the face into a two-dimensional image seen from the front, that is, generates a two-dimensional image captured by a virtual camera placed in front of the three-dimensional model of the face in a three-dimensional virtual space, thereby acquiring a facial image of the subject seen from the front in which missing portions in the image captured by the terminal device 3 are completed.

[0032] The nerve damage estimation unit 11d performs a process of estimating the presence or absence of a nerve damage in the subject based on the facial image of the subject complemented by the facial image complementing unit 11c. Note that if the facial image of the subject acquired from the terminal device 3 is a frontal view, the nerve damage estimation unit 11d may estimate the presence or absence of a nerve damage in the subject based on the image acquired from the terminal device 3 without performing the above complementing process.

[0033] In this embodiment, the nerve damage estimation unit 11d extracts various feature points, such as the positions of the eyes, mouth, forehead, and cheeks, as well as the angles of the corners of the mouth and eyebrows, from a facial image of the subject, and compares the extracted feature points between the left and right sides of the face to check for symmetry. The nerve damage estimation unit 11d calculates, for example, the amount of deviation between the left and right sides of various facial features, and can estimate the presence of a nerve damage if the calculated amount of deviation exceeds a predetermined threshold.

[0034] The neuropathy estimation unit 11d may also compare the subject's most recent facial image with a previous facial image and estimate the presence or absence of a neuropathy based on the difference between the two. To this end, the server device 1 stores, in the reference information storage unit 12b, a facial image (which may include feature information extracted from the facial image or a 3D model generated from the facial image) previously captured when the subject is in a normal state (i.e., without a neuropathy). The neuropathy estimation unit 11d extracts various features from the facial image acquired from the terminal device 3 (or a facial image obtained by complementing the facial image acquired from the terminal device 3) and compares them with the features extracted from the normal facial image stored in the reference information storage unit 12b. The neuropathy estimation unit 11d calculates the amount of deviation between the features of both facial images, and if the calculated amount of deviation exceeds a predetermined threshold, it can estimate the presence of a neuropathy.

[0035] Furthermore, in this embodiment, the neuropathy estimation unit 11d compares the latest facial image of the subject with past facial images separately for the left and right halves of the face, and if the amount of misalignment on either the left or right half exceeds a threshold, it estimates that the subject has a neuropathy. If the amount of misalignment on both the left and right halves of the face exceeds the threshold, or if the amount of misalignment on both the left and right halves of the face does not exceed the threshold, the neuropathy estimation unit 11d estimates that the subject does not have a neuropathy. Note that if the amount of misalignment on both the left and right halves of the face exceeds the threshold, the neuropathy estimation unit 11d may estimate that a neuropathy exists.

[0036] When the neuropathy estimation unit 11d estimates that the subject has a neuropathy, the second estimation unit 11e additionally estimates the presence or absence of a neuropathy in the subject using a sensor other than the camera provided in the terminal device 3. In the information processing system according to the present embodiment, the face shape of the subject is measured using a distance measurement sensor provided in the terminal device 3, the server device 1 acquires the measurement results from the terminal device 3, and the second estimation unit 11e estimates the presence or absence of a neuropathy in the subject by, for example, determining the symmetry of the face shape of the subject or comparing it with a normal shape. Note that the additional estimation by the second estimation unit 11e may be performed using any sensor provided in the terminal device 3, and may be performed by any method based on the information acquired by the sensor.

[0037] The diagnostic test processing unit 11f performs processing for conducting a diagnostic test on the subject using the terminal device 3 when the neuropathy estimation unit 11d and the second estimation unit 11e estimate that the subject has a neuropathy. In this embodiment, the diagnostic test processing unit 11f conducts a diagnostic test for stroke based on a judgment index such as the CPSS, the NIHSS, or the KPSS. In this embodiment, the diagnostic test can be conducted, for example, by displaying a question message on the display unit of the terminal device 3 and receiving a response from the subject via text input, voice input, or the like. Alternatively, the diagnostic test can be conducted by displaying a message requesting the subject to perform a predetermined action or facial expression on the display unit of the terminal device 3 and capturing an image of the subject performing the requested action or facial expression with a camera. Note that the above-described method of conducting the diagnostic test is merely an example and is not limited thereto. The diagnostic test processing unit 11f may conduct any diagnostic test.

[0038] The diagnostic test processing unit 11f transmits information such as questions or requests related to the diagnostic test to the terminal device 3. Based on this information, the terminal device 3 outputs messages such as questions or requests related to the diagnostic test, accepts answers from the subject or photographs their actions, and transmits information such as the accepted answers or photographed images to the server device 1. The diagnostic test processing unit 11f of the server device 1, which has received information related to the diagnostic test from the terminal device 3, determines the presence or absence of a neurological disorder in the subject and the extent of the disorder based on the received information.

[0039] The notification processing unit 11g performs processing for notifying the subject of the estimation result of the neuropathy estimation unit 11d, the estimation result of the second estimation unit 11e, and / or the diagnosis result of the diagnostic test processing unit 11f. The notification processing unit 11g notifies the subject by transmitting information on these estimation results or diagnosis results to the terminal device 3. The notification processing unit 11g may also notify other users related to the subject, such as family members, based on information such as a pre-registered email address or phone number.

[0040] 3 is a block diagram showing the configuration of a terminal device 3 according to this embodiment. The terminal device 3 according to this embodiment is configured to include a processing unit 31, a memory unit (storage) 32, a communication unit (transceiver) 33, a display unit (display) 34, an operation unit 35, a camera 36, ​​and a distance measurement sensor 37. The terminal device 3 is a device installed in a home or the like where a subject whose presence or absence of a neurological disorder is to be estimated lives. The terminal device 3 may be, for example, a device that is fixedly installed in a home or the like, or may be a portable device such as a smartphone or tablet terminal device mounted on a stand or the like.

[0041] The processing unit 31 is configured using an arithmetic processing device such as a CPU or an MPU, a ROM, a RAM, etc. The processing unit 31 reads and executes a program 32a stored in the storage unit 32, thereby performing various processes such as a process of photographing the subject with the camera 36, ​​a process of detecting information about the subject with the distance measurement sensor 37, and a process of inputting and outputting information for a diagnostic test between the subject and the processing unit 31.

[0042] The storage unit 32 is configured using, for example, a non-volatile memory element such as a flash memory or a storage device such as a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data required for processing by the processing unit 31. In this embodiment, the storage unit 32 stores the program 32a executed by the processing unit 31. In this embodiment, the program 32a is distributed by a remote server device or the like, and the terminal device 3 acquires it via communication and stores it in the storage unit 32. However, the program 32a may also be written to the storage unit 32, for example, during the manufacturing stage of the terminal device 3. For example, the program 32a may be recorded on a recording medium 98 such as a memory card or an optical disk, and the terminal device 3 reads the program 32a and stores it in the storage unit 32. For example, the program 32a may be recorded on the recording medium 98 and read by a writing device and written to the storage unit 32 of the terminal device 3. The program 32a may be provided in the form of distribution via a network or in the form of being recorded on the recording medium 98.

[0043] The communication unit 33 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, the Internet, etc. In this embodiment, the communication unit 33 communicates with the server device 1 via the network N. The communication unit 33 transmits data provided by the processing unit 31 to other devices, and provides data received from other devices to the processing unit 31.

[0044] The display unit 34 is configured using a liquid crystal display or the like, and displays various images, characters, etc. based on the processing of the processing unit 31. The operation unit 35 accepts user operations and notifies the processing unit 31 of the accepted operations. For example, the operation unit 35 accepts user operations using an input device such as a mechanical button or a touch panel provided on the surface of the display unit 34. Furthermore, for example, the operation unit 35 may be an input device such as a mouse and a keyboard, and these input devices may be configured to be detachable from the terminal device 3.

[0045] The camera 36 is configured using an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera 36 provides data of images (moving images) captured by the imaging element to the processing unit 31. The camera 36 may be built into the terminal device 3 or may be configured to be detachable from the terminal device 3.

[0046] The ranging sensor 37 is a sensor that measures the distance to an object by emitting, for example, infrared rays, ultrasonic waves, or electromagnetic waves and detecting the reflected waves. The ranging sensor 37 may be, for example, a sensor called LiDAR (Light Detection and Ranging). In this embodiment, the ranging sensor 37 is used for additional estimation processing that is performed when a nerve disorder is estimated based on the nerve disorder estimation processing of the subject's facial image captured by the camera 36. The ranging sensor 37 can measure the shape of the subject's face, and the server device 1 can estimate the presence or absence of a nerve disorder of the subject based on this measurement result. Note that the sensor included in the terminal device 3 for the additional estimation processing is not limited to the ranging sensor 37 and may be, for example, a sensor (microphone) that detects sound, or any other sensor.

[0047] In addition, in the terminal device 3 according to this embodiment, the processing unit 31 reads and executes the program 32a stored in the memory unit 32, thereby realizing the photography processing unit 31a, the ranging processing unit 31b, the diagnostic test processing unit 31c, etc. as software functional units in the processing unit 31.

[0048] The image capture processing unit 31a captures an image of the subject using the camera 36 and transmits the captured image to the server device 1. The image capture processing unit 31a repeatedly captures images, for example, several times to several tens of times per second. The image capture processing unit 31a continuously captures images using the camera 36 regardless of whether the subject is present in the house or not, and transmits the captured images to the server device 1. However, the image capture processing unit 31a may perform processes such as detecting people from the captured images, detecting people's faces, and identifying the people in the images, and then select images containing the subject's face and transmit them to the server device 1. The image capture processing unit 31a may also extract partial images containing the subject's face from the images captured by the camera 36 and transmit the extracted partial images to the server device 1. If the terminal device 3 is equipped with a human presence sensor, for example, the image capture processing unit 31a may capture images only when a person is present around the terminal device 3.

[0049] The distance measurement processing unit 31b performs a process of measuring the surface shape of the subject's face by measuring the distance to the subject's face using the distance measurement sensor 37. In this embodiment, measurement by the distance measurement processing unit 31b is not performed continuously, but is started when an instruction to perform additional estimation processing is given from the server device 1. The distance measurement processing unit 31b determines the position of the subject's face based on, for example, an image captured by the camera 36, ​​extracts information corresponding to the position of the subject's face from the distance information measured by the distance measurement sensor 37, and transmits the extracted information to the server device 1 as information on the surface shape of the subject's face.

[0050] The diagnostic test processing unit 31c performs a diagnostic test for a nerve disorder by outputting messages and the like to the subject and accepting input from the subject. For example, the diagnostic test processing unit 31c displays a question message provided by the server device 1 on the display unit 34, accepts an answer to the question from the subject via the operation unit 35, and transmits the accepted answer to the server device 1. The output of the question and the input of the answer may be performed via voice input and output. Furthermore, for example, the diagnostic test processing unit 31c displays a message regarding a request for exercise or facial expression provided by the server device 1 on the display unit 34, captures the exercise or facial expression performed by the subject in response to the request using the camera 36, ​​and transmits the captured image (moving image or still image) to the server device 1.

[0051] <Pre-processing> In the information processing system according to the first embodiment, facial information of a subject in a normal state (a state without neurological disorders) is acquired in advance as pre-processing. FIG. 4 is a flowchart for explaining the procedure of the pre-processing performed in the information processing system according to the first embodiment. FIG. 5 is a schematic diagram for explaining the pre-processing performed in the information processing system according to the first embodiment. Note that the pre-processing may be performed by the server device 1, the terminal device 3, or one or more other devices. In this embodiment, the pre-processing is performed using the server device 1 and the terminal device 3.

[0052] In the pre-processing, the server device 1, for example, uses the camera 36 of the terminal device 3 to capture a normal image of the face of the subject for whom the presence or absence of a nerve disorder is to be predicted (step S1). At this time, for example, the terminal device 3 is manually or automatically moved relative to the subject's face, or the subject moves their face relative to the camera 36 of the terminal device 3, thereby acquiring multiple captured images of the subject's face from multiple directions. Also, at this time, for example, a message is displayed on the display unit 34 of the terminal device 3 requesting the subject to take various facial expressions, and multiple captured images of the face with various facial expressions are acquired. The various facial expressions preferably include asymmetrical expressions. The upper part of Figure 5 shows an example of a facial image of the subject obtained by the processing of step S1.

[0053] The server device 1 also measures the three-dimensional shape of the subject's face using, for example, the distance measurement sensor 37 of the terminal device 3 (step S2). It is preferable to measure the subject's facial expressions in this measurement as well. The photographing in step S1 and the measurement in step S2 may be performed simultaneously. Based on the measurement results in step S2, the server device 1 creates a three-dimensional shape model that reproduces the subject's face in a three-dimensional virtual space (step S3). The shape model created by the server device 1 in step S3 is a model that reproduces the shape (structure) of the subject's face, and does not reproduce the color or pattern of the face's surface. An example of a shape model created in the processing of step S3 is shown in the middle of Figure 5.

[0054] Next, the server device 1 generates a three-dimensional model of the subject's face by pasting the image of the subject's face captured in step S1 onto the surface of the shape model created in step S3 (step S4). The three-dimensional model generated in step S4 is a model whose orientation can be changed in three-dimensional virtual space. Furthermore, it is preferable that multiple three-dimensional models are generated corresponding to multiple facial expressions of the subject, or that the model's facial expression can be changed in three-dimensional virtual space. The lower part of Figure 5 shows an example of a three-dimensional model generated by the processing of step S4, and multiple images corresponding to multiple orientations of the three-dimensional model are arranged horizontally.

[0055] The server device 1 stores the facial image acquired in step S1 and the three-dimensional model generated in step S4 in the reference information storage unit 12b of the storage unit 12 (step S5), and ends the pre-processing. The information stored in the reference information storage unit 12b of the server device 1 is used to complement the facial image of the subject captured by the terminal device 3 when performing processing to estimate the presence or absence of a neurological disorder in the subject.

[0056] <Estimation Process> In the information processing system according to this embodiment, the terminal device 3 installed in the subject's home continuously takes images using the camera 36 and continuously transmits the images of the subject (images of the subject's face) obtained by the images to the server device 1. The server device 1 acquires the images of the subject's face taken by the terminal device 3 and performs a process of estimating the presence or absence of a neurological disorder in the subject. Figure 6 is a flowchart showing the steps of the process of estimating the presence or absence of a neurological disorder performed by the server device 1 according to this embodiment.

[0057] The facial image acquisition unit 11a of the processing unit 11 of the server device 1 according to this embodiment communicates with the terminal device 3 via the communication unit 13 and acquires a facial image of the subject captured by the terminal device 3 using the camera 36 (step S21). The facial image acquisition unit 11a determines whether the facial image acquired in step S21 is a front-view image of the subject's face captured from the front (step S22). If the image is a front-view image (YES in S22), the processing unit 11 proceeds to step S24. If the image is not a front-view image (NO in S22), the processing unit 11 performs a facial image complementation process (step S23) and proceeds to step S24. The facial image complementation process performed in step S23 will be described later.

[0058] The neuropathy estimation unit 11d of the processing unit 11 performs a process of estimating the presence or absence of a neuropathy in the subject based on the facial image acquired from the terminal device 3 or a facial image obtained by correcting this facial image (step S24). At this time, the neuropathy estimation unit 11d compares the left and right halves of the subject's face captured in the facial image and estimates the presence or absence of a neuropathy based on left-right symmetry. The neuropathy estimation unit 11d also compares the facial image acquired from the terminal device 3 or a facial image obtained by correcting this facial image with a facial image of the subject in a normal state stored in the reference information storage unit 12b and estimates the presence or absence of a neuropathy based on the difference between the two.

[0059] The processing unit 11 determines whether the subject is estimated to have a neuropathy through the neuropathy estimation process of step S24 (step S25). If it is estimated that the subject does not have a neuropathy (S25: NO), the processing unit 11 returns to step S21 and repeats the above-described process. If it is estimated that the subject has a neuropathy (S25: YES), the second estimation unit 11e of the processing unit 11 performs a second estimation process using the distance measurement sensor 37 provided in the terminal device 3 (step S26). In the second estimation process, the second estimation unit 11e, for example, sends a command to the terminal device 3 to perform measurement using the distance measurement sensor 37, and acquires from the terminal device 3 the measurement results (face information) of the subject's facial surface shape measured by the distance measurement sensor 37. The second estimation unit 11e can estimate the presence or absence of a neuropathy in the subject using the distance measurement sensor 37, for example, as in the case of estimation processes based on facial images, by estimating the presence or absence of a neuropathy based on the bilateral symmetry of the subject's facial surface shape, or by estimating the presence or absence of a neuropathy based on a comparison with the subject's facial surface shape in a normal state.

[0060] The processing unit 11 determines whether the subject is estimated to have a neuropathy through the second estimation process of step S26 (step S27). If the subject is estimated to have no neuropathy (S27: NO), the processing unit 11 returns to step S21 and repeats the above-described process. If the subject is estimated to have a neuropathy (S27: YES), the diagnostic test processing unit 11f of the processing unit 11 conducts a diagnostic test for the neuropathy by conducting a question-and-answer session with the subject via the terminal device 3 (step S28). The diagnostic test processing unit 11f, for example, sends a question message to the subject to the terminal device 3, causing the terminal device 3 to output the message, and acquires the answer received from the subject by the terminal device 3. The diagnostic test processing unit 11f also sends a message to the terminal device 3 requesting the subject to perform an action or facial expression, and acquires an image of the subject's action or facial expression in response to the message, captured by the camera 36, ​​from the terminal device 3. The diagnostic test processing unit 11f performs these processes multiple times to collect information about the subject, and determines the presence or absence and degree of neurological disorder based on indices such as the CPSS, NIHSS, or KPSS.

[0061] The notification processing unit 11g of the processing unit 11 notifies the subject of the results of the estimation and diagnosis regarding the neuropathy by transmitting information regarding the result of the estimation processing in step S24, the result of the estimation processing in step S26, and / or the result of the diagnostic test in step S28 to the terminal device 3 (step S29), and then ends the processing. Note that the notification processing unit 11g may send the notification to a terminal device other than the terminal device 3 installed in the subject's home, such as a terminal device used by the subject's family or a terminal device used by the subject's doctor.

[0062] Fig. 7 is a flowchart showing the procedure of the facial image complementation process performed by the server device 1 according to embodiment 1. The facial image complementation process shown in this flowchart is executed in step S23 when it is determined in the flowchart shown in Fig. 7 that the facial image acquired from the terminal device 3 is not a frontal view image. The 3D model generation unit 11b of the processing unit 11 of the server device 1 reads out the reference 3D model stored in the reference information storage unit 12b (step S41).

[0063] The 3D model generation unit 11b generates a 3D model of the current subject's face by pasting the facial image of the subject captured by the camera 36 of the terminal device 3 onto the reference 3D model read out in step S41 (step S42). At this time, the 3D model generation unit 11b extracts feature points from the facial image, for example, and changes the facial expression of the reference 3D model so that the facial feature points of the reference 3D model match the feature points of the facial image, or selects a reference 3D model with the most matching feature points from among multiple reference 3D models with different facial expressions.

[0064] The facial image complementing unit 11c of the processing unit 11 generates a two-dimensional image of a front view from the three-dimensional model of the subject's face generated in step S42, thereby generating a facial image in which missing portions of the facial image captured by the terminal device 3 are complemented (step S43), and the facial image complementing process ends. The facial image complementing unit 11c can obtain the stored facial image, for example, by setting a virtual camera in front of the three-dimensional model of the subject's face in a three-dimensional virtual space and acquiring a two-dimensional image of the three-dimensional model captured by the virtual camera.

[0065] In this embodiment, the facial image of a subject that is the target of the facial image complementation process is a facial image that includes at least one feature point that indicates a characteristic such as facial distortion if the subject has a neurological disorder. For example, for a subject with a characteristic such as distortion on the right half of their face, a facial image that captures only the left half of their face is not the target of the facial image complementation process. If the facial image captures the left half of the face and at least a portion of the right half of the face (the portion where a characteristic such as distortion appears), a reference 3D model can be selected based on the feature point included in this portion, and a front-view facial image with a characteristic such as distortion on the right half of the face can be generated by the facial image complementation process.

[0066] Conversely, for example, in the case of a subject whose right half of the face shows characteristics such as distortion, the facial image complementing unit 11c can complement the left half of the undistorted face based on a facial image showing the distorted right half of the face and a portion of the undistorted left half of the face, thereby generating a front-view facial image in which the right half of the face is distorted and the left half is undistorted.

[0067] FIG. 8 is a schematic diagram illustrating a method for estimating the presence or absence of a nerve disorder performed by the server device 1 according to this embodiment. FIG. 8 shows an example of a facial image of a subject who may be estimated to have a nerve disorder. This facial image is a photographed image taken by the camera 36 of the terminal device 3, or an image in which missing portions are complemented based on this photographed image. The nerve disorder estimation unit 11d of the processing unit 11 of the server device 1 extracts feature points from the facial image of the subject. The facial image in FIG. 8 shows the feature points of the corners of the mouth and the ends of the eyebrows extracted from the facial image of the subject.

[0068] The nerve damage estimation unit 11d divides the subject into two halves, left and right, along the median plane (the center line indicated by the dashed line in FIG. 8 ), and compares the positional relationship of feature points on the left and right halves of the subject's face to determine the left-right symmetry of the subject's face. For example, when comparing the positional relationship between the right and left corners of the subject's mouth in FIG. 8 , the right corner of the mouth is lower than the left corner of the mouth. Similarly, the end of the subject's right eyebrow is lower than the end of the left eyebrow. The nerve damage estimation unit 11d calculates the difference in coordinates or distance from the center line of feature points on the left and right sides of the face, such as the corners of the mouth and the end of the eyebrows, and can estimate the presence of a nerve damage if this difference exceeds a predetermined threshold.

[0069] Although not shown, the neuropathy estimation unit 11d compares a reference face image of the subject taken in advance in a normal state with a face image of the subject taken by the camera 36 of the terminal device 3 or a face image complemented therefrom to estimate the presence or absence of a neuropathy in the subject. The neuropathy estimation unit 11d extracts feature points from the reference face image stored in the reference information storage unit 12b, and also extracts feature points from the face image acquired from the terminal device 3, and compares the feature points of both images. The neuropathy estimation unit 11d can estimate the presence of a neuropathy when the difference in the positions of corresponding feature points in both images exceeds a predetermined threshold.

[0070] <Display Processing> In the information processing system according to the present embodiment, when the server device 1 estimates that a subject has a neuropathy, the server device 1 notifies the subject by displaying information on the terminal device 3. FIG. 9 is a schematic diagram showing an example of a notification screen of the neuropathy estimation result. The notification screen shown in FIG. 9 may be displayed, for example, after the estimation process based on a facial image is performed and before a diagnostic test is conducted, or may be displayed, for example, after the diagnostic test is completed. The server device 1 transmits information such as an image and a message for displaying the notification screen to the terminal device 3 at an appropriate timing, and the terminal device 3, upon receiving the information, displays the illustrated notification screen on the display unit 34.

[0071] Based on the information from the server device 1, the terminal device 3 displays, for example, a facial image of the subject from one week ago and the current facial image used for estimation, side by side, as shown in Fig. 9 . At this time, if the missing portion of the facial image captured by the camera 36 has been complemented, the terminal device 3 displays the current facial image so that the actually captured portion and the complemented portion of the current facial image can be distinguished. To display the difference, a display method such as displaying in a different color or by shading can be used. In the example shown in Fig. 9 , the current facial image is displayed with the complemented portion shaded.

[0072] Furthermore, the terminal device 3 displays a message below the two facial images as the result of the server device 1's estimation of a nerve disorder, such as, "The right corner of the mouth and the tail of the eyebrow are lower than the left, suggesting a nerve disorder. We recommend that you undergo a detailed examination at a hospital." The current facial image also displays a display indicating which facial features (feature points) are the basis for the estimation of a nerve disorder. In the example shown in FIG. 9 , the terminal device 3 displays a circle surrounding the right corner of the mouth and the tail of the right eyebrow superimposed on the facial image as the basis for the estimation of a nerve disorder. Note that the method of displaying the evidence features is not limited to the superimposition of circles shown in FIG. 9 , but may also be, for example, colored or shaded, or may be displayed in a different way from other features, or any other display method may be used.

[0073] <Information Collection Processing> In the information processing system according to this embodiment, a terminal device 3 installed in the subject's home or the like continuously photographs the subject. The server device 1 estimates the presence or absence of a nerve disorder in the subject based on the facial image of the subject photographed by the terminal device 3, and the server device 1 may store the facial image used when it is estimated that the subject does not have a nerve disorder. The server device 1 stores and collects facial images of the subject in a normal state (a state without a nerve disorder), and can use the collected facial image as a reference facial image in subsequent estimation processing.

[0074] Furthermore, the terminal device 3 may continuously perform measurements using the distance measurement sensor 37, and the server device 1 may store and collect the measurement results from the distance measurement sensor 37 along with the facial image of the subject in a normal state. The server device 1 may collect facial images from the camera 36 and the measurement results from the distance measurement sensor 37 (i.e., the surface shape of the subject's face) and generate a 3D model of the subject's face based on the collected information. The server device 1 continuously collects this information, periodically generates a 3D model of the subject's face, and stores the collected facial images and the generated 3D model as the reference facial image and reference 3D model of the subject in the reference information storage unit 12b.

[0075] The collection of facial image and surface shape information by the server device 1 does not target all information acquired from the terminal device 3, but may be, for example, a few pieces of information per day. The server device 1 stores, for example, a few pieces of information selected based on various conditions from a plurality of facial images and surface shape information captured by the terminal device 3 in a day. The selection conditions may include, for example, a frontal view of the subject, an image in which the brightness of the image exceeds a predetermined threshold, an image in which the proportion of the face in the entire captured image exceeds a predetermined threshold, an image with a predetermined facial expression such as a smile, etc. The selection conditions may be any conditions.

[0076] Furthermore, the server device 1 may generate the reference 3D model at a predetermined cycle, such as once every few months. At this time, the server device 1 may update the already stored old reference face image and reference 3D model by overwriting the new reference face image and reference 3D model, or may retain the old reference face image and reference 3D model and add and store the new reference face image and reference 3D model.

[0077] <Summary> In the information processing system according to the present embodiment, the server device 1 acquires a facial image (facial information) of the subject captured (detected) by the camera 36 (sensor) of the terminal device 3, generates a three-dimensional model (facial structure information) of the subject's face based on the acquired facial image, complements missing portions of the facial image based on the generated three-dimensional model, and estimates the presence or absence of a neurological disorder of the subject based on the complemented facial image. This allows the information processing system to accurately estimate the presence or absence of a neurological disorder of the subject based on facial images captured from directions other than the frontal view. Furthermore, because the positional relationship between the subject and a device, such as a camera, capturing the subject's face is not limited, restrictions on where the device can be installed within a home, etc., can be alleviated, enabling real-time monitoring of the subject and abnormality detection.

[0078] In the information processing system according to this embodiment, the server device 1 stores a pre-generated reference 3D model of the subject in the reference information storage unit 12b, and generates a 3D model of the subject based on a facial image captured by the camera 36 of the terminal device 3 and the reference 3D model stored in the reference information storage unit 12b. The reference 3D model is generated based on multi-view facial images (multi-view facial information) captured by a camera from two or more directions of the subject's face, and a shape model (3D facial information) reproduced by measuring the surface shape of the subject's face with a distance sensor. Instead of generating and storing the reference 3D model in advance, the server device 1 may store multi-view facial images and a shape model for generating the reference 3D model, and generate the reference 3D model as needed. This information processing system is expected to generate a reference 3D model with high accuracy and accurately complement missing parts of the facial image.

[0079] Furthermore, in the information processing system according to this embodiment, the server device 1 stores and accumulates in the storage unit 12 the facial image of the subject captured by the camera 36 of the terminal device 3 and the shape information of the subject's face measured by the distance measurement sensor 37. Based on this accumulated information, the server device 1 can update the reference 3D model stored in the reference information storage unit 12b, and it is expected that the use of the updated reference 3D model will accurately complement missing parts of the facial image.

[0080] Furthermore, in the information processing system according to the present embodiment, the server device 1 estimates the presence or absence of a neurological disorder of the subject based on the results of comparing the right half of the facial image of the subject captured by the terminal device 3 or a complemented facial image thereof with the right half of the facial image stored in the reference information storage unit 12b, and the results of comparing the left half of the facial image of the subject captured by the terminal device 3 or a complemented facial image thereof with the left half of the facial image stored in the reference information storage unit 12b. For example, the server device 1 can estimate that the subject has a neurological disorder if the right half of the complemented facial image is lower than the right half of the facial image stored in the reference information storage unit 12b, or if the left half of the complemented facial image is lower than the left half of the facial image stored in the reference information storage unit 12b. This allows the information processing system according to the present embodiment to be expected to accurately estimate the presence or absence of a neurological disorder of the subject.

[0081] Furthermore, the information processing system according to the present embodiment estimates the presence or absence of a neurological disorder of a subject based on the symmetry between the left and right halves of the subject's face, which is based on a facial image of the subject captured by the terminal device 3 or a facial image obtained by complementing the captured image. The server device 1 can estimate that the subject has a neurological disorder, for example, if the right half of the face is sagging compared to the left half, or if the left half of the face is sagging compared to the right half. This allows the information processing system according to the present embodiment to be expected to accurately estimate the presence or absence of a neurological disorder of a subject.

[0082] Furthermore, in the information processing system according to this embodiment, when the face image captured by the terminal device 3 is a face image other than a frontal view, the terminal device 3 displays a complemented frontal view face image on the display unit of the terminal device 3. The terminal device 3 also displays the complemented face image and the reference face image stored in the reference information storage unit 12b side by side. The terminal device 3 may also display the complemented face image and the reference face image superimposed on each other rather than side by side. The terminal device 3 also displays, together with the face image of the subject, which part of the face image is the basis for the estimation result of the presence or absence of a nerve disorder. As a result, the information processing system according to this embodiment is expected to present the estimation result of the presence or absence of a nerve disorder to the subject in an easy-to-understand manner.

[0083] Furthermore, in the information processing system according to the present embodiment, when the server device 1 estimates that a subject has a neurological disorder based on a facial image of the subject captured by the camera 36 of the terminal device 3, the terminal device 3 measures the surface shape of the subject's face using the distance measuring sensor 37, and the server device 1 further estimates the presence or absence of a neurological disorder in the subject based on the measured surface shape. Note that the information processing system may first perform estimation based on the measurement results of the distance measuring sensor 37 and then perform estimation based on the image captured by the camera 36. Furthermore, when the information processing system according to the present embodiment estimates that the subject has a neurological disorder based on these estimation processes, it performs a diagnostic test on the subject. As a result, the information processing system according to the present embodiment is expected to accurately determine the presence or absence and degree of a neurological disorder in the subject.

[0084] In the present embodiment, the camera 36 is used as a sensor for acquiring facial information of the subject in the first process of estimating the presence or absence of a nerve disorder. However, this is merely an example and is not intended to be limiting. The sensor may be, for example, a distance sensor 37 or other sensors. In the present embodiment, the distance sensor 37 is used as the second sensor in the second process of estimating the presence or absence of a nerve disorder. However, this is merely an example and is not intended to be limiting. The second sensor may be, for example, a microphone, an event-driven camera that extracts brightness changes, a millimeter-wave sensor, an ultrasonic sensor, a thermographic camera, or other sensors. The second sensor may include multiple types of sensors. Furthermore, the information acquired by the second sensor is not limited to facial information, but may also be information indicating abnormalities in the subject, such as changes in arm sway (body sway / tremor), arm weakness, or speech abnormalities (aphasia / dysarthria).

[0085] In addition, in the information processing system according to this embodiment, the server device 1 performs a process of estimating whether or not the subject has a neurological disorder based on a facial image captured by a camera on the terminal device 3, but this is not limited to this, and the terminal device 3 may also perform a process of estimating whether or not the subject has a neurological disorder, in which case the information processing system does not need to include the server device 1.

[0086] <Embodiment 2> In the information processing system according to Embodiment 1, the server device 1 uses a pre-stored reference 3D model and a pre-stored reference face image to complement missing portions of a face image of a subject photographed by the terminal device 3. In contrast, in the information processing system according to Embodiment 2, the server device 1 complements missing portions of a face image of a subject photographed by the terminal device 3 by using a learning model (so-called AI (Artificial Intelligence)) that has been machine-learned in advance, without pre-storing reference information such as a reference 3D model and a pre-stored reference face image.

[0087] 10 is a schematic diagram for explaining the configuration of a learning model provided in the server device 1 according to the second embodiment. The server device 1 according to the second embodiment does not have a reference information storage unit 12b in the storage unit 12, and instead stores information about a trained learning model in the storage unit 12. The information about the learning model may include, for example, information defining the structure of the learning model and information such as the values ​​of internal parameters determined by machine learning. The server device 1 according to the second embodiment performs a process of complementing missing portions of a facial image of a subject using two learning models, a shape estimation model 51 and a complementation model 52.

[0088] The shape estimation model 51 is a learning model that has undergone machine learning in advance to accept a two-dimensional facial image as input, estimate the three-dimensional surface shape of the face depicted in the facial image, and output the estimation result as shape information. For example, a trained face mesh learning model may be used for the shape estimation model 51. The face mesh learning model is a machine learning model that detects facial key points (feature points) from an image and can output hundreds of feature points from a human face in three-dimensional coordinates. FIG. 11 is a schematic diagram showing an example of shape information output by the shape estimation model 51. This diagram plots multiple feature points output by the face mesh learning model in a three-dimensional virtual space, reproducing the shape of a human face and making it possible to change the facial orientation, etc., in the three-dimensional virtual space. Note that since the face mesh learning model is an existing technology, detailed explanation of the machine learning method, etc. will be omitted. Furthermore, a learning model other than a face mesh learning model may be used for the shape estimation model 51.

[0089] In this embodiment, if the subject has a neurological disorder, shape estimation model 51 can output shape information that reproduces the subject's facial distortion, etc., by inputting a face image that includes at least one feature point that indicates a feature such as facial distortion. Even if the subject has a neurological disorder, if a face image that does not include any feature points that indicate a feature such as facial distortion is input, shape estimation model 51 can output shape information that reproduces the subject's face in a normal state without reproducing the subject's facial distortion, etc.

[0090] The interpolation model 52 is a learning model that has undergone machine learning in advance to receive as input a two-dimensional facial image and shape information estimated from the facial image by the shape estimation model 51, and generate a three-dimensional model of the face depicted in the facial image. The three-dimensional model output by the interpolation model 52 is a three-dimensional model in which the facial image is pasted onto the input shape information and missing portions of the facial image are complemented. The interpolation model 52 may employ a learning model such as a deep neural network (DNN) or a convolutional neural network (CNN). The interpolation model 52 may be generated by acquiring a facial image, shape information, and a three-dimensional model using a procedure similar to that used for generating a three-dimensional model as pre-processing in the information processing system according to embodiment 1 (see FIG. 5 ), and then performing machine learning using these as training data. Note that the machine learning processes of these learning models are existing technologies, and detailed descriptions thereof will be omitted.

[0091] 12 is a flowchart showing the procedure of the facial image complementation process performed by the server device 1 according to Embodiment 2. The 3D model generation unit 11b of the processing unit 11 of the server device 1 according to Embodiment 2 inputs a facial image of a subject captured by the camera 36 of the terminal device 3 to the shape estimation model 51 that has been machine-learned in advance (step S61). The 3D model generation unit 11b acquires shape information output by the shape estimation model 51 (step S62).

[0092] Next, the 3D model generation unit 11b inputs the subject's facial image and the shape information acquired in step S62 to the complementary model 52 (step S63). The 3D model generation unit 11b acquires the 3D model output by the complementary model 52 (step S64) to generate a 3D model of the subject's face. The facial image completion unit 11c of the processing unit 11 generates a front-view 2D image from the 3D model of the subject's face generated in step S63, thereby generating a facial image in which missing portions of the facial image captured by the terminal device 3 are completed (step S65), and the facial image completion process ends.

[0093] In the information processing system according to the second embodiment, similarly to the information processing system according to the first embodiment, the terminal device 3 stores and accumulates face images captured by the camera 36 and / or shape information measured by the distance measuring sensor 37. This accumulated information can be used to re-learn the complementary model 52. The server device 1 stores information several times a day, for example, and can re-learn the complementary model 52 once a week, for example.

[0094] Furthermore, the server device 1 according to the second embodiment complements a facial image of a subject captured by the camera 36 of the terminal device 3, estimates that the subject has a neurological disorder based on the complemented facial image, and then, for example, estimates that the subject does not have a neurological disorder without performing the complementation process based on a front-view facial image captured by the camera 36 of the terminal device 3 within a predetermined time period. The server device 1 stores, as information for re-learning, information such as the facial image used in the initial estimation process that was erroneous, the complemented facial image, the results of measurements by the distance measuring sensor 37 performed subsequent to the initial estimation process, and facial images used in subsequent estimation processes. In this way, by accumulating information when the server device 1 determines that the estimation result was erroneous and using it for re-learning the complementation model 52, it is expected that the accuracy of facial image complementation by the complementation model 52 can be improved.

[0095] In the information processing system according to the second embodiment configured as described above, the server device 1 includes a learning model (a learning model combining the shape estimation model 51 and the complementation model 52) that has undergone machine learning to accept as input a facial image of the subject's face and output a three-dimensional model of the subject. The server device 1 inputs a facial image captured by the camera 36 of the terminal device 3 into the learning model and acquires the three-dimensional model output by the learning model, thereby generating a three-dimensional model from the facial image of the subject. As a result, the information processing system according to the second embodiment is expected to accurately generate a three-dimensional model of the subject and accurately complement missing parts of the facial image.

[0096] The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0097] REFERENCE SIGNS LIST 1 Server device 3 Terminal device 11 Processing unit 11a Facial image acquisition unit 11b 3D model generation unit 11c Facial image completion unit 11d Neurological disorder estimation unit 11e Second estimation unit 11f Diagnostic test processing unit 11g Notification processing unit 12 Storage unit 12a Server program 12b Reference information storage unit 13 Communication unit 31 Processing unit 31a Photography processing unit 31b Distance measurement processing unit 31c Diagnostic test processing unit 32 Storage unit 32a Program 33 Communication unit 34 Display unit 35 Operation unit 36 ​​Camera 37 Distance measurement sensor 51 Shape estimation model 52 Complementary model 98, 99 Recording medium N Network

Claims

1. On the computer, Acquire face information of the subject detected by the sensor, generating facial structure information of the subject based on the acquired facial information; Complementing missing facial information of the subject based on the generated facial structure information; Based on the complemented face information, the presence or absence of a neurological disorder of the subject is estimated. A computer program that executes a process.

2. Acquire previously generated reference facial structure information of the subject from a storage unit; generating facial structure information of the subject based on the facial information detected by the sensor and the reference facial structure information stored in the storage unit; 2. The computer program product of claim 1.

3. The reference facial structure information includes multi-view face information acquired by a sensor from two or more directions with respect to the front of the face of the subject.

3. A computer program according to claim 2.

4. The reference facial structure information includes three-dimensional facial information obtained by a sensor that captures a three-dimensional shape of the subject's face. A computer program according to claim 2 or 3.

5. The reference facial structure information includes facial structure information generated based on multi-view face information acquired by a sensor from two or more directions with respect to the front of the face of the subject, and / or facial structure information generated based on three-dimensional face information acquired by a sensor of a three-dimensional shape of the face of the subject. A computer program according to claim 2 or 3.

6. storing the acquired face information or face structure information generated based on the face information in the storage unit as the reference face structure information; A computer program according to claim 2 or 3.

7. estimating the presence or absence of a neurological disorder of the subject based on a comparison result between the right half of the face of the subject based on the complemented facial information and the right half of the face based on the reference facial structure information, and based on a comparison result between the left half of the face of the subject based on the complemented facial information and the left half of the face based on the reference facial structure information; A computer program according to claim 2 or 3.

8. When the right half of the face based on the complemented facial information is lowered compared to the right half of the face based on the reference facial structure information, or when the left half of the face based on the complemented facial information is lowered compared to the left half of the face based on the reference facial structure information, it is estimated that the subject has a neurological disorder.

8. A computer program according to claim 7.

9. A learning model that has been machine-learned to receive facial information of a subject as an input and output facial structure information of the subject is used to input the facial information detected by the sensor, and the facial structure information output by the learning model is obtained, thereby generating facial structure information of the subject. A computer program according to any one of claims 1 to 3.

10. Inferring the presence or absence of a neurological disorder of the subject based on the symmetry of the right and left halves of the subject's face based on the complemented face information. A computer program according to any one of claims 1 to 3.

11. If the right half of the face is lowered compared to the left half, or if the left half of the face is lowered compared to the right half, the subject is presumed to have a neurological disorder.

11. A computer program product as claimed in claim 10.

12. After estimating that the subject has a neurological disorder based on facial information supplemented with the facial information detected by the sensor, if it is estimated that the subject does not have a neurological disorder without supplementation based on the facial information detected by the sensor, the facial information at the time of estimating that the subject has a neurological disorder is stored in a storage unit. A computer program according to any one of claims 1 to 3.

13. When the face information detected by the sensor is face information other than a front view of the subject, the complemented front view face information is displayed on a display unit. A computer program according to any one of claims 1 to 3.

14. The complemented face information and the face information of the subject pre-stored in a storage unit are displayed side by side or superimposed on the display unit.

14. A computer program product as claimed in claim 13.

15. and displaying on the display unit a part involved in the result of estimating the presence or absence of the nerve disorder together with face information.

14. A computer program product as claimed in claim 13.

16. When it is estimated that the subject has a neuropathy, information of the subject is acquired by a second sensor different from the first sensor; Further estimating the presence or absence of a neurological disorder of the subject based on the information acquired by the second sensor. A computer program according to any one of claims 1 to 3.

17. Acquire information of the subject using a second sensor different from the sensor; Inferring the presence or absence of a neurological disorder of the subject based on the information acquired by the second sensor; When it is estimated that the subject has a neurological disorder, the presence or absence of the neurological disorder of the subject is further estimated based on the face information acquired by the sensor. A computer program according to any one of claims 1 to 3.

18. administering a diagnostic test to the subject if the subject is suspected to have a neurological disorder. A computer program according to any one of claims 1 to 3.

19. The sensor is a camera that captures two-dimensional images. A computer program according to any one of claims 1 to 3.

20. The sensor is a distance measurement sensor. A computer program according to any one of claims 1 to 3.

21. An information processing device, Acquire face information of the subject detected by the sensor, generating facial structure information of the subject based on the acquired facial information; Complementing missing facial information of the subject based on the generated facial structure information; Based on the complemented face information, the presence or absence of a neurological disorder of the subject is estimated. Information processing methods.

22. An acquisition unit that acquires face information of a target person detected by a sensor; A generation unit that generates facial structure information of the subject based on the acquired facial information; a complementing unit that complements missing facial information of the subject based on the generated facial structure information; an estimation unit that estimates the presence or absence of a neurological disorder of the subject based on the complemented face information; An information processing device comprising: