Program, information processing method, and information processing device

Abstract

A program according to one embodiment of the present disclosure detects a rotational movement of the head of a subject, and determines whether the subject has a brain disorder on the basis of the movement range of the rotational movement of the head. Furthermore, the program according to the one embodiment of the present disclosure detects the rotational movement of the head of the subject with respect to a vertical axis, and determines, on the basis of the difference between the movement range of the head in the left direction and the movement range of the head in the right direction, whether the subject is experiencing symptoms in which left and / or right visual information is missing.

Description

Program, information processing method, and information processing device

[0001] The present technology relates to a program, an information processing method, and an information processing device.

[0002] A cognitive function testing method has been proposed in the past, which involves presenting multiple visual stimuli in a predetermined order at a predetermined position for each visual stimulus within the subject's visual field, detecting the position of the gaze point relative to the visual stimuli from eye movement during the presentation of the visual stimuli, measuring the left-right deviation of the gaze point position, and evaluating the left-right difference in the subject's cognitive function based on the measurement results (see Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2006-087833

[0004] However, the cognitive function testing method described in Patent Document 1 cannot simply determine whether or not a subject has a brain dysfunction.

[0005] The present disclosure has been made in consideration of the above circumstances, and aims to provide a program or the like that can easily determine whether or not a subject has a brain dysfunction.

[0006] (1) A program according to one embodiment of the present disclosure detects a rotational movement of a subject's head and determines whether the subject has a brain dysfunction based on the range of motion of the rotational movement of the head.

[0007] (2) The program in (1) detects the rotational movement of the subject's head around the vertical axis, and determines whether the subject has a symptom of a lack of visual information on at least one of the left and right sides based on the difference between the range of movement of the head to the left and the range of movement to the right.

[0008] (3) The program (1) or (2) outputs instructions to the subject to move the head to the left and right, and determines whether the subject has hemispatial neglect based on the rotational movement detected after the output of the instructions.

[0009] (4) The program described in any one of (1) to (3) above acquires a plurality of time-series images including the subject's head, extracts feature points of the subject from the acquired plurality of images, and determines whether the subject has a brain dysfunction based on the difference in the positions of the feature points between the extracted plurality of images.

[0010] (5) The program described in any one of (1) to (4) above detects the rotational movement of the subject's head around a vertical axis and determines whether the subject has a brain dysfunction based on the difference between the range of movement of the head to the left and the range of movement to the right.

[0011] (6) The program described in any one of (1) to (5) above determines that the subject has a brain dysfunction if the difference between the angle of rotation of the head to the left and the angle of rotation to the right is greater than or equal to a predetermined angle.

[0012] (7) The program described in any one of (1) to (6) above stores the subject's past range of head movement, and if the difference between the stored range of movement and the range of movement of the detected rotational movement is greater than or equal to a predetermined value, determines that the subject has a brain dysfunction.

[0013] (8) The program described in any one of (1) to (7) above determines whether the subject has a brain dysfunction based on the difference between the range of movement of the head to the left and the range of movement to the right relative to a specified object.

[0014] (9) The program according to any one of (1) to (8) above displays the object on a display device.

[0015] (10) The program described in any one of (1) to (9) above acquires an image of the subject's surroundings and identifies the object from objects contained in the acquired image.

[0016] (11) The program described in any one of (1) to (10) above determines whether the subject has a brain dysfunction based on the positional relationship between the object and the subject, using either a first determination method based on the difference between the range of movement of the subject's head to the left and the range of movement of the subject's head to the right when the head is rotated about a vertical axis, or a second determination method based on the difference between the range of movement of the subject's head to the left and the range of movement of the subject's head to the right when the head is rotated about a specified object.

[0017] (12) The program described in any one of (1) to (11) above determines whether the subject can see the object based on the positional relationship between the object and the subject, and if it is determined that the subject cannot see the object, determines whether the subject has a brain dysfunction using the first determination method, and if it is determined that the subject can see the object, determines whether the subject has a brain dysfunction using the second determination method.

[0018] (13) The program described in any one of (1) to (12) above acquires the position of the subject relative to an object as perceived by the subject, acquires the actual position of the subject relative to the object, determines whether the position perceived by the subject matches the actual position, and if they do not match, determines that the subject has a brain dysfunction, and if they match, determines whether the subject has a brain dysfunction based on the range of motion of the head rotation movement.

[0019] (14) An information processing method according to one embodiment of the present disclosure detects a rotational movement of a subject's head and determines whether the subject has a brain dysfunction based on the range of motion of the rotational movement of the head.

[0020] (15) An information processing device according to one embodiment of the present disclosure includes a control unit that detects a rotational movement of a subject's head and determines whether the subject has a brain dysfunction based on the range of motion of the rotational movement of the head.

[0021] A program according to an embodiment of the present disclosure can easily determine whether a subject has a brain dysfunction.

[0022] FIG. 1 is an explanatory diagram showing an example of the configuration of a brain dysfunction determination system. FIG. 1 is an explanatory diagram showing an example of the configuration of an information processing device. FIG. 1 is an explanatory diagram related to a posture estimation model. FIG. 2 is a schematic diagram explaining the rotational movement of the head of a subject. FIG. 2 is a flowchart showing an example of the processing by a control unit of an information processing device. FIG. 1 is an explanatory diagram showing an example of the configuration of an information processing device according to embodiment 2. FIG. 2 is an explanatory diagram related to a feature point extraction model. FIG. 3 is an explanatory diagram explaining the rotational movement of the head of a subject according to embodiment 2. FIG. 3 is a flowchart showing an example of the processing by a control unit of an information processing device according to embodiment 2. FIG. 4 is an explanatory diagram showing an example of the configuration of an information processing device according to embodiment 3. FIG. 4 is an explanatory diagram showing an example of a history table. FIG. 5 is a flowchart showing an example of the processing by a control unit of an information processing device according to embodiment 3. FIG. 5 is an explanatory diagram showing an example of the configuration of a brain dysfunction determination system according to embodiment 4. FIG. 6 is an explanatory diagram related to a posture estimation model and an object detection model according to embodiment 4. FIG. 6 is a schematic diagram explaining the rotational movement of the head of a subject according to embodiment 4. FIG. 7 is a flowchart showing an example of the processing by a control unit of an information processing device according to embodiment 4. FIG. 7 is an explanatory diagram related to a posture estimation model and an object detection model according to embodiment 5. FIG. 8 is a flowchart showing an example of the processing by a control unit of an information processing device according to embodiment 6. 13 is an explanatory diagram of a posture estimation model and an object detection model according to embodiment 7. FIG. 14 is a flowchart showing an example of processing by a control unit of an information processing device according to embodiment 7.

[0023] The present invention will be described in detail below with reference to the drawings illustrating embodiments. (Embodiment 1) FIG. 1 is an explanatory diagram illustrating an example of the configuration of a brain dysfunction assessment system S. The brain dysfunction assessment system S according to embodiment 1 includes an information processing device 1, a photographing device 2, and an audio output device 3. The information processing device 1 is, for example, a server computer installed by a medical institution, and determines whether a subject has a brain dysfunction based on an image of the subject acquired from the photographing device 2. The information processing device 1 may be a personal computer, smartphone, tablet device, or the like owned by the subject. The photographing device 2 is, for example, a camera (such as a surveillance camera or security camera) installed in the subject's living room, and transmits images of the subject to the information processing device 1. The photographing device 2 may be a camera built into a smartphone, smart display, or tablet device, or a camera installed in an examination room, examination room, or hospital room of a medical institution. The audio output device 3 is, for example, a speaker, and outputs audio instructions to the subject acquired from the information processing device 1. The audio output device 3 may be a speaker built into a smartphone, smart display, or tablet terminal, or a speaker installed in a medical institution's examination room, testing room, or hospital room. The brain dysfunction assessment system S may include a display device instead of or in addition to the audio output device 3, and may display and output instructions to the subject. The information processing device 1, the image capture device 2, and the audio output device 3 are communicatively connected to each other via a communication network N. The information processing device 1 and the image capture device 2 may be connected via a wired connection. At least two of the information processing device 1, the image capture device 2, and the audio output device 3 may be integrated into one device, and their functions may be realized by, for example, a smartphone, smart display, tablet terminal, or personal computer. The device in which the image capture device 2 and the audio output device 3 are integrated may be an AR (Augmented Reality) device or a VR (Virtual Reality) device.

[0024] The information processing device 1 determines whether or not a subject included in an image acquired from the imaging device 2 has a brain dysfunction. In this embodiment, the brain dysfunction is a visual impairment caused by, for example, a stroke, including cerebral infarction and cerebral hemorrhage, or a brain tumor, and includes symptoms in which there is a deficit in at least one of the left and right visual information. Furthermore, symptoms in which there is a deficit in at least one of the left and right visual information include hemianopia and unilateral spatial neglect, which is a type of directional attention disorder. Below, a description is given of an aspect in which the information processing device 1 determines whether or not a subject has unilateral spatial neglect, but the information processing device 1 may also determine whether or not the subject is hemianopic, for example.

[0025] FIG. 2 is an explanatory diagram showing an example configuration of the information processing device 1. The information processing device 1 includes a control unit 11, a storage unit 12, and a communication unit 13. The control unit 11 includes, for example, a central processing unit (CPU), a read-only memory (ROM), and a random access memory (RAM). The ROM included in the control unit 11 stores control programs and the like that control the operation of each hardware unit included in the information processing device 1. The CPU in the control unit 11 reads and executes the control programs stored in the ROM and the computer programs (described below) stored in the storage unit 12, and controls the operation of each hardware unit, thereby causing the entire device to function as the information processing device 1 of the present disclosure. The RAM included in the control unit 11 temporarily stores data used during calculation and control. Note that the processing performed by the control unit 11 may be distributed among multiple processors or multiple computers.

[0026] In this embodiment, the control unit 11 is configured to include a CPU, a ROM, and a RAM, but the configuration of the control unit 11 is not limited to the above. The control unit 11 may be one or more control circuits or arithmetic circuits including, for example, a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processor (DSP), a quantum processor, a volatile or non-volatile memory, or the like.

[0027] The storage unit 12 includes a storage device such as a hard disk drive (HDD), a solid state drive (SSD), etc. The storage unit 12 stores a computer program P executed by the control unit 11, a posture estimation model M1, and various data used by the control unit 11.

[0028] The computer program P is provided by a non-transitory recording medium 12a on which the computer program P is readably recorded. The recording medium 12a is a portable memory such as a CD-ROM, a USB memory, or an SD (Secure Digital) card. The control unit 11 reads the computer program P from the recording medium 12a using a reading device (not shown) and stores the read computer program P in the memory unit 12. The computer program P may be provided to the information processing device 1 by communication via the communication network N.

[0029] The communication unit 13 includes a communication interface for transmitting and receiving various data to and from an external device. The communication interface of the communication unit 13 may be a communication interface conforming to a communication standard such as Wi-Fi (registered trademark), LAN (Local Area Network), Bluetooth (registered trademark), ZigBee (registered trademark), 3G, 4G, 5G, or LTE (Long Term Evolution). When data to be transmitted is input from the control unit 11, the communication unit 13 transmits the data to the destination external device, and when data transmitted from the external device is received, the communication unit 13 outputs the received data to the control unit 11. In this embodiment, examples of external devices are the imaging device 2 and the audio output device 3.

[0030] FIG. 3 is an explanatory diagram of the pose estimation model M1. The pose estimation model M1 is configured by a learning model that estimates the pose of a person in an image, such as Open Pose, which includes a neural network (NN). The pose estimation model M1 may also be configured by a learning model capable of detecting objects in an image, such as a convolutional neural network (CNN), a single-shot multibox detector (SSD), or a YOLO (you-only-look-at-once) model. The pose estimation model M1 has multiple neurons that receive input pixel values ​​of an image and passes the input pixel values ​​to a middle layer. The middle layer has multiple neurons that extract image features of the image and passes the extracted image features to an output layer. The output layer outputs the positions of key points (feature points on the head and joints) and bones (lines connecting each key point) of a person (subject) included in the image based on the image features. In the example shown in the image of the detection results in Figure 3, key points (circles) indicating the subject's eyes (left and right), ears (left and right), nose, neck joint, and shoulder joint (left and right), and bones (dotted lines) connecting each key point between the eyes and ears, eyes and nose, nose and neck joint, and neck joint and shoulder joint are detected and displayed on the image.

[0031] The control unit 11 of the information processing device 1 determines the orientation of the subject's head based on the positions of the output key points. The control unit 11 determines the subject's vertical axis, for example, based on the subject's shoulder and neck key points. The subject's vertical axis is the axis of rotation perpendicular to the horizontal plane when the subject's head rotates left and right, and is the center of the head when the subject is viewed from above. In other words, the vertical axis is an axis passing through the midpoint between the shoulder key points and the neck key point. The control unit 11, for example, determines an XY plane in the space of the captured image that intersects the vertical axis perpendicularly and includes the nose key point. The control unit 11 determines the direction from the intersection of the vertical axis and the XY plane toward the nose key point as the orientation of the subject's head on the XY plane. In FIG. 3 , the subject's vertical axis is indicated by a dashed line, and the head orientation is indicated by a solid line with an arrow. Note that the control unit 11 may also determine the orientation of the subject's head based on the angle between the vertical axis and the bone connecting the nose key point and the neck key point. 3, it can be seen that the subject's head is rotated about 20 degrees to the right as seen from the patient relative to the imaging position. The control unit 11 may also identify the head orientation using a learning model that outputs the left-right orientation of the head (the angle relative to the imaging position) when the position coordinates in the image of a key point identified on the subject's head are input.

[0032] The control unit 11 of the information processing device 1 acquires multiple images in a time series that are continuously captured from the image capture device 2. The control unit 11 inputs the multiple consecutive images into the posture estimation model M1 and identifies the head orientation in each image based on the positions of the output key points. The control unit 11 detects the subject's head rotational movement about the vertical axis based on changes in head orientation between images. The control unit 11 identifies a reference image, a leftmost image, and a rightmost image from the multiple images in the time series. The reference image is the image first acquired from the image capture device 2 after the control unit 11 outputs an instruction to the subject to look widely left and right around their surroundings. In this embodiment, the leftmost image is the image in the multiple images in the time series in which the subject's head rotates furthest to the left as seen from the subject. The rightmost image is the image in the multiple images in the time series in which the subject's head rotates furthest to the right as seen from the subject. In addition, the control unit 11 may identify the leftmost image as the image in which the subject's head is rotating furthest to the left as seen by a third party other than the subject, and the rightmost image as the image in which the subject's head is rotating furthest to the right as seen by a third party other than the subject.

[0033] FIG. 4 is a schematic diagram illustrating the rotation of the subject's head. FIG. 4 shows projections (XY plane including the nose keypoint) of the subject relative to the imaging device 2 from above, corresponding to a reference image, a leftmost image (an image in which the subject's head is rotated most to the left as seen from the subject's perspective), and a rightmost image (an image in which the subject's head is rotated most to the right as seen from the subject's perspective) for a case in which the subject is healthy and a case in which the subject has left-sided hemispatial neglect. In FIG. 4, the head orientation of the subject identified as shown in FIG. 3 is indicated by a solid line with an arrow. Furthermore, in the leftmost and rightmost images of FIG. 4, the head orientation in the reference image is indicated by a dashed line with an arrow. When the subject is healthy, the difference in the angle of the head orientation to the left in the leftmost image relative to the head orientation in the reference image (left angle) and the difference in the angle of the head orientation to the right in the rightmost image (right angle) are approximately equal. In contrast, when the subject has hemispatial neglect, the head rotation angle toward the ignored space is smaller than the head rotation angle toward the opposite side of the ignored space. In the example shown on the right side of Figure 4, because the left angle is smaller than the right angle, the control unit 11 of the information processing device 1 determines that the left side is the side of the space to be ignored and determines that the subject has left-sided hemispatial neglect. In this way, the control unit 11 calculates the range of left and right movement of the subject's head by angle, and determines whether the subject has left-sided hemispatial neglect based on the difference between the range of leftward movement of the subject's head (left angle) and the range of rightward movement of the subject's head (right angle).

[0034] FIG. 5 is a flowchart showing an example of processing by the control unit 11 of the information processing device 1. For example, when an abnormality (such as facial paralysis, upper limb paralysis, or a symptom suspected to be a brain dysfunction, such as an articulation disorder) is detected through screening of the subject's daily life, or when the subject or a third party transmits (inputs) information indicating an abnormality to the information processing device 1, the control unit 11 starts the following processing. The control unit 11 causes the audio output device 3 to output a voice command to the subject to move into the imaging range of the image capture device 2 (S1). Note that if the control unit 11 has previously detected that the subject is within the imaging range of the image capture device 2, the processing related to S1 may not be executed. After the subject positions (moves) into the imaging range of the image capture device 2, the control unit 11 causes the audio output device 3 to output a voice command to the subject to look around widely to the left and right (S2). The control unit 11 then causes the image capture device 2 to start capturing an image of the subject (S3). When the photographing device 2 starts photographing images, it photographs consecutive images (moving images) for, for example, 10 seconds and transmits the photographed consecutive images to the information processing device 1. The control unit 11 of the information processing device 1 acquires the consecutive images from the photographing device 2 (S4).

[0035] The control unit 11 of the information processing device 1 inputs multiple images acquired from the image capture device 2 into the posture estimation model M1 (S5) and outputs the positions of the subject's key points and bones in each image (S6). The control unit 11 identifies the vertical axis of the subject's head based on the positions of the output key points (S7). The control unit 11 identifies the orientation of the subject's head based on the vertical axis and the subject's nose key point (S8). The control unit 11 calculates the difference in the left-right angle between the head orientation (left-right range of head rotation) in each image and the head orientation (left-right range of head rotation) in the image first captured by the image capture device 2 (reference image) (S9).

[0036] The control unit 11 of the information processing device 1 identifies the image (leftmost image) with the largest difference in the angle of the head orientation to the left (left angle) relative to the head orientation in the reference image, and the image (rightmost image) with the largest difference in the angle of the head orientation to the right (right angle) relative to the head orientation in the reference image (S10). The control unit 11 determines whether the difference between the left angle in the leftmost image and the right angle in the rightmost image is equal to or greater than a predetermined angle (e.g., 30 degrees) (S11). If the difference between the left angle and the right angle is equal to or greater than the predetermined angle (S11: YES), the control unit 11 determines that the subject has hemispatial neglect (S12). The control unit 11 then notifies, for example, the subject's doctor that the subject has hemispatial neglect (S13), and ends the process. The control unit 11 may also notify the subject's family or call an emergency services center. In addition, in S13, the control unit 11 may notify the subject of the degree of hemispatial neglect, including the direction of the hemispatial neglect (left or right), or the level of severity of the symptom determined based on the difference between the left angle and the right angle or the difference between the left angle and the right angle (for example, the greater the difference between the left angle and the right angle, the higher the level of severity can be set). If the difference between the left angle and the right angle is less than a predetermined angle (S11: NO), the control unit 11 ends the processing. Note that the control unit 11 may also determine whether the subject has hemispatial neglect based on the ratio between the left angle and the right angle.

[0037] According to the configuration and processing of the first embodiment, the information processing device 1 determines whether or not the subject has hemispatial neglect based on the image of the subject captured by the imaging device 2. This makes it possible to easily and quickly determine whether or not the subject has a cerebral dysfunction, and to promptly and accurately address, for example, acute cerebral artery occlusion.

[0038] (Embodiment 2) The information processing device 1 according to embodiment 2 extracts feature points of a subject from an image acquired from the imaging device 2, and determines whether the subject has a brain dysfunction based on the difference in the positions of the extracted feature points.

[0039] 6 is an explanatory diagram showing an example of the configuration of an information processing device 1 according to embodiment 2. The storage unit 12 of the information processing device 1 according to embodiment 2 stores a feature point extraction model M2.

[0040] FIG. 7 is an explanatory diagram of the feature point extraction model M2. The image captured by the imaging device 2 according to the second embodiment and input to the feature point extraction model M2 is an image captured so as to include the subject's head. The feature point extraction model M2 is a model that has a segmentation function, such as a CNN, an RCNN (Regions with Convolutional Neural Network), a Fast RCNN, a Faster RCNN, SSD, YOLO, or a Vision Transformer, and detects objects. When the feature point extraction model M2 is configured as a neural network including a CNN that extracts image features, such as an RCNN, it has multiple neurons that accept input pixel values ​​of the image and passes the input pixel values ​​to an intermediate layer. The intermediate layer has multiple neurons that extract image features of the image and passes the extracted image features to an output layer. The output layer outputs the position of the feature, using the subject's ears included in the image as feature points based on the image features. Note that the positions of the subject's eyes, nose, or mouth may also be used as feature points. In the output image of FIG. 7, the ear feature points are indicated by diagonal lines.

[0041] FIG. 8 is an explanatory diagram illustrating the rotation of the subject's head according to the second embodiment. The control unit 11 of the information processing device 1 according to the second embodiment acquires a plurality of time-series images including the subject's head, which are continuously captured by the image capture device 2. As shown in FIGS. 7 and 8 , the images according to the second embodiment include the subject's head. When the subject rotates his or her head, the positions of feature points in the images change. The control unit 11 identifies the image in which the subject faces furthest to the left or right based on the positions of the feature points. For example, if the feature points are ears, the control unit 11 identifies the image in which the left ear is located furthest to the right of the subject as the image facing furthest to the right (the rightmost image), and the image in which the right ear is located furthest to the left of the subject as the image facing furthest to the left (the leftmost image). FIG. 8 shows a reference image, a leftmost image, and a rightmost image for a healthy subject and a subject with right-sided hemispatial neglect, respectively. The control unit 11 calculates the difference (left ear movement distance) between the position of the left ear in the image (reference image) first captured by the image capture device 2 and the position of the left ear in the rightmost image, and the difference (right ear movement distance) between the position of the right ear in the reference image and the position of the right ear in the leftmost image. In the example shown in FIG. 8 , the left ear movement distance and the right ear movement distance are calculated by comparing the positions of the outermost parts of the subject's ears (the parts of the ears farthest from the subject's nose). If the subject has hemispatial neglect, the movement distance of the ear on the side of the ignored space will be shorter. If the difference between the left ear movement distance and the right ear movement distance is equal to or greater than a predetermined distance (e.g., 3% of the left-right width of the subject's head in the reference image (the distance between the left and right ears)) then the control unit 11 determines that the subject has hemispatial neglect (cerebral dysfunction).

[0042] In the example shown in Figure 8 for a case where hemispatial neglect is present, the left ear movement distance is smaller than the right ear movement distance. Therefore, the control unit 11 of the information processing device 1 determines that the right side is the ignored side of the space and determines that the subject has right-sided hemispatial neglect. In this way, the control unit 11 calculates the range of left and right movement of the subject's head rotation based on the ear movement distance in the image, and determines whether the subject has hemispatial neglect based on the difference between the left ear movement distance and the right ear movement distance. The control unit 11 may also extract feature points of the eyes, nose, or mouth using the feature point extraction model M2 and determine whether the subject has hemispatial neglect based on the movement distance of the eye, nose, or mouth feature points.

[0043] 9 is a flowchart showing an example of processing by the control unit 11 of the information processing device 1 according to the second embodiment. The control unit 11 causes the audio output device 3 to output audio instructions to the subject to move into the shooting range of the image capture device 2 (S21). Note that if the control unit 11 has previously detected that the subject is within the shooting range of the image capture device 2, the processing of S21 need not be executed. After the subject is positioned (moved) into the shooting range of the image capture device 2, the control unit 11 causes the image capture device 2 to output audio instructions to the subject to look around widely to the left and right (S22). The control unit 11 then causes the image capture device 2 to start capturing images of the subject (S23). When the image capture starts, the image capture device 2 captures consecutive images (moving images) for, for example, 10 seconds and transmits the captured consecutive images to the information processing device 1. The control unit 11 of the information processing device 1 acquires the consecutive images from the image capture device 2 (S24).

[0044] The control unit 11 of the information processing device 1 inputs multiple images acquired from the image capture device 2 into the feature point extraction model M2 (S25) and outputs the positions of the subject's feature points in each image (S26). The control unit 11 identifies the rightmost and leftmost images based on the output feature point positions (S27). The control unit 11 calculates the left ear movement distance based on the reference image and the rightmost image (S28). The control unit 11 calculates the right ear movement distance based on the reference image and the leftmost image (S29).

[0045] The control unit 11 of the information processing device 1 determines whether the difference between the left ear movement distance and the right ear movement distance is equal to or greater than a predetermined distance (e.g., 3% of the horizontal width of the subject's head in the reference image) (S30). If the difference between the left ear movement distance and the right ear movement distance is equal to or greater than the predetermined distance (S30: YES), the control unit 11 determines that the subject has hemispatial neglect (S31), notifies the subject's doctor, for example (S32), and terminates the process. The control unit 11 may also notify the subject's family or call an ambulance. In addition, the control unit 11 may notify the subject of the direction of hemispatial neglect (left or right), the difference between the left ear movement distance and the right ear movement distance, or the degree of hemispatial neglect, including the level of severity of symptoms determined based on the difference between the left ear movement distance and the right ear movement distance. If the difference between the left ear movement distance and the right ear movement distance is less than the predetermined distance (S30: NO), the control unit 11 terminates the process. The control unit 11 may determine whether or not the subject has unilateral spatial neglect based on the ratio between the left ear movement distance and the right ear movement distance.

[0046] According to the configuration and processing of the second embodiment, the information processing device 1 determines whether or not the subject has hemispatial neglect based on the image of the subject captured by the imaging device 2. This makes it possible to easily and quickly determine whether or not the subject has a cerebral dysfunction, and to promptly and appropriately address, for example, acute cerebral artery occlusion.

[0047] (Embodiment 3) The storage unit 12 of the information processing device 1 according to embodiment 3 stores the range of left and right head rotation movements of the subject in the past, and if the difference between the stored range of movement and the range of detected rotation movement is equal to or greater than a predetermined value, it determines that the subject has a cerebral dysfunction (hemispatial neglect). If the subject has a history of hemispatial neglect, even if they do not currently have a stroke or brain tumor, a difference in the range of head movement between the left and right sides may occur due to sequelae of the stroke or brain tumor. The control unit 11 compares the subject's past head movement range with the subject's current head movement range to determine whether the subject has recently developed hemispatial neglect.

[0048] 10 is an explanatory diagram showing an example of the configuration of an information processing device 1 according to embodiment 3. The storage unit 12 of the information processing device 1 according to embodiment 3 stores a history table 121.

[0049] 11 is an explanatory diagram showing an example of the history table 121. The history table 121 records the range of rotational movement of the subject's head that was previously detected by the control unit 11. The management items (fields) of the history table 121 include, for example, a recording time field, a recording location field, a left angle field, a right angle field, an angle difference field, and a status field. The recording time field stores the date, hour, and minute when the range of movement of the subject's head was recorded in the history table 121.

[0050] The recording location field of the history table 121 stores the location where the subject's head motion range was detected. In this embodiment, the subject's head motion range is detected not only at the subject's home but also, for example, at a rehabilitation facility. A subject with a history of hemispatial neglect due to a stroke or brain tumor undergoes rehabilitation to resolve the hemispatial neglect, for example, at a rehabilitation facility. When the recording location is a rehabilitation facility, the history table 121 stores the left angle and right angle detected by the control unit 11 of the information processing device 1 based on images captured during rehabilitation for hemispatial neglect or measured by a medical professional. Note that the location where the subject's rehabilitation is performed is not limited to a rehabilitation facility. The subject's rehabilitation may also be performed at the subject's home through home rehabilitation or online rehabilitation. When rehabilitation is performed at the subject's home, the recording location field may store, for example, "Home (Rehabilitation)."

[0051] The left angle field of the history table 121 stores the left angle calculated by the control unit 11 based on the image. The right angle field stores the right angle calculated by the control unit 11 based on the image. The angle difference field stores the absolute value of the difference between the left angle and the right angle. The status field stores the determination result (presence of hemispatial neglect, absence of hemispatial neglect, or presence of sequelae (a state affected by the after-effects of a stroke)) made by the control unit 11 of the information processing device 1 based on the left angle and right angle, or, if the recording location is a rehabilitation facility, the determination result input by a medical professional (presence of hemispatial neglect, absence of hemispatial neglect, or presence of sequelae (a state affected by the after-effects of a stroke)).

[0052] FIG. 12 is a flowchart showing an example of processing performed by the control unit 11 of the information processing device 1 according to the third embodiment. The processing performed in steps S41 to S50 is the same as the processing performed in steps S1 to S10 shown in FIG. 5 . The control unit 11 reads, for example, a record recorded as having a recent sequelae from the history table 121 (S51). If no record recorded as having a recent sequelae is present in the history table 121 in S51, the control unit 11 may read a record recorded as having no recent hemispatial neglect or a reference record. The reference record may be, for example, the oldest record recorded as having no hemispatial neglect or having a sequelae, or a record recorded as having no hemispatial neglect or having a sequelae with the smallest angular difference. The control unit 11 determines whether the angular difference between the read record and the difference between the left angle calculated in S49 for the leftmost image identified in S50 and the right angle calculated in S49 for the rightmost image identified in S50 differs by a predetermined angle (e.g., 30 degrees) or more (S52). If the angular difference of the read record differs by more than a predetermined angle from the difference between the left angle in the leftmost image and the right angle in the rightmost image (S52: YES), the control unit 11 determines that the subject has hemispatial neglect (S53). The control unit 11 notifies, for example, the subject's doctor (S54), stores the left angle, right angle, angular difference, and the determination result (status) in the history table 121 along with the recording time and recording location (S55), and terminates the process. The control unit 11 may also notify the subject's family or send an emergency call. Furthermore, if the difference between the left angle in the leftmost image and the right angle in the rightmost image differs by more than a predetermined angle in S52, the control unit 11 may calculate the angular difference between the left angle of the read record and the left angle in the leftmost image (left angular difference), calculate the angular difference between the right angle of the read record and the right angle in the rightmost image (right angular difference), compare the left angular difference and the right angular difference, and determine that the space in the direction in which the angular difference is large is being ignored. In addition, in S54, the subject may be notified of the degree of hemispatial neglect, including the direction of the hemispatial neglect (left or right), or the level of severity of the symptom determined based on the difference between the read record and the left angle in the leftmost image and the right angle in the rightmost image, or the difference between the read record and the left angle in the leftmost image and the right angle in the rightmost image.If the angle difference of the read record and the difference between the left angle in the leftmost image and the right angle in the rightmost image do not differ by more than a predetermined angle (S52: NO), the control unit 11 stores the left angle, right angle, angle difference, and the judgment result (status) in the history table 121 together with the recording time and recording location (S55), and ends the processing. Note that the control unit 11 may also determine whether the subject has hemispatial neglect based on the difference or ratio between the left angle of the read record and the left angle in the leftmost image, or the difference or ratio between the right angle of the read record and the right angle in the rightmost image.

[0053] According to the configuration and processing of the third embodiment, even if a subject has a history of hemispatial neglect and has a difference in the range of motion between the left and right sides of the head due to sequelae, if the subject newly develops hemispatial neglect due to a stroke or the like, the control unit 11 of the information processing device 1 can determine that the subject has hemispatial neglect. In this embodiment, the control unit 11 stores left angles and right angles based on key points output by the posture estimation model M1, but this is not limited to this. The control unit 11 may store left ear movement distances and right ear movement distances measured based on the positions of feature points output by the feature point extraction model M2, and determine whether the subject has hemispatial neglect by comparing the stored left ear movement distances and right ear movement distances with the newly measured left ear movement distances and right ear movement distances.

[0054] (Fourth embodiment) A control unit 11 of an information processing device 1 according to a fourth embodiment causes an object to be displayed on a display device 4. The control unit 11 determines whether or not the subject has a brain dysfunction based on the difference between the movement range of the subject's head to the left and the movement range to the right relative to the object.

[0055] FIG. 13 is an explanatory diagram showing an example of the configuration of a brain dysfunction assessment system S according to embodiment 4. The brain dysfunction assessment system S according to embodiment 4 includes a display device 4. The display device 4 is, for example, a liquid crystal display. The display device 4 may be a television, a tablet terminal, a smartphone, a smart display, or a display unit of a personal computer. The display device 4 may also be integrated with the information processing device 1, the photographing device 2, or the audio output device 3. When determining whether or not a subject has a brain dysfunction (hemispatial neglect), the control unit 11 of the information processing device 1 causes the display device 4 to display a predetermined object. In the example shown in FIG. 13, the display device 4 displays an illustration of an apple as the object. The object is displayed, for example, at the center of the display device 4. In this embodiment, the center of the apple illustration is displayed so that it coincides with the center of the display device 4. The control unit 11 of the information processing device 1 determines whether or not the subject has hemispatial neglect based on the leftward and rightward rotation angles of the head relative to a line (reference line) connecting the object displayed on the display device 4 (the center of the display device 4) and the center of the subject's head.

[0056] 14 is an explanatory diagram showing an example of the configuration of an information processing device according to embodiment 4. A storage unit 12 of the information processing device 1 according to embodiment 4 stores a posture estimation model M1 and an object detection model M3. A communication unit 13 communicates with the display device 4 via a communication network N.

[0057] FIG. 15 is an explanatory diagram of a posture estimation model M1 and an object detection model M3 according to the fourth embodiment. The image input to the posture estimation model M1 and the object detection model M3 according to the fourth embodiment is, for example, an image captured by the photographing device 2 from above of the subject and the display device 4, as shown in FIG. 15. The configuration of the posture estimation model M1 according to this embodiment is the same as the configuration of the posture estimation model M1 according to the first embodiment. Note that when a subject is photographed from above, the posture estimation model M1 does not identify key points of the eyes and neck, but identifies key points of, for example, the nose and shoulders. Note that the posture estimation model M1 may also identify key points of the ears.

[0058] The object detection model M3 is a model that has a segmentation function such as CNN, RCNN, Fast RCNN, Faster RCNN, SSD, YOLO, or Vision Transformer and detects objects. When the object detection model M3 is configured as a neural network including a CNN that extracts image features such as RCNN, it has multiple neurons that accept input of pixel values ​​of the image and passes the input pixel values ​​to an intermediate layer. The intermediate layer has multiple neurons that extract image features of the image and passes the extracted image features to an output layer. The output layer identifies the position of the display device 4 included in the image based on the image features.

[0059] The output image in FIG. 15 shows the positions of the key points output by the posture estimation model M1, as well as the positions of the display device 4 and the object (the center of the display device 4) output by the object detection model M3. The control unit 11 of the information processing device 1 determines the midpoint between the key points on the subject's shoulders as the center point of the subject's head (the point through which the vertical axis of the subject's head passes), and determines the line connecting the center point of the subject's head and the center of the display device 4 (the position where the object is displayed) as the reference line. The control unit 11 also determines the direction from the center point of the subject's head toward the key point on the nose as the direction of the subject's head. In FIG. 15, the reference line is indicated by a dashed line with an arrow, and the head orientation is indicated by a solid line with an arrow. Note that the control unit 11 may also determine the midpoint between the key points on the subject's ears as the center point of the head.

[0060] 13 , the control unit 11 of the information processing device 1 causes the display device 4 to display, for example, an illustration of an apple, and causes the audio output device 3 to output audio instructions to the subject to rotate their head left and right until the apple (object) displayed on the display device 4 disappears from view. The image capturing device 2 continuously captures multiple images while the subject is rotating their head (for example, 10 seconds). The control unit 11 of the information processing device 1 identifies the reference line and the orientation of the head in each image.

[0061] The control unit 11 of the information processing device 1 also calculates the angle (left reference difference angle or right reference difference angle) of the head orientation relative to the reference line in each image. The control unit 11 identifies the reference image, the leftmost image, and the rightmost image based on the left reference difference angle or right reference difference angle in each image. The reference image in this embodiment is, for example, the image first captured by the imaging device 2 after the control unit 11 of the information processing device 1 starts determining whether or not there is a brain dysfunction. The leftmost image is the image in which the subject's head rotates most to the left among multiple images in the time series. The rightmost image is the image in which the subject's head rotates most to the right among multiple images in the time series. The control unit 11 determines whether or not the subject has hemispatial neglect based on the left reference difference angle when the subject is facing most to the left and the right reference difference angle when the subject is facing most to the right.

[0062] Fig. 16 is a schematic diagram illustrating the rotational movement of the subject's head according to embodiment 4. Fig. 16 shows a reference image, a leftmost image, and a rightmost image for a case in which the subject is healthy and a case in which the subject has left-sided hemispatial neglect. In Fig. 16, the reference line is indicated by a dashed line with an arrow, and the head direction is indicated by a solid line with an arrow.

[0063] If a subject has left-sided hemispatial neglect, the subject can simply rotate their head slightly to the right to make the object invisible (unrecognizable). As a result, as shown in the leftmost and rightmost images of a case in which hemispatial neglect is present in Figure 16 , when the subject rotates their head until the object disappears, the angle of rotation to the right (right reference difference angle) becomes smaller than the angle of rotation to the left (left reference difference angle). The control unit 11 of the information processing device 1 determines that the subject has hemispatial neglect when the difference between the left reference difference angle and the right reference difference angle is equal to or greater than a predetermined angle (e.g., 30 degrees). The ignored space is the space opposite the smaller of the left reference angle difference and the right reference angle difference.

[0064] In this embodiment, if the relative positions of the image capture device 2 and the display device 4 installed in the subject's room are fixed, the position of the object displayed on the display device 4 in the image remains constant. Therefore, the control unit 11 may receive a designation of the object's position in the image in advance and identify the reference line based on the designated position. In this embodiment, the image is an image of the subject captured from above by the image capture device 2. However, this is not limited to this. For example, the image may include the subject's head and be captured by an image capture device 2 installed above the display device 4. In this case, the reference line may be a horizontal line connecting a point on the vertical axis of the subject's head and the capture position. Furthermore, the control unit 11 of the information processing device 1 may receive a voice input from the subject indicating that they have completed their head rotation, and identify the leftmost or rightmost image based on the received voice. Alternatively, if the subject remains stationary for a certain period of time or longer, the control unit 11 may identify any image captured while the subject remains stationary as the leftmost or rightmost image.

[0065] 17 is a flowchart showing an example of processing by the control unit 11 of the information processing device 1 according to the fourth embodiment. The control unit 11 causes the audio output device 3 to output a voice instruction to the subject to move to a position where the center of the screen of the display device 4 is visible (S61). Note that if the control unit 11 has previously detected that the subject is located in a position where the center of the screen of the display device 4 is visible, the processing of S61 does not need to be executed. Here, as long as the subject can see the center of the screen of the display device 4, the subject does not need to be located directly in front of the display device 4 or in a position where an object can be displayed on the display device 4 so as to face the subject (hereinafter, a position where the subject faces the display device 4 directly and a position where an object can be displayed on the display device 4 so as to face the subject may be collectively referred to as a "position where a direct display is possible"). After the subject has positioned (moved) to a position where the center of the display device 4 is visible, the control unit 11 causes the display device 4 to display an object (S62). The control unit 11 then outputs a voice instruction to the subject to rotate his or her head left and right until the object is no longer visible (S63). The control unit 11 causes the image capturing device 2 to start capturing images of the subject (S64). When the image capturing starts, the image capturing device 2 captures consecutive images (moving images) for, for example, 10 seconds, and transmits the captured consecutive images to the information processing device 1. The control unit 11 of the information processing device 1 acquires the consecutive images from the image capturing device 2 (S65). Note that the position where the image can be displayed facing forward is, for example, within 10% of the periphery from the center of the screen of the display device 4.

[0066] The control unit 11 of the information processing device 1 inputs multiple images acquired from the image capture device 2 into the posture estimation model M1 (S66) and outputs the positions of the subject's key points and bones in each image (S67). The control unit 11 also inputs multiple images acquired from the image capture device 2 into the object detection model M3 (S68) and outputs the position of the object (the center of the display device 4) in each image (S69). The control unit 11 identifies the center point of the subject's head based on the positions of the output key points (S70). The control unit 11 identifies a reference line in each image based on the positions of the center point and the object (S71). The control unit 11 identifies the head orientation based on the positions of the output key points (S72). The control unit 11 calculates a left reference difference angle or a right reference difference angle in each image based on the reference line and the head orientation (S73).

[0067] The control unit 11 of the information processing device 1 identifies the image with the largest left reference difference angle (the leftmost image) and the image with the largest right reference difference angle (the rightmost image) (S74). The control unit 11 determines whether the difference between the left reference difference angle and the right reference difference angle is equal to or greater than a predetermined angle (e.g., 30 degrees) (S75). If the difference between the left reference difference angle of the leftmost image and the right reference difference angle of the rightmost image is equal to or greater than the predetermined angle (S75: YES), the control unit 11 determines that the subject has hemispatial neglect (S76), notifies the subject's doctor, for example (S77), and terminates the processing. The control unit 11 may also notify the subject's family or call an ambulance. In S77, the control unit 11 may notify the subject of the direction of the hemispatial neglect (left or right), or the degree of hemispatial neglect, including the level of severity of the symptoms determined based on the difference between the left reference difference angle of the leftmost image and the right reference difference angle of the rightmost image, or the difference between the left reference difference angle of the leftmost image and the right reference difference angle of the rightmost image. If the difference between the left reference difference angle of the leftmost image and the right reference difference angle of the rightmost image is less than a predetermined angle (S75: NO), the control unit 11 terminates the processing. The control unit 11 may also determine whether the subject has hemispatial neglect based on the ratio between the left reference difference angle of the leftmost image and the right reference difference angle of the rightmost image. In this embodiment, the control unit 11 determines hemispatial neglect based on the difference between the left reference difference angle of the leftmost image and the right reference difference angle of the rightmost image, but this is not limited to this. The control unit 11 may read a previously recorded record of the left reference difference angle of the leftmost image and the right reference difference angle of the rightmost image, and determine that right hemispatial neglect exists if the difference between the left reference difference angle of the leftmost image in the read record and the measured left reference difference angle of the leftmost image is equal to or greater than a predetermined angle (e.g., 30 degrees). Alternatively, the control unit 11 may determine that left hemispatial neglect exists if the difference between the right reference difference angle of the rightmost image in the read record and the measured right reference difference angle of the rightmost image is equal to or greater than a predetermined angle (e.g., 30 degrees).

[0068] According to the configuration and processing of the fourth embodiment, the information processing device 1 determines whether or not the subject has hemispatial neglect based on the orientation of the subject's head relative to an object in an image of the subject captured by the imaging device 2. This makes it possible to determine with higher accuracy, simply, and in a short time whether or not the subject has a cerebral dysfunction, and makes it possible to deal with, for example, acute cerebral artery occlusion promptly and appropriately.

[0069] (Embodiment 5) The control unit 11 of an information processing device 1 according to embodiment 5 acquires an image of the subject's surroundings from the imaging device 2. The control unit 11 identifies an object from objects included in the acquired image. The control unit 11 determines whether the subject has a brain dysfunction based on the difference between the range of movement of the subject's head to the left and the range of movement to the right relative to the object. The storage unit 12 of the information processing device 1 according to embodiment 5 stores an object detection model M3. The brain dysfunction assessment system S according to this embodiment does not include a display device 4 (see FIG. 1), but the storage unit 12 of the information processing device 1 according to this embodiment stores a posture estimation model M1 and an object detection model M3, similar to the storage unit 12 according to embodiment 4 (see FIG. 14).

[0070] 18 is an explanatory diagram of a posture estimation model M1 and an object detection model M3 according to embodiment 5. The configurations of the posture estimation model M1 and the object detection model M3 according to this embodiment are the same as the configurations of the posture estimation model M1 and the object detection model M3 according to embodiment 4. Furthermore, the images input to the posture estimation model M1 and the object detection model M3 in this embodiment are images of a subject and an object photographed from above, as in embodiment 5. Note that the output layer of the object detection model M3 according to embodiment 5 identifies the position of an object included in an image based on image features.

[0071] The control unit 11 of the information processing device 1 identifies an object detected by the object detection model M3 in the image as an object. In the example of the output image of FIG. 18 , the control unit 11 identifies a chair detected by the object detection model M3 in the image as an object. Note that if the object detection model M3 detects multiple objects in the image, the control unit 11 identifies, for example, the object that is most central in the horizontal direction of the image as the object. The control unit 11 identifies the vertical axis of the subject's head based on the positions of the keypoints output by the posture estimation model M1, and outputs a line (reference line) connecting the identified vertical axis and the object on the image. In the example of the output image of FIG. 18 , the control unit 11 uses the line connecting the center point of the subject's head (the point through which the vertical axis of the subject's head passes) and the center point of the chair closest to the subject in the horizontal direction as the reference line. Furthermore, the control unit 11 identifies the head orientation (the direction toward the subject's nose relative to the vertical axis) based on the positions of the keypoints output by the posture estimation model M1. In the output image of Figure 18, the line indicating the reference line is indicated by a dashed line with an arrow, and the line indicating the head orientation is indicated by a solid line with an arrow. The control unit 11, using the object detected by the object detection model M3 as the object, identifies the reference line and the head orientation, and calculates the angle of the head orientation relative to the reference line to the left or right (left reference difference angle or right reference difference angle) in the same manner as in embodiment 4 (see Figure 16). The control unit 11 inputs a plurality of time-series images captured continuously by the image capture device 2 into the posture estimation model M1 and the object detection model M3, and identifies the reference line and the head orientation for each image. The control unit 11 determines whether the subject has hemispatial neglect based on the left reference difference angle when the subject is facing farthest left and the right reference difference angle when the subject is facing farthest right.

[0072] FIG. 19 is a flowchart showing an example of processing by the control unit 11 of the information processing device 1 according to the fifth embodiment. The control unit 11 causes the audio output device 3 to output a voice instruction to the subject to move into the shooting range of the camera device 2 (S81). Note that if the control unit 11 has previously detected that the subject is within the shooting range of the camera device 2, the processing of S81 may not be executed. After the subject positions (moves) within the shooting range of the camera device 2, the control unit 11 acquires an image from the camera device 2 (S82). The control unit 11 inputs the acquired image into the object detection model M3 (S83) and outputs the position of an object contained in the image (S84). Of the objects detected by the object detection model M3, the control unit 11 identifies, for example, the object closest to the center in the horizontal direction of the image as the object (S85). The control unit 11 of the information processing device 1 then outputs a voice instruction to the subject to rotate their head left and right until the identified object disappears (S86). The processing from S87 to S100 is the same as the processing from S64 to S77 shown in FIG.

[0073] According to the configuration and processing of the fifth embodiment, the information processing device 1 determines whether or not the subject has hemispatial neglect based on the orientation of the subject's head relative to an object identified in an image of the subject captured by the image capturing device 2. As a result, even if there is no display device that displays an object within the image capturing range of the image capturing device 2 or no display device that displays an object within a distance visible to the subject, as long as the subject is within the image capturing range of the image capturing device 2, the information processing device 1 can easily and quickly determine whether or not the subject has a cerebral dysfunction with higher accuracy, thereby enabling, for example, prompt and appropriate treatment for acute cerebral artery occlusion.

[0074] (Embodiment 6) The control unit 11 of the information processing device 1 according to embodiment 6 determines a method for assessing brain dysfunction (hemispatial neglect) based on the positional relationship between the display device 4 and the subject in an image captured by the image capture device 2. When neither the display device 4 nor a potential object is in a position visible to the subject, when the display device 4 is not within a distance visible to the subject, or when the subject cannot view the center of the screen of the display device 4, the control unit 11 determines whether the subject has hemispatial neglect using the first assessment method described in embodiment 1. That is, the control unit 11 determines whether the subject has hemispatial neglect based on the difference between the leftward and rightward movement ranges of the head relative to the vertical axis of the head. When the display device 4 is in a position visible to the subject in an image captured by the image capture device 2, and the subject is located in a position where an object cannot be displayed on the display device 4 so as to face the subject, or when a potential object is in a position visible to the subject, the control unit 11 determines whether the subject has hemispatial neglect using the second assessment method described in embodiment 4 or 5. That is, the control unit 11 determines whether the subject has hemispatial neglect based on the difference between the range of head movement to the left and the range of head movement to the right relative to a line connecting the object and the vertical axis of the subject's head. Furthermore, when the subject faces the display device 4 directly, or when the subject is positioned so that the object can be displayed on the display device 4 facing the subject, the control unit 11 performs a four-finger test to determine whether the subject has hemispatial neglect based on the results of displaying four fingers on the display device 4 and asking the subject how many fingers they can see. The four-finger test serves as a third determination method for determining whether the subject has hemispatial neglect. In addition to the four-finger test, the third determination method may also be a known hemispatial neglect test, such as the standard tests of the BIT behavioral neglect test (line cancellation test, letter cancellation test, star cancellation test, imitation test, line bisection test, and drawing test).

[0075] FIG. 20 is a flowchart showing an example of processing by the control unit 11 of the information processing device 1 according to the sixth embodiment. The control unit 11 of the information processing device 1 causes the audio output device 3 to output audio instructions to the subject to move into the imaging range of the image capture device 2 (S101). Note that if the control unit 11 has previously detected that the subject is within the imaging range of the image capture device 2, the processing of S101 may not be executed. After the subject positions (moves) into the imaging range of the image capture device 2, the control unit 11 acquires captured images from the image capture device 2 (S102). The control unit 11 inputs the acquired images into the posture estimation model M1 (S103) and outputs the positions of the subject's key points and bones in each image (S104). The control unit 11 also inputs the images acquired from the image capture device 2 into the object detection model M3 (S105) and outputs the positions of objects, including the display device 4 and objects other than the display device 4 (S106). The control unit 11 of the information processing device 1 determines whether the display device 4 is included in the image (S107). Note that, if the image capturing device 2 and the display device 4 are configured as an integrated unit and the control unit 11 can estimate the position coordinates of the display device 4 (for example, if the position coordinates of the display device 4 can be estimated using GPS or radio waves, if the position coordinates of the display device 4 can be estimated by exchanging information between multiple sensor devices, or if the position coordinates of the display device 4 can be estimated from the positional relationship between the subject and the display device 4 based on the known relative relationship between the center position of the display device 4 and the center position of the image capturing device 2 and the relative relationship between the image capturing device 2 and the subject), steps S105 to S106 may be skipped. In this case, the control unit 11 of the information processing device 1 determines that the display device 4 is included (S107: YES).

[0076] If the display device 4 is not included in the image (S107: NO), the control unit 11 determines whether an object other than the display device 4 that could be the object is located in a position visible to the subject (S108). In S108, the control unit 11 estimates the subject's field of view assuming the subject is healthy (without brain dysfunction) from the positions of the subject's key points, and determines whether an object that could be the object is located in a position visible to the subject based on the estimated field of view and the position of the object. If an object that could be the object is located in a position visible to the subject (S108: YES), the control unit 11 identifies the object located in a position visible to the subject as the object (S109), determines whether the subject has hemispatial neglect using a second determination method (S110), and terminates the process. Note that the second determination method of S110 executes the same processes as S82 to S100 shown in FIG. 19 of the fifth embodiment. If there is no object that can be the object in a position visible to the subject (S108: NO), the control unit 11 determines whether the subject has hemispatial neglect using the first determination method (S111), and ends the process. That is, in S111, the same processes as S2 to S13 shown in FIG. 5 are executed.

[0077] If the display device 4 is included in the image (S107: YES), the control unit 11 measures the distance and angle of the subject relative to the display device 4 based on the positions of the subject's key points and the position of the display device 4 (S112). In this embodiment, the distance of the subject relative to the display device 4 is, for example, the distance between the subject's vertical axis and the center of the screen of the display device 4. The angle of the subject relative to the display device 4 is, for example, the angle from the center of the screen of the display device 4 toward the subject's vertical axis when viewing the display device 4 and the subject from above, and is measured with the left side of the screen as 0 degrees, the front of the screen as 90 degrees, the right side of the screen as 180 degrees, and the back of the screen as 270 degrees. The control unit 11 determines whether the measured distance is less than a predetermined distance (the distance at which the subject can see the display device 4, for example, 2 m) (S113). If the measured distance is equal to or greater than the predetermined distance (S113: NO), the control unit 11 proceeds to S111. If the measured distance is less than the predetermined distance (S113; YES), the control unit 11 determines whether the subject is located directly in front of the display device 4 based on the measured angle (S114). In S114, the control unit 11 determines that the subject is located directly in front of the display device 4, for example, if the angle of the subject relative to the display device 4 is between 85 degrees and 95 degrees. If the subject is not located directly in front of the display device 4 (S114: NO), the control unit 11 determines whether the subject can view the center of the display device 4 based on the positions of the subject's key points and the position of the display device 4 (S115). In S115, the control unit 11 determines that the subject can view the center of the display device 4, for example, if the angle of the subject relative to the display device 4 is between 30 degrees and 150 degrees. If the subject cannot view the center of the display device 4 (S115: NO), the control unit 11 proceeds to S111. If the subject can see the center of the display device 4 (S115: YES), the control unit 11 determines whether the subject has hemispatial neglect using the second determination method (S116), and ends the process. Note that in the second determination method of S116, the same processes as S62 to S77 shown in FIG. 17 of the fourth embodiment are executed. If the subject is positioned directly in front of the display device 4 (S114: YES), the control unit 11 performs a four-finger test (S117), and ends the process.

[0078] That is, the control unit 11 of the information processing device 1 determines whether the subject can see the display device 4 based on the positional relationship (distance and angle) between the display device 4 (object) and the subject, and if it is determined that the subject cannot see the display device 4, it determines whether the subject has hemispatial neglect (cerebral dysfunction) using the first determination method, if it is determined that the subject can see the display device 4 but is not located directly in front of the display device 4, it determines whether the subject has hemispatial neglect using the second determination method, and if it is determined that the subject can see the display device 4 and is located directly in front of the display device 4, it determines whether the subject has hemispatial neglect using the third determination method using a four-finger test.

[0079] According to the configuration and processing of the sixth embodiment, the control unit 11 of the information processing device 1 changes the method for determining whether or not the subject has a brain dysfunction (hemispatial neglect) based on the presence or absence or position of an object included in an image. This enables the control unit 11 to determine whether or not the subject has a brain dysfunction (hemispatial neglect) using a suitable determination method depending on the situation.

[0080] (Embodiment 7) The control unit 11 of the information processing device 1 according to embodiment 7 uses a different method for determining whether the subject has hemispatial neglect (cerebral dysfunction) based on the positional relationship between the object and the subject. Furthermore, the control unit 11 determines that the subject has hemispatial neglect (cerebral dysfunction) when the subject's perceived position relative to the object does not match the subject's actual position. The cerebral dysfunction determination system S according to embodiment 7 includes a display device 4 (see FIG. 13). Furthermore, the storage unit 12 of the information processing device 1 according to embodiment 7 stores a posture estimation model M1 and an object detection model M3 (see FIG. 14). Note that if the object detection model M3 detects an object in a position visible to the subject, the cerebral dysfunction determination system S does not necessarily include the display device 4. In this case, the control unit 11 determines that the subject has hemispatial neglect (cerebral dysfunction) when the subject's perceived position relative to the object detected by the object detection model M3 does not match the subject's actual position.

[0081] 21 is an explanatory diagram of a posture estimation model M1 and an object detection model M3 according to embodiment 7. The configurations of the posture estimation model M1 and the object detection model M3 according to this embodiment are the same as the configurations of the posture estimation model M1 and the object detection model M3 according to embodiment 4. Note that the object detection model M3 according to embodiment 7 detects the display device 4 included in the input image as an object, but the object to be detected is not limited to the display device 4 and may be, for example, an object other than the display device 4, such as a chair, detected in the image by the object detection model M3.

[0082] The control unit 11 of the information processing device 1 causes the audio output device 3 to output an instruction to move in front of the object (display device 4) relative to the subject. After the subject moves to the recognized position based on the instruction, the information processing device 1 acquires an image captured by the imaging device 2. The control unit 11 inputs the acquired image to the posture estimation model M1 and the object detection model M3. The control unit 11 outputs the key points identified by the posture estimation model M1 and the position of the object (display device 4) detected by the object detection model M3 on the same image. Based on the output key points and the position of the object (display device 4), the control unit 11 determines whether the position of the subject relative to the object recognized by the subject matches the actual position of the subject relative to the object.

[0083] When the subject hears an instruction to move in front of the object (display device 4), the subject moves to a position that the subject perceives as being in front of the object (display device 4). The position in front of the object (display device 4) is a position where the midline of the object (display device 4) (a line passing through the center of the object (display device 4) and perpendicular to the object (display device 4)) and the subject's midline (a line connecting the point (center point) through which the vertical axis of the subject's head passes and the key point (nose)) approximately coincide. If the subject does not have hemispatial neglect, the subject can move in front of the object (display device 4). That is, the subject's perceived position relative to the object coincides with the subject's actual position. If the subject has hemispatial neglect, the subject moves to a position on the opposite side of the ignored space in the left-right direction, rather than in front of the object (display device 4). That is, the subject's perceived position relative to the object does not coincide with the subject's actual position. The control unit 11 of the information processing device 1 determines whether the subject is located in front of the object (display device 4) based on the subject's key points and the positions of the object (display device 4). The control unit 11 determines whether the subject is located in front of the object (display device 4) and thereby determines whether the position of the subject relative to the object as perceived by the subject matches the position of the subject relative to the actual object. In the output image of FIG. 21 , the midline of the display device 4 and the midline of the subject are indicated by dashed lines. In the example shown in FIG. 21 , the midline of the display device 4 and the midline of the subject do not match, and the subject is not in front of the display device 4 but is located to the left of the center of the display device 4. Therefore, the control unit 11 determines that the position of the subject relative to the object as perceived by the subject does not match the position of the subject relative to the actual object.

[0084] 22 is a flowchart showing an example of processing by the control unit 11 of the information processing device 1 according to the seventh embodiment. The control unit 11 of the information processing device 1 causes the audio output device 3 to output an audio instruction to the subject to move in front of the display device 4 (S121). After the subject moves to the recognized position based on the instruction, the control unit 11 acquires captured images from the image capture device 2 (S122). The control unit 11 inputs the images acquired from the image capture device 2 into the posture estimation model M1 (S123) and outputs the positions of the subject's key points and bones in each image (S124). The control unit 11 also inputs the images acquired from the image capture device 2 into the object detection model M3 (S125) and outputs the position of the display device 4 (S126).

[0085] The control unit 11 of the information processing device 1 determines whether the position of the subject relative to the display device 4 (object) recognized by the subject matches the actual position of the subject relative to the display device 4 (object) based on the positions of the output key points and the position of the display device 4 (S127). In S127, if the subject is located in front of the display device 4 (the midline of the display device 4 matches the midline of the subject), the control unit 11 determines that the position of the subject relative to the display device 4 (object) recognized by the subject matches the actual position of the subject relative to the display device 4 (object), and if the subject is not located in front of the display device 4 (the midline of the display device 4 does not match the midline of the subject), the control unit 11 determines that the position of the subject relative to the display device 4 (object) recognized by the subject does not match the actual position of the subject relative to the display device 4 (object). If the subject's perceived position relative to the display device 4 (object) does not match the subject's actual position relative to the display device 4 (object) (S127: NO), the control unit 11 determines that the subject has hemispatial neglect (S128), and notifies, for example, the subject's attending physician, that the subject has hemispatial neglect (S129), and ends the process. If the subject's perceived position relative to the display device 4 (object) matches the subject's actual position relative to the display device 4 (object) (S127: YES), the control unit 11 proceeds to S130. The processes of S130 to S143 are the same as the processes of S62 to S75 shown in FIG. 17.

[0086] According to the configuration and processing of the seventh embodiment, when the position of the subject facing the object (display device 4) that the subject recognizes differs from the actual position of the subject facing the object (display device 4), the control unit 11 of the information processing device 1 can detect that the subject is ignoring one side of the space in the left-right direction and determine that the subject has a cerebral dysfunction (hemispatial neglect). This makes it possible to more simply determine whether the subject has a cerebral dysfunction, and enables prompt and accurate treatment of, for example, acute cerebral artery occlusion.

[0087] The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The technical features described in each embodiment may be combined with one another, and the scope of the present invention is intended to include all modifications within the scope of the claims and equivalents thereto. Furthermore, independent and dependent claims described in the claims may be combined with one another in any and all combinations, regardless of the reference format. Furthermore, while the claims use a format in which a claim references two or more other claims (multiple claim format), this is not limiting. Multiple claims (multiple multiple claims) that reference at least one other claim may also be used.

[0088] REFERENCE SIGNS LIST 1 Information processing device 11 Control unit 12 Storage unit 12a Recording medium 121 History table M1 Posture estimation model M2 Feature point extraction model M3 Object detection model 13 Communication unit 2 Imaging device 3 Audio output device 4 Display device N Communication network P Computer program S Brain dysfunction assessment system

Claims

1. A program that causes a computer to execute a process of detecting a subject's head rotation movement and determining whether the subject has a brain dysfunction based on the range of movement of the head rotation movement.

2. The program according to claim 1, which detects rotational movement of the subject's head about a vertical axis, and determines whether the subject has a symptom of a lack of visual information on at least one of the left and right sides based on the difference between the range of movement of the head to the left and the range of movement to the right.

3. The program according to claim 1 or 2, which outputs instructions to the subject to move the head in left and right directions, and determines whether the subject has hemispatial neglect based on the rotational movement detected after the output of the instructions.

4. The program according to claim 1 or 2, which acquires a plurality of time-series images including the subject's head, extracts feature points of the subject from the acquired plurality of images, and determines whether the subject has a brain dysfunction based on the difference in the positions of the feature points between the extracted plurality of images.

5. The program according to claim 1 or 2, which detects rotational movement of the subject's head about a vertical axis, and determines whether the subject has a brain dysfunction based on the difference between the range of movement of the head to the left and the range of movement to the right.

6. The program according to claim 5, wherein the subject is determined to have a brain dysfunction if the difference between the angle of rotation of the head to the left and the angle of rotation to the right is equal to or greater than a predetermined angle.

7. The program according to claim 1 or 2, which stores the subject's past range of head movement, and determines that the subject has a brain dysfunction if the difference between the stored range of movement and the range of movement of the detected rotational movement is equal to or greater than a predetermined value.

8. The program according to claim 1 or 2, which determines whether the subject has a brain dysfunction based on the difference between the range of movement of the head to the left and the range of movement to the right in relation to a specified object.

9. The program according to claim 8, which displays the object on a display device.

10. The program according to claim 8, which acquires an image of the subject's surroundings, and identifies the object from objects contained in the acquired image.

11. A program as described in claim 1 or 2, which determines whether or not the subject has a brain dysfunction based on the positional relationship between the object and the subject, using either a first determination method based on the difference between the range of movement of the subject's head to the left and the range of movement to the right when the subject's head is rotated about a vertical axis, or a second determination method based on the difference between the range of movement of the subject's head to the left and the range of movement to the right when the subject's head is rotated about a specified object.

12. The program described in claim 11, which determines whether the subject can see the object based on the positional relationship between the object and the subject, determines whether the subject has a brain dysfunction using the first determination method if it is determined that the subject cannot see the object, and determines whether the subject has a brain dysfunction using the second determination method if it is determined that the subject can see the object.

13. A program as described in claim 1 or 2, which acquires the position of the subject relative to an object as perceived by the subject, acquires the actual position of the subject relative to the object, determines whether the position perceived by the subject matches the actual position, determines that the subject has a brain dysfunction if they do not match, and determines whether the subject has a brain dysfunction based on the range of motion of the head rotation movement if they match.

14. An information processing method that detects head rotational movement of a subject and determines whether the subject has a brain dysfunction based on the range of movement of the head rotational movement.

15. An information processing device comprising a control unit that detects rotational movement of the subject's head and determines whether the subject has a brain dysfunction based on the range of movement of the rotational movement of the head.

Images