Motion image correction device, method, and program

The motion image correction device addresses the challenge of asymmetrical motion image recognition in dementia patients by symmetrizing the image with the imitator's orientation and correcting hand structure, enhancing the effectiveness of imitation by ensuring the image is recognizable and imitable.

WO2024247032A1PCT designated stage expired Publication Date: 2024-12-05NT T INC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/019882
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-29
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Individuals with dementia, particularly those exhibiting symptoms of apraxia or visual agnosia, face difficulties in imitating eating behaviors due to asymmetrical orientation and posture of projected motion images, making it hard for them to recognize body parts, which hinders effective imitation.

Method used

A motion image correction device and method that detects the imitator's orientation and visual agnosia characteristics to symmetrize the motion image with the display surface and correct the hand skeletal structure, ensuring the projected motion image is recognizable and imitable by aligning the image with the imitator's perspective and posture.

Benefits of technology

The solution effectively induces imitation of actions in individuals with dementia by ensuring the motion image is displayed in a symmetrical and recognizable manner, enhancing the imitator's ability to recognize and replicate the eating behavior.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023019882_05122024_PF_FP_ABST
    Figure JP2023019882_05122024_PF_FP_ABST
Patent Text Reader

Abstract

This motion image correction device according to one embodiment comprises: a detection unit that detects the direction of a follower with respect to a display screen of a motion image and detects the direction in which the motion image will be symmetrical to the detected direction with respect to the display screen of the motion image; a first correction unit that calculates a correction of the direction of the motion image detected by the detection unit in a direction in which the follower can visually recognize a palm of the motion image on the basis of a visual loss characteristic of the follower; and a second correction unit that calculates a correction of a skeleton structure of a hand of the motion image on the basis of the visual loss characteristic of the follower.
Need to check novelty before this filing date? Find Prior Art

Description

Motion image correction device, method, and program

[0001] FIELD Embodiments of the present invention relate to a motion image correction apparatus, method, and program.

[0002] In generating moving images, a method has been proposed for a moving image generating device that generates realistic moving images by applying changes to an image area that is extracted from a still image and that is to be moved (see, for example, Patent Document 1).

[0003] Japanese Patent Publication No. 2000-67259

[0004] Conventionally, for people with dementia who exhibit apraxia or agnosia, and other people who have difficulty initiating the act of eating, i.e., the act of bringing food to their mouths, caregivers have assisted them with the eating action, or they have been shown the eating action of others and asked to imitate them.

[0005] By using a motor image generating device such as that described above, it is possible to project an image of an eating action as an imitated motor image onto a screen or the like and have the person with dementia imitate it. However, in order to effectively induce the person with dementia to imitate eating actions, it is necessary to adjust the direction in which the motor image is displayed and the angle at which the body part is displayed so that it is symmetrical to the imitator, and furthermore, to correct the motor image based on the visual agnosia characteristics of the imitator.

[0006] First, imitation is difficult if the orientation and posture of the body parts in the movement image are asymmetrical with those of the imitation, so the orientation and posture of the body parts in the movement shown to the person with dementia must be adjusted to match the orientation and posture of the imitation's body parts.

[0007] On the other hand, people with dementia exhibit symptoms of visual agnosia, making it difficult for them to recognize the body parts of the person being impersonated depending on the angle and posture from which they observe the body parts. Therefore, it is necessary to always present the body parts of the person being impersonated in an orientation and posture that makes them easy to recognize.

[0008] In other words, if the orientation and posture of the body parts of the imitated person's movements are matched to the orientation and posture of the imitator's body parts, it may be difficult for the person with dementia to recognize the projected body parts at that observation angle and posture.

[0009] The present invention has been made in light of the above-mentioned circumstances, and an object of the present invention is to provide a movement image correcting apparatus, method, and program that can effectively induce an imitator to imitate a movement.

[0010] A motion image correction device according to one aspect of the present invention includes a detection unit that detects the orientation of an imitation with respect to a display surface of a motion image and detects an orientation of the motion image with respect to the display surface of the motion image that is symmetrical to the detected orientation; a first correction unit that calculates a correction of the orientation of the motion image detected by the detection unit to an orientation in which the palm of the motion image can be seen by the imitation, based on the visual agnosia characteristics of the imitation; and a second correction unit that calculates a correction of the skeletal structure of the hand of the motion image, based on the visual agnosia characteristics of the imitation.

[0011] A motion image correcting method according to one aspect of the present invention is a method performed by a motion image correcting device, and includes: detecting, by a detection unit of the motion image correcting device, an orientation of an imitation with respect to a display surface of a motion image; and detecting an orientation of the motion image with respect to the display surface of the motion image that is symmetrical to the detected orientation; calculating, by a first correction unit of the motion image correcting device, a correction to the orientation of the motion image detected by the detection unit to an orientation in which the palm of the motion image can be seen by the imitation, based on the visual agnosia characteristics of the imitation; and calculating, by a second correction unit of the motion image correcting device, a correction to the skeletal structure of the hand of the motion image, based on the visual agnosia characteristics of the imitation.

[0012] According to the present invention, it is possible to effectively induce an imitator to imitate an action.

[0013] FIG. 1 is a diagram showing an example of the relationship between the direction of a motion image and the direction of an imitation. FIG. 2 is a diagram showing an example of an application of a motion image display system according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of the relationship between each unit of the motion image display system and an imitation. FIG. 4 is a diagram showing an example of the processing flow by each unit of a correction device. FIG. 5 is a flowchart showing an example of the procedure of a processing operation by an imitation direction detection unit of the correction device. FIG. 6 is a diagram showing an example of the direction of a motion image and the direction of an imitation relative to the display surface of the motion image. FIG. 7 is a flowchart showing an example of the procedure of a processing operation by a visual agnosia characteristic input unit of the correction device. FIG. 8 is a diagram showing an example of the definition of the palm angle. FIG. 9 is a diagram showing an example of the test results of the hand angle and the recognition accuracy for the front and back of the hand. FIG. 10 is a diagram showing an example of the test results of an angle having a predetermined difference with respect to the angle recognition accuracy. FIG. 11 is a diagram showing an example of the difference in recognition accuracy at a predetermined angle. FIG. 12 is a flowchart showing an example of the procedure of a processing operation by a motion image direction correction unit. FIG. 13 is a diagram showing an example of the direction of a motion image before and after correction by the motion image direction correction unit. FIG. 14 is a flowchart showing an example of the procedure of a processing operation by a hand skeletal structure correction unit. Fig. 15 is a diagram showing an example of the correction content by the hand skeletal structure correction unit. Fig. 16 is a block diagram showing an example of the hardware configuration of a correction device according to an embodiment of the present invention.

[0014] An embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is a diagram showing an example of the relationship between the direction of a movement image and the direction of an imitation. As shown in Fig. 1, in order to effectively induce imitation of a movement image (symbol a in Fig. 1) by an imitation person (symbol b in Fig. 1) such as a person with dementia, it is effective to make the direction of the movement image (symbol a1 in Fig. 1), in this case the direction of the hand, symmetrical with the direction of the imitation person's hand (symbol b1 in Fig. 1) when viewed from the front.

[0015] On the other hand, in order to allow an imitation to recognize the hand of the motor image, it is effective to set the orientation of the motor image when viewed from the front (symbol a2 in FIG. 1 ) to a palm orientation at an angle of 0 degrees. However, in this case, the orientation of the motor image when viewed from the front (symbol a2 in FIG. 1 ) is no longer symmetrical with the orientation of the imitation's hand when viewed from the front (symbol b2 in FIG. 1 ). This relationship contradicts the relationship for effectively inducing imitation of the motor image by the imitation (symbol c in FIG. 1 ). This embodiment aims to resolve this contradiction. In this embodiment, when showing a video of an eating behavior to induce an eating behavior in a person with dementia or the like, the motor image is displayed so that the sagittal horizontal axis of the movement relative to the display screen of the motor image and the sagittal horizontal axis of the imitation relative to the display screen of the motor image are symmetrical, thereby effectively inducing imitation in a person with dementia or the like.

[0016] Fig. 2 is a diagram showing an application example of a motion image display system according to one embodiment of the present invention. As shown in Fig. 2, the motion image display system according to one embodiment of the present invention includes a correction device (motion image correction device) 100, a motion image generation device 200, and a motion image presentation unit 300. The motion image generation device 200 and the motion image presentation unit 300 may be configured using existing technology.

[0017] The correction device 100 also has an imitation orientation detection unit 110, a visual agnosia characteristic input unit (calculation unit) 120, a movement image orientation correction unit (first correction unit) 130, and a hand skeletal structure correction unit (second correction unit) 140.

[0018] FIG. 3 is a diagram showing an example of the relationship between each unit of the movement image display system and the imitation. In this embodiment, when generating an imitated movement image, the direction in which the movement image is displayed is determined so that the sagittal horizontal axis of the movement relative to the screen displaying the movement image in the movement image presentation unit 300 is symmetrical with the sagittal horizontal axis (symbol b in FIG. 3 ) of the imitation (symbol a in FIG. 3 ) relative to the screen displaying the movement image. Symbol c in FIG. 3 indicates the direction of the imitation relative to the screen displaying the movement image. Furthermore, in this embodiment, the direction in which the movement image is displayed and the skeletal structure of the hand of the movement image are corrected according to the characteristics of the imitation's visual agnosia (symbol d in FIG. 3 ).

[0019] In this way, in this embodiment, the direction in which the projected movement image is displayed and the angle at which the hand of the movement image is displayed are corrected according to the characteristics of the visual agnosia of the target dementia patient, thereby making it possible to display a movement image that is easy for dementia patients with visual agnosia to imitate.

[0020] FIG. 4 is a diagram showing an example of the processing flow by each unit of the correction device. Details of the processing flow shown in FIG. 4 will be described below. FIG. 5 is a flowchart showing an example of the processing procedure by the imitation orientation detection unit of the correction device. FIG. 6 is a diagram showing an example of the orientation of the motion image and the orientation of the imitation relative to the display surface of the motion image. The imitation orientation detection unit 110 of the correction device 100 inputs imitation skeletal data, etc., acquired from an infrared sensor, etc., as shown in FIG. 6, and based on the input results, detects the orientation of the imitation, expressed by the angle of the imitation's sagittal horizontal axis relative to the display surface of the motion image, etc., as shown in FIG. 6. Then, the imitation orientation detection unit 110 calculates the orientation of the motion image such that the sagittal horizontal axis of the motion image relative to the screen displaying the motion image and the imitation's sagittal horizontal axis relative to the screen displaying the motion image are symmetrical (S11). The imitation orientation detection unit 110 outputs the orientation of the motion image (before correction) calculated in S11 to the motion image orientation correction unit 130 (S12).

[0021] FIG. 7 is a flowchart showing an example of the processing procedure performed by the visual agnosia characteristic input unit of the correction device. FIG. 8 is a diagram showing an example of the definition of palm angle. FIG. 9 is a diagram showing an example of test results for recognition accuracy for hand angles and the front and back of the hand. The visual agnosia characteristic input unit 120 inputs the visual agnosia characteristic of the imitator. As shown in FIG. 8, the visual agnosia characteristic input unit 120 tests the recognition accuracy at angles of 0, 45, 90, 135, 180, 225, 270, and 315° in the direction of rotation toward the median axis of the body, with the median axis of the hand at an angle of 0°. The test results shown in FIG. 9 are, for example, test results obtained when a "hand laterality task" is used, in which the task is to recognize whether the hand is left or right. Furthermore, the test method is not particularly limited as long as it involves hand recognition accuracy.

[0022] The visual agnosia characteristic input unit 120 extracts, for the palm and back of the hand, the values ​​of angle A, which is 40 points or more lower than the recognition accuracy at an angle of 0°, as shown in Fig. 10, and angle B, which is 35 points or more lower than the recognition accuracy at an angle of 315°, as shown in Fig. 10 (S21). The visual agnosia characteristic input unit 120 outputs the extraction results to the hand skeletal structure correction unit 140 (S22). Here, angle A is between 0° and 180°, and angle B is between 180° and 315°.

[0023] Fig. 10 is a diagram showing an example of test results for angles having a predetermined difference in angle recognition accuracy. Fig. 11 is a diagram showing an example of the difference in recognition accuracy at a predetermined angle. The method of defining angles A and B shown in Fig. 10 is an example when using the test results of the above-mentioned "hand laterality task," and a method of defining angles A and B can be used depending on the type of visual agnosia test used.

[0024] The visual agnosia characteristic input unit 120 calculates the difference C between the recognition accuracy of the palm and the back of the hand at an angle of 180°, as shown in Fig. 11 (S23). The visual agnosia characteristic input unit 120 outputs the result of this calculation to the movement image direction correction unit 130 (S24). The visual agnosia characteristic is determined by steps S21 to S24.

[0025] Fig. 12 is a flowchart showing an example of the procedure of the processing operation by the movement image direction correction unit. Fig. 13 is a diagram showing an example of the direction of the movement image before and after correction by the movement image direction correction unit. The movement image direction correction unit 130 inputs a value C which is the difference between the direction of the movement image (before correction) output by the imitation direction detection unit 110 and the visual agnosia characteristic input unit 120.

[0026] When the input value C is equal to or greater than X points, that is, when the input value C indicates that the recognition accuracy of the back of the hand angle is lower than the recognition accuracy of the palm angle by X points or more at an angle of 180° (Yes in S31), the movement image direction correction unit 130 generates a command to correct the direction in which the movement image is displayed so that the palm is visible, as shown in Fig. 13 (S32). The direction in which the movement image is displayed is determined by steps S31 to S33, taking into account whether or not correction is required.

[0027] The value of X is 10 points when the test results of the hand laterality task are used. The value of X is determined based on the criteria for the type of visual agnosia test used. If the answer is No in S31, the movement image direction correction unit 130 does not generate a command to correct the direction in which the movement image is displayed (S33). The movement image direction correction unit 130 outputs the command to correct the direction in which the movement image is displayed, generated in S32, to the movement image generation device 200.

[0028] Fig. 14 is a flowchart showing an example of the procedure of the processing operation by the hand skeletal structure correction unit. Fig. 15 is a diagram showing an example of the correction content by the hand skeletal structure correction unit. The hand skeletal structure correction unit 140 receives the angle values ​​A and B output by the visual agnosia characteristic input unit 120.

[0029] When the hand angle of the movement image before correction (sometimes referred to as the hand angle before correction) is in the range of "A° to 180°" (Yes in S41), the hand skeletal structure correcting unit 140 generates a command to correct the hand angle of the movement image before correction to A° as shown in Fig. 15 (S42). On the other hand, if the hand angle of the movement image before correction is not in the range of "A° to 180°" (No in S41) but is in the range of "180° to B°" (Yes in S43), the hand skeletal structure correcting unit 140 generates a command to correct the hand angle of the movement image before correction to B° as shown in Fig. 15 (S43).

[0030] Furthermore, if the hand angle of the movement image before correction is neither in the range of "A° to 180°" nor in the range of "180° to B°" (No in S43), the hand skeletal structure correcting unit 140 does not generate a command to correct the hand angle of the movement image before correction (S45). The hand skeletal structure correcting unit 140 outputs the command to correct the hand angle of the movement image, generated in S42 or S44, to the movement image generation device 200. The hand skeletal structure is determined by S41 to S45.

[0031] The movement image generating device 200 inputs an instruction regarding correction of the direction in which the movement image is displayed, output from the movement image direction correction unit 130, and an instruction regarding correction of the angle of the hand in the movement image, output from the hand skeletal structure correction unit 140.

[0032] The movement image generating device 200 generates a skeletal structure of a movement image, calculates element change information for causing deformations related to the imitation's eating movement for this generated skeletal structure based on the various input commands, and synthesizes a movement image video that moves the segmented images in accordance with the calculated element change information. The movement image generating device 200 outputs this synthesized movement image video to the movement image presenting unit 300.

[0033] The movement image presentation unit 300 presents the corrected movement image output from the movement image generation device 200 to an imitator such as a person with dementia eating via an LCD monitor or a projector.

[0034] In the above-described embodiment, when generating an imitated motor image, the direction of displaying the motor image is determined so that the horizontal sagittal axis of the movement relative to the screen displaying the motor image and the horizontal sagittal axis of the imitator relative to the screen displaying the motor image are symmetrical. Furthermore, the direction of displaying the motor image and the skeletal structure of the hand of the motor image are corrected according to the characteristics of the imitator's visual agnosia. This allows the direction of displaying the projected motor image and the angle of displaying the hand to be corrected according to the characteristics of the visual agnosia of the target, such as a dementia patient, thereby realizing the display of a motion image that is easy to imitate even for a dementia patient with visual agnosia. This effectively induces the imitator to imitate the motion.

[0035] Fig. 16 is a block diagram showing an example of the hardware configuration of a correction device according to an embodiment of the present invention. In the example shown in Fig. 16, the correction device 100 according to the above embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 511A such as a CPU. A program memory 511B, a data memory 512, an input / output interface 513, and a communication interface 514 are connected to this hardware processor 511A via a bus 515. The same applies to the motion image generation device 200 and the like.

[0036] The communication interface 514 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from the communication network NW. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0037] An input device 600 and an output device 700, which are attached to the correction apparatus 100 and used by a user or the like, are connected to the input / output interface 513. The input / output interface 513 can take in operation data input by a user or the like through the input device 600, such as a keyboard, a touch panel, a touchpad, or a mouse, and can output output data to an output device 700, which includes a display device using a liquid crystal or an organic electroluminescence (EL) display, for display. The input device 600 and the output device 700 may be devices built into the correction apparatus 100, or may be input devices and output devices of other information terminals that can communicate with the correction apparatus 100 via a network NW.

[0038] The program memory 511B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a ROM, and can store programs necessary to execute various control processes, etc., according to one embodiment.

[0039] The data memory 512 is a tangible storage medium that is, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM, and can be used to store various data or information acquired and created during various processes.

[0040] The correction device 100 according to one embodiment of the present invention can be configured as a data processing device having the components shown in FIG. 2 as software processing functional units.

[0041] The information storage unit used as a work memory or the like by each unit of the correction device 100 can be configured by using the data memory 512 shown in Fig. 16. However, these configured storage areas are not essential components within the correction device 100, and may be areas provided in, for example, an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0042] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0043] The methods described in each embodiment may be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), or may be transmitted and distributed via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only executable programs but also tables and data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-described processing by having the operation controlled by this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0044] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0045] REFERENCE SIGNS LIST 100 Correction device 110 Imitator direction detection unit 120 Visual agnosia characteristic input unit 130 Movement image direction correction unit 140 Hand skeletal structure correction unit 200 Movement image generation device 300 Movement image presentation unit

Claims

1. A motor image correction device comprising: a detection unit which detects an orientation of an imitator with respect to a display surface of a motor image, and detects an orientation of the motor image that is symmetrical to the detected orientation with respect to the display surface of the motor image; a first correction unit which calculates a correction of the orientation of the motor image detected by the detection unit to an orientation in which the imitator can see the palm of the motor image, based on the visual agnosia characteristics of the imitator; and a second correction unit which calculates a correction of the skeletal structure of the hand of the motor image, based on the visual agnosia characteristics of the imitator.

2. The motion image correction device of claim 1, further comprising a calculation unit that calculates a difference in recognition accuracy of the palm and back of the motion image by the imitator based on a visual agnosia characteristic of the imitator, wherein the first correction unit calculates a correction to the orientation of the motion image detected by the detection unit to an orientation in which the imitator can visually recognize the palm of the motion image, based on the difference calculated by the calculation unit.

3. The movement image correcting device of claim 1, further comprising a calculation unit that calculates an angle at which the palm of the movement image is visible to the imitator based on a visual agnosia characteristic of the imitator, wherein the second correction unit calculates a correction to the skeletal structure of the hand of the movement image based on the visible angle calculated by the calculation unit.

4. The movement image correction device of claim 1, further comprising a calculation unit that calculates the angle at which the palm of the movement image is visible to the imitator based on a visual agnosia characteristic of the imitator and calculates the difference in recognition accuracy of the palm and back of the movement image by the imitator, wherein the first correction unit calculates a correction to the orientation of the movement image detected by the detection unit to an orientation at which the palm of the movement image is visible to the imitator based on the difference calculated by the calculation unit, and the second correction unit calculates a correction to the skeletal structure of the movement image's hand based on the visible angle calculated by the calculation unit.

5. The motion image correction device of claim 1, wherein the detection unit detects an orientation of the imitation represented using a sagittal horizontal axis of the imitation relative to a display surface of the motion image, and detects an orientation of the motion image represented using a sagittal horizontal axis of the motion image that is symmetrical to the detected orientation relative to the display surface of the motion image.

6. The motion image correcting device of claim 1, wherein the second correction unit, when the angle of the motion image's hand is between the angle at which the palm of the motion image is visible to the imitator and 180 degrees, calculates a correction of the angle of the motion image's hand to the visible angle as a correction of the skeletal structure of the motion image's hand.

7. A method performed by a motion image correction device, comprising: detecting, by a detection unit of the motion image correction device, an orientation of an imitator with respect to a display surface of a motion image, and detecting an orientation of the motion image with respect to the display surface of the motion image that is symmetrical to the detected orientation; calculating, by a first correction unit of the motion image correction device, a correction of the orientation of the motion image detected by the detection unit to an orientation in which the imitator can visually recognize the palm of the motion image, based on the visual agnosia characteristics of the imitator; and calculating, by a second correction unit of the motion image correction device, a correction of the skeletal structure of the hand of the motion image, based on the visual agnosia characteristics of the imitator.

8. A motion image correction processing program for causing a processor to function as each part of the motion image correction device according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Rehabilitation aiding device

    JP1997056697A

  • Exercise supporting device, exercise supporting system, exercise supporting method and computer program

    JP2009213782A

  • Apparatus for evaluating and training higher-order brain dysfunction

    JP2010131118A

  • Game device, display method and program

    JP2013066613A

  • Training device, training system, program, and control device

    JP2019005243A