Assistance system, assistance method, and program

Abstract

One aspect of the present invention is an assistance system comprising a control unit that performs: target posture inference processing for inferring a target posture, which is a posture to be transitioned to by an action receiver having a plurality of joints, on the basis of current posture information that has been obtained on the basis of the result of imaging the action receiver with a first imaging device and that indicates the current posture of the action receiver at the time of imaging, and training information that was obtained in the past and that indicates a change in posture; and presentation processing for superimposing, on the action receiver, a target computer graphic image which is an image of the action receiver in the target posture, thereby presenting the target computer graphic image to an action performer who will move the action receiver.

Description

Support system, support method and program

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 439,155, filed January 16, 2023, the contents of which are incorporated herein by reference.

[0002] Opportunities for providing care for elderly people and the like are increasing. In recent years, a method has been proposed for using augmented reality (AR) to provide text information, video, and other information to instruct and teach caregivers how to perform caregiving tasks (see, for example, Non-Patent Document 1).

[0003] Jan Patrick Kopetz, Daniel Wessel and Nicole Jochems. "User-Centered Development of Smart Glasses Support for Skills Training in Nursing Education", i-com, vol. 18, no. 3, 2019, pp. 287-299.

[0004] In caregiving, there are many opportunities for caregivers to move the care recipient's body, such as getting the care recipient out of bed. However, such tasks are often difficult to carry out unless the caregiver is an experienced caregiver. Opportunities for aspiring caregivers to learn caregiving know-how, such as how to move the body, from such experienced caregivers are limited, and there is also the problem that it is a heavy burden for experienced caregivers to educate each aspiring caregiver individually.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique for reducing the burden required for care.

[0006] One aspect of the present invention is an assistance system comprising: a control unit that executes a target posture estimation process that estimates a target posture, which is a posture to which the acted object will transition, based on current posture information, which is information obtained based on the results of photography by a first photography device that photographs an acted object having a plurality of joints and indicates the current posture, which is the posture of the acted object at the time of photography, and teacher information, which is information obtained in the past and indicates a change in posture; and a presentation process that superimposes a target computer graphics image, which is an image of the acted object in the target posture, on the acted object and presents it to an acting subject that moves the acted object.

[0007] One aspect of the present invention is an assistance method having a control step of executing a target posture estimation process that estimates a target posture, which is a posture to which the acted object will transition, based on current posture information, which is information obtained based on the results of photography by a first camera that photographs an acted object having a plurality of joints and is information indicating the current posture, which is the posture of the acted object at the time of photography, and a presentation process that superimposes a target computer graphics image, which is an image of the acted object in the target posture, on the acted object and presents it to an acting subject that moves the acted object.

[0008] One aspect of the present invention is a program for causing a computer to function as the above-described support system.

[0009] The present invention makes it possible to reduce the burden of caregiving.

[0010] 1 is a first explanatory diagram for explaining a support system according to an embodiment. FIG. 2 is a second explanatory diagram for explaining a support system according to an embodiment. FIG. 3 is a flowchart showing an example of a processing flow executed by a teaching control device according to an embodiment. FIG. 4 is an example of a transition process of an image of a care recipient and a target computer graphics image superimposed thereon, which are displayed on an AR device of the support system according to an embodiment.

[0011] (Embodiment) FIG. 1 is a first explanatory diagram illustrating a support system 100 according to an embodiment. FIG. 2 is a second explanatory diagram illustrating the support system 100 according to an embodiment. The support system 100 includes a first image capturing device 1, an AR device 2, and a support device 3. The AR device 2 may be an MR (Mixed Reality) device. The support system 100 supports a caregiver 901 in caring for a care recipient 902. The care recipient 902 is a person who receives care, and has multiple joints. The care recipient 902 is an example of an acted object having multiple joints. The caregiver 901 is an example of an actor that moves an acted object.

[0012] The first imaging device 1 is an imaging device such as a camera that captures an image of the care recipient 902. The camera may be, for example, a depth camera.

[0013] The AR device 2 is an AR device that is worn by the caregiver 901. The AR device 2 superimposes a target computer graphics image, which is an image of the care recipient 902 in a target posture, on the care recipient 902 at the time of photographing by the first photographing device 1 (hereinafter referred to as the "current time"), and displays the image on a display device such as a display or a transparent lens capable of displaying images. The target posture is a posture to which the care recipient 902 will transition from the current posture (i.e., the posture to which the care recipient 902 should be next).

[0014] Image G1 shown in Fig. 2 is an example of an image seen by the caregiver 901 wearing the AR device 2. Image G101 in Fig. 2 is an example of a target computer graphics image displayed superimposed on the care recipient 902. The posture of image G101 is a transition destination posture from the current posture. The display of the transition destination posture by the AR device 2 may be repeated until the transition to the destination posture is achieved. After the posture of the care recipient 902 has become the transition destination posture, the AR device 2 displays an image of the next transition destination posture to the caregiver.

[0015] The caregiver 901 can provide appropriate care to the care recipient 902 by transitioning the care recipient 902 according to the image of the movement of each step in accordance with the image displayed on the AR device 2. In this way, the target computer graphics image is, for example, a three-dimensional model that imitates the care recipient 902 (i.e., an avatar of the care recipient 902) and is in a target posture. The three-dimensional model may be a display of an image of a part of the care recipient 902 that is to be transitioned, or may be an entire image of the care recipient 902.

[0016] In this way, by using a three-dimensional model image superimposed on the image of the care recipient 902 acquired by the first photographing device 1, the caregiver 901 wearing the AR device 2 can easily visually recognize the difference between the current posture of the care recipient 902 and the target posture.

[0017] The display device 201 in FIG. 1 is an example of a display device included in the AR device 2, such as a display or a transmission lens.

[0018] The target posture is, for example, the posture of the care recipient 902 while the care recipient 902 is being cared for by an experienced caregiver. Such posture information is acquired, for example, based on the results of capturing images of the care recipient 902 being cared for by the experienced caregiver. When capturing images of the care recipient 902 being cared for by the experienced caregiver, a series of actions during the care are captured.

[0019] As a result, a time series of postures of the care recipient 902 while being cared for by the skilled caregiver is acquired by predetermined information processing, such as using a mathematical model that has been trained by motion capture or machine learning and obtains posture information from images (hereinafter referred to as a "posture information acquisition model"). The time series acquired in this manner is an example of training information, which will be described later. The image of the three-dimensional model of the target posture is the posture shown by each sample in the time series thus obtained.

[0020] If there is a three-dimensional model image based on each sample of such a time series, even an inexperienced caregiver can provide care similar to that of an experienced caregiver by moving the care recipient 902 so as to reproduce the posture shown in the time series. Therefore, the support system 100 presents the caregiver 901 with a target posture corresponding to the current posture of the care recipient 902.

[0021] The posture of the care recipient 902 is not limited to that captured by the first imaging device 1. If the AR device 2 is an AR device equipped with, for example, a transmission lens, the posture of the care recipient 902 is, for example, the posture of the care recipient 902 while the caregiver 901 is visually observing the care recipient 902 through the transmission lens. If the AR device 2 is an AR device equipped with, for example, a display, the AR device 2 may also be equipped with an imaging device, and the imaging results of the imaging device are displayed on the display.

[0022] The AR device 2 superimposes the target computer graphics image on the care recipient 902 and presents it to the caregiver 901. Note that such presentation processing by the AR device 2 is executed under the control of a control unit 31 included in the support apparatus 3, as will be described later. Therefore, it can be said that the control unit 31 executes the presentation processing.

[0023] The presentation process is a process of displaying the target computer graphics image on a predetermined display destination such as the display device 201 of the AR device 2, thereby superimposing the target computer graphics image on the care recipient 902 and presenting it to the caregiver 901. As a result of the presentation process, the caregiver 901 sees the target computer graphics image superimposed on the care recipient 902.

[0024] Note that the image processing technique for superimposing may be any known technique. For example, the control unit 31 estimates a superimposition transformation, which is a transformation that matches the image of the care recipient 902 captured by the first image capturing device 1 with the image of the care recipient 902 seen by the caregiver 901, based on relative positioning information that indicates the relative position and orientation relationship between the first image capturing device 1 and the AR device 2.

[0025] The relative position information may be obtained using technology such as GPS (Global Positioning System) or other well-known positioning technology. For example, if a predetermined mark such as a QR code (registered trademark, the same applies hereinafter) is printed on the first image capturing device 1 and the AR device 2 is equipped with an image capturing device, the relative position information may be obtained by reading the position of the predetermined mark using the image capturing device.

[0026] If the superposition transformation is obtained in this way, the control unit 31 can convert the image of the care recipient 902 in the target state as seen from the first image capturing device 1 into an image of the care recipient 902 in the target state as seen from the caregiver 901. If the image of the care recipient 902 in the target state as seen from the first image capturing device 1 is estimated in this way, it is possible to superimpose the target computer graphics image on the care recipient 902 and present it to the caregiver 901.

[0027] The image of the care recipient 902 in the target state as seen by the first image capturing device 1 is estimated using, for example, a target posture estimation process described later. Note that if the difference between the image of the care recipient 902 as seen by the first image capturing device 1 and the image of the care recipient 902 as seen by the caregiver 901 is small, it is not necessarily necessary to perform the convolution transformation.

[0028] The posture information acquisition model described above is a mathematical model that estimates posture information such as the position and orientation of each joint in a joint coordinate system, which will be described later, from an image. Such a posture information acquisition model is obtained by learning using pairs of images and posture information as learning data, with a mathematical model that estimates a posture for an input image as the learning object. In such learning, the learning object mathematical model is updated so as to reduce the difference between the posture estimated by the learning object mathematical model for the input image and the posture indicated by the posture information included in the learning data. The learning object at the point in time when a predetermined condition for terminating learning is satisfied is the posture information acquisition model.

[0029] The support device 3 includes a control unit 31 having a processor 91 such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit) and a memory 92 connected by a bus, and executes a program. The support device 3 functions as a device having the control unit 31, an interface unit 32, and a storage unit 33 by executing the program.

[0030] More specifically, the processor 91 reads out a program stored in the storage unit 33 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, causing the support device 3 to function as a device including the control unit 31, the interface unit 32, and the storage unit 33.

[0031] The control unit 31 controls the operation of each functional unit included in the support device 3. The control unit 31 controls, for example, the operation of the interface unit 32, and acquires the results of imaging by the first imaging device 1. The control unit 31 acquires, for example, information stored in the memory unit 33. Specifically, the process of acquiring the information stored in the memory unit 33 is reading.

[0032] The control unit 31 executes, for example, a target posture estimation process. The target posture estimation process is a process for estimating a target posture based on current posture information, which is information obtained based on the result of image capture by the first image capture device 1 and indicates the current posture of the care recipient 902, which is the posture of the care recipient 902 at the time of image capture, and teacher information. The teacher information is information obtained in the past and indicates a change in posture.

[0033] The teacher information may be information that modifies posture changes depending on the physical condition, state, or other circumstances of the care recipient 902. This allows the support device 3 to provide optimal care to the care recipient 902. The teacher information may also be information that changes the level of the care action depending on the caregiver 901's caregiving proficiency or physical strength. In this case, the teacher information indicates changes in the caregiving posture depending on the level. For example, the control unit 31 acquires and displays a care action performed in one step, a care action performed in multiple steps, or a care action using an assistive device from the information stored in the storage unit 33. In such a case, when the control unit 31 obtains information indicating the level, it displays the teacher information for that level on a predetermined display, such as the display device 201 provided in the AR device 2, based on the level indicated by the level-indicating information. This allows even a caregiver 901 who is unable to perform a particular step due to insufficient caregiving skills or lack of physical strength to perform the step by using multiple steps or tools.

[0034] The control unit 31 controls the display operation of the AR device 2 via, for example, the interface unit 32. In this way, the control unit 31 executes, for example, a presentation process.

[0035] The interface unit 32 includes a communication interface for connecting the support device 3 to an external device. The interface unit 32 communicates with the external device via a wired or wireless connection. The external device is, for example, the first image capturing device 1. The external device is, for example, the AR device 2.

[0036] The storage unit 33 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 33 stores various information related to the assistance device 3. The storage unit 33 stores, for example, teacher information in advance. The storage unit 33 stores, for example, various information generated by the operation of the control unit 31.

[0037] 3 is a flowchart showing an example of the flow of processing executed by the support device 3 in this embodiment. The control unit 31 acquires the results of photography by the first photography device 1, etc. (step S101). Next, the control unit 31 executes a target posture estimation process (step S102). Next, the control unit 31 executes a presentation process (step S103).

[0038] After step S103, the caregiver 901 moves the care recipient 902 so that the care recipient 902 approaches the target computer graphics image presented in step S103. The first image capturing device 1 repeatedly captures images at a predetermined timing, and the control unit 31 synchronizes with the timing and determines, based on the captured images, whether or not the difference between the posture of the care recipient 902 and the target posture is a predetermined difference.

[0039] If the control unit 31 determines that the difference is within a predetermined difference, the target posture estimated by the target posture estimation process is updated. If the difference is greater than the predetermined difference, the target posture is not updated even if the target posture estimation process is executed. In this way, by repeating the processes of steps S101 to S103, the caregiver 901 can move the care recipient 902 in accordance with the instruction information.

[0040] The support device 3 configured in this manner controls the AR device 2 to present the target computer graphics image to the caregiver 901, superimposed on the care recipient 902. Thanks to the control of the AR device 2 by the support device 3 in this way, the caregiver 901 can visually more easily grasp how to move the care recipient 902 next. Therefore, by using the support device 3, even a caregiver 901 with low skills can provide appropriate care without the direct guidance of a skilled caregiver.

[0041] Furthermore, by using the support device 3, a less skilled caregiver 901 can learn the caregiving techniques of an experienced caregiver. As a result, by using the support device 3, it is possible to reduce the burden of instruction on the experienced caregiver and the burden of care on the caregiver 901. Furthermore, since the support system 100 configured in this manner is equipped with the support device 3, it is possible to reduce various burdens required for caregiving.

[0042] In Non-Patent Document 1, what to do is presented using text information or video, but the posture of the care recipient 902 does not necessarily match the presented content. Also, with video information, the information is limited to the viewpoint at the time of shooting. However, since the viewpoint of the caregiver 901 during the caregiving action varies from time to time, it is desirable to present an appearance that matches the current viewpoint of the caregiver 901. In the case of the support system 100, the caregiving action can be presented to the caregiver 901 according to the target care recipient 902, which reduces the burden on the caregiver 901 and contributes to skill acquisition.

[0043] FIG. 4 shows an example of the transition process of the image of the care recipient 902 displayed on the AR device 2 of the support system 100 and the target computer graphics image G101 superimposed thereon.

[0044] Image D101 in Fig. 4 shows an initial state in which a target computer graphics image G101 in an initial state is superimposed on the current posture of the care recipient 902. Image D102 in Fig. 4 shows a state in which the target computer graphics image G101 superimposed on the care recipient 902 has started to move toward a transition posture. Image D103 in Fig. 4 shows a state in which the target computer graphics image G101 superimposed on the care recipient 902 is approaching a transition posture. Image D104 in Fig. 4 shows a state in which the target computer graphics image G101 superimposed on the care recipient 902 has reached a final transition posture.

[0045] The control unit 31 may control the operation of the AR device 2 to repeatedly cause the AR device 2 to play back and display the target computer graphics image G101 up to that point until the caregiver 901 places the care recipient 902 in the final transition posture. The control unit 31 may also execute processing to change the playback display speed of the target computer graphics image G101. The control unit 31 may also execute processing to turn off the playback display of the target computer graphics image G101. The control unit 31 may be a CPU included in an external device such as the assistance device 3, or may be a CPU installed in the main body of the AR device 2.

[0046] (Variation) The teacher information may be, for example, information obtained based on the sample acted action, and information indicating the posture of the sample acted object at each timing while the sample acting subject is performing the sample acting action.

[0047] The sample acted action is a change in the posture (i.e., movement) of the sample acted object that occurs as a result of the execution of the sample acting action by the sample acting subject that executes the sample acting action. The sample acted object is an acted object that is the same as or different from the acted object photographed by the first photographing device 1. For example, the sample acted object is the teacher in the role of the care recipient when two teachers who teach nursing care techniques to students are divided into the roles of the care recipient and the caregiver and play each role.

[0048] The sample acted upon object is an acted upon object that is the same as or different from the acted upon object photographed by the first photographing device 1. The sample action is a predetermined action that moves the sample acted upon object. The sample actor is someone who performs the sample action. The sample actor is, for example, an experienced caregiver. Therefore, the sample acted upon object is, for example, the teacher who plays the role of caregiver when two teachers who teach caregiving techniques to students play the roles of the care recipient and the caregiver.

[0049] The teaching information may be information obtained based on the results of photographing the sample acted action by a second photographing device which is the same as or different from the first photographing device 1, for example.

[0050] The current posture information may indicate the posture of the acted upon object by the position and orientation of a joint coordinate system, which is a coordinate system predefined for each joint. The teacher information may indicate the posture of the acted upon object by the position and orientation of the joint coordinate system. The position and orientation of the joint coordinate system refer to the position of the origin of the joint coordinate system and the orientation of each axis.

[0051] Furthermore, in the current posture information and the teacher information, the position and orientation of the joint coordinate system of the (n+1)th joint, which is one of two joints adjacent to the nth joint (n is an integer equal to or greater than 1) among the plurality of joints and is a joint different from the (n-1)th joint, may be represented by the joint coordinate system of the nth joint. Note that the position and orientation of the joint coordinate system of the n=1 joint are represented by the world coordinate system, which is one of the coordinate systems used in the field of three-dimensional computer graphics.

[0052] The target computer graphics image presented in the presentation process may be presented in a state in which the position and orientation of a joint coordinate system of a reference joint, which is a specific joint among the multiple joints, match the position and orientation of the joint coordinate system of the reference joint of the acted object. The reference joint is the starting point of the movement, such as the shoulder joint in the movement of bending the arm of the care recipient 902. In such a case, when the target computer graphics image is presented in the above-described matched state, the caregiver 901 sees an image in which the shoulder joint of the care recipient 902 matches the shoulder joint in the image G101, as shown in image G1 in FIG. 2 . In this case, the starting point of the movement is defined by a human hand or an estimation result by a CPU when creating training data. The definition of the starting point of the movement may be determined in advance by the user based on which joint is the root and which joint is the end of the movement that is easy to intuitively teach to the user wearing the AR device 2.

[0053] Although it is common to superimpose a computer graphics image onto a specific location in real space, no technology has been established for superimposing a multi-joint target computer graphics image onto a multi-joint object with restricted movement.

[0054] For example, when the care recipient 902 bends his / her arm, the following three coordinate axes are usually calculated as part of the process for visualizing changes in the coordinate axes of the joints using coordinates: One of the three coordinate axes is a coordinate axis that indicates the movement of the shoulder joint, and is calculated by multiplying a matrix that represents the acquired coordinate axes of the shoulder joint by a matrix that represents the time-series changes in the coordinate axes of the shoulder joint (teaching data).

[0055] The other of the three coordinate axes is a coordinate axis indicating the movement of the elbow joint, and is calculated using a matrix representing the acquired coordinate axis of the shoulder joint, a matrix representing the time series changes in the coordinate axis of the shoulder joint (teacher data), and a matrix representing the time series changes in the coordinate axis of the elbow joint (teacher data). The remaining one of the three coordinate axes is calculated by multiplying the acquired matrix representing the coordinate axis of the shoulder joint, a matrix representing the time series changes in the coordinate axis of the shoulder joint (teacher data), a matrix representing the time series changes in the coordinate axis of the elbow joint (teacher data), and a matrix representing the time series changes in the coordinate axis of the wrist joint (teacher data).

[0056] The visualization process described above requires calculation of each of the three coordinate axes, and therefore the amount of calculation increases with each additional joint. Since the AR device 2 often uses a computer with relatively low performance for calculations in order to reduce weight, it is desirable to reduce the amount of calculation.

[0057] In the assistance system 100 and the assistance device 3, by using a joint coordinate system in which the position of the starting point of movement is the reference joint, the calculation amount of the processor 91 of the control unit 31 can be reduced while making it easier to directly edit the teaching information. Reducing the calculation amount means reducing the calculation amount of the process for visualization. Making it easy to edit means making it easier for the user or the CPU to modify the teaching information by issuing commands specifying the direction and degree of rotation of the distal joint for each joint. The control unit 31 can determine the joint group positions and orientations of a multi-joint object in real space, such as the care recipient 902, and the joint group positions and orientations of a multi-joint object in virtual space displayed by the AR device 2, and superimpose and display a target computer graphics image.

[0058] In the support system 100 and the support device 3, the starting point of the movement is determined by the control unit 31 or a human hand. For example, in the case of a movement to bend the arm of the care recipient 902, the shoulder joint on the root side is set as the starting point as the reference joint for the rotation of the joint coordinate system group from the shoulder joint on the root side of the multiple joints to the fingertips on the distal side.

[0059] When the control unit 31 determines the starting point in this way and places the target computer graphics image in accordance with the real space, it acquires a coordinate system of the starting point of the operation and displays the coordinate system as the position and orientation of the target computer graphics image when it is operated. Note that the starting point may also be determined manually.

[0060] The teacher information is stored in the storage unit 33 in a storage format in which the base side of a multi-joint object is set as the starting point of the movement. Specifically, for example, in the movement of bending the arm of the care recipient 902, time-series information on the "position and posture of the shoulder joint," the "position and posture of the elbow joint viewed from the shoulder joint," and the "position and posture of the wrist joint viewed from the elbow joint" is stored as teacher information.

[0061] When appropriately reproducing such time-series information, the control unit 31 first performs coordinate transformation to correct the "position and posture of the shoulder joint" in the current situation of the care recipient 902 and the "position and posture of the shoulder joint" in the teacher information. Note that "appropriate" here is defined as not causing any discomfort to the caregiver 901. As a result, the control unit 31 superimposes the shoulder joint coordinate axis in real space with the shoulder joint coordinate axis in the teacher information. Next, the control unit 31 displays the joint movement between the shoulder joint as the root side and the hand as the distal side on a predetermined display device. At this time, the control unit 31 displays the joint movement between the root side and the distal side on a predetermined display device by displaying the changes in each coordinate axis from the root side to the distal side on a predetermined display device. This allows the target computer graphics image obtained from the teacher information to be reproduced by being appropriately superimposed on real space.

[0062] Furthermore, when the training information indicates the movement of each joint with the base of a multi-jointed object as the starting point, a command specifying in what direction and by how many degrees the distal joint is to be rotated for each joint can be directly written into the training information by a user or a computer, and information editing can be easily performed. Note that the starting point in this invention is defined as the point at which the movement of the multi-jointed object begins in a transition operation.

[0063] Incidentally, there are situations in which the reference joint serving as the starting point is a joint with n=1, and while the target posture changes and the position of the reference joint changes, for example, in nursing care, the relative positions and orientations of joints with n equal to or greater than 2 with respect to the reference joint do not change. In such cases, if the positions and orientations of all joints are expressed in a world coordinate system, the positions and orientations of all joints must be re-estimated by computationally intensive processing such as executing motion capture or executing a trained model based on machine learning, such as a posture information acquisition model.

[0064] On the other hand, when the position and orientation of the joint coordinate system of the (n+1)th joint are represented by the joint coordinate system of the nth joint, they represent the relative position and orientation from adjacent joints, so as long as the change in the position of the reference joint is estimated, it is only necessary to perform matrix multiplication, which requires a small amount of calculation, for the other joints. In this way, by representing the position and orientation of the joint coordinate system of the (n+1)th joint by the joint coordinate system of the nth joint, rather than representing all joints in the world coordinate system, the effect of reducing the amount of calculation is achieved.

[0065] As described above, the sample object being cared for when teacher information is acquired may differ from the person being cared for (i.e., the care recipient 902) when the caregiver 901 is providing care. In such a case, taking into consideration that the distance between joints, such as differences in arm length, differs from person to person, it is preferable for the caregiver 901 to have the image of the avatar of the care recipient 902 superimposed on the care recipient 902 as the target computer graphics image, rather than having the image of the avatar of the sample object being superimposed on the care recipient 902.

[0066] Therefore, the control unit 31 may estimate the difference in the distance between the joints of the care recipient 902 and the sample acted object based on the result of imaging by the first imaging device 1, and may present to a predetermined presentation destination a target computer graphics image in which the distance between the joints is adjusted to match the distance between the joints of the care recipient 902 based on the estimation result. The process of adjusting the distance between the joints to match the distance between the joints of the care recipient 902 is, for example, a process of moving the origin of each joint coordinate system to match the distance between the joints.

[0067] The first image capturing device 1 may be attached to, for example, the AR device 2. The AR device 2 may include, for example, an accelerometer and be capable of acquiring information indicating the position of the device itself. The AR device 2 may include an image capturing device as described above, and in such a case, the image capturing device may be, for example, a depth camera.

[0068] In addition, the control unit 31 may control the AR device 2 to present an image of a three-dimensional model that resembles the caregiver 901 (i.e., an avatar of the caregiver 901) and indicates the next action that the caregiver 901 should perform.

[0069] The support device 3 may be implemented using a plurality of information processing devices connected to each other via a network so as to be able to communicate with each other. In this case, the processes executed by the control unit 31 may be distributed among the plurality of information processing devices.

[0070] The display device 201 provided in the AR device 2 is an example of a predetermined display destination that displays a target computer graphics image superimposed on a current acted object.

[0071] All or part of the functions of the assistance system 100 and the assistance device 3 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0072] The support system 100 and the support device 3 of the present invention are not limited to nursing care, and may be any device that moves an acted object having multiple joints to a predetermined destination posture.

[0073] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0074] REFERENCE SIGNS LIST 100... support system, 1... first image capture device, 2... AR device, 201... display device, 3... support device, 31... control unit, 32... interface unit, 33... storage unit, 91... processor, 92... memory

Claims

1. a control unit which executes a target posture estimation process which estimates a target posture which is a posture to which the acted object will transition based on current posture information which is information obtained based on a result of photographing an acted object having a plurality of joints by a first photographing device and which indicates a current posture which is a posture of the acted object at the time of photographing, and teacher information which is information obtained in the past and which indicates a change in posture, and a presentation process which superimposes a target computer graphics image which is an image of the movement of the acted object in a state of the target posture on the acted object and presents it to an acting subject which moves the acted object; Equipped with The control unit changes the level based on level information corresponding to the proficiency or physical strength of the agent, and executes the target posture estimation process and the presentation process based on the teacher information of the level.

2. the object having a plurality of joints, the target computer graphics image, and the teaching information are expressed in a joint coordinate system in which a position serving as a starting point of the movement is set as a reference joint; The support system according to claim 1 .

3. The presentation process is a process of displaying a target computer graphics image on a display or a transmission lens of an AR (Augmented Reality) device equipped with a display capable of displaying an image or a transmission lens, The support system according to claim 1 .

4. The teaching information is information obtained based on a sample acted action, which is a change in posture of the sample acted object resulting from the execution of the sample acting action by a sample acting subject, the sample acting action being a predetermined action that moves a sample acted object that is the same or different from the acted object, and is information indicating the posture of the sample acted object at each timing during the execution of the sample acting action by the sample acting subject. The support system according to claim 1 .

5. The teaching information is information obtained based on a result of photographing the sample acted action with a second photographing device that is the same as or different from the first photographing device. The support system according to claim 4.

6. The current posture information and the teacher information indicate postures based on positions and orientations of a joint coordinate system, which is a coordinate system defined in advance for each joint. The support system according to claim 4.

7. The position and orientation of a (n+1)th joint, which is one of two joints adjacent to an nth joint (n is an integer equal to or greater than 1) among the plurality of joints and is a joint different from the (n-1)th joint, are represented by the joint coordinate system of the nth joint. The support system according to claim 6.

8. the target computer graphics image presented in the presentation process is presented in a state in which a position and an orientation of a joint coordinate system of a reference joint, which is a predetermined joint among the plurality of joints, coincide with a position and an orientation of a joint coordinate system of a reference joint of the acted object. The support system according to claim 7.

9. a control step of executing a target posture estimation process for estimating a target posture which is a posture to which the acted object will transition based on current posture information, which is information obtained based on the results of photographing an acted object having a plurality of joints by a first photographing device and which indicates a current posture which is a posture of the acted object at the time of photographing, and teacher information, which is information obtained in the past and which indicates a change in posture, and a presentation process for superimposing a target computer graphics image which is an image of the movement of the acted object in a state of the target posture, on the acted object and presenting it to an acting subject which moves the acted object; having The control step changes a level according to level information corresponding to a proficiency level or physical strength of the subject, and executes the target posture estimation process and the presentation process based on the teacher information of the level. How to help.

10. A program for causing a computer to function as the support system according to any one of claims 1 to 8.