Information processing system, information processing method, and program

The information processing system addresses the challenge of improving human posture during movement by projecting a 3D avatar based on IMU sensor data, allowing users to visually correct their posture in real-time, thus enhancing walking posture and gait.

WO2024247851A1PCT designated stage expired Publication Date: 2024-12-05TAMAGAWA ACADEMY & UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/018887
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2024-05-22
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Conventional methods fail to effectively improve human posture during movement, as they focus on static measurements and do not allow users to visually observe and correct their posture in real-time while performing actions like walking.

Method used

An information processing system utilizing IMU sensors, a 3DCG projection device, and a control device to generate and project a 3D avatar that mirrors the user's posture in real-time, enabling users to visualize and correct their movements dynamically.

Benefits of technology

Enables users to improve their posture unconsciously by visually referencing their avatar, effectively enhancing walking posture and gait through real-time feedback and comparison with ideal postures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024018887_05122024_PF_FP_ABST
    Figure JP2024018887_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present invention enables movement, such as walking, while looking at one's own figure. A control device 1: acquires, from a wireless motion capture suit 2, time-series IMU data acquired from a plurality of IMU sensors arranged respectively at a plurality of positions on the skeleton of a user U; measures each of a plurality of skeletal information items about the skeleton; generates a 3D CG video showing an avatar AU assuming a predetermined posture based on the measurement result; and displays the video on MR goggles 3.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing system, information processing method and program

[0001] The present invention relates to an information processing system, an information processing method, and a program.

[0002] In recent years, with the evolution of devices, technologies that utilize virtual reality (VR) for various physical training, including rehabilitation, have emerged. For example, Patent Document 1 describes an invention regarding a VR sports system that allows trainees (patients, etc.) to participate in VR sports to restore function and strengthen the body (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2005-066185

[0004] However, in order to improve posture while walking or performing other actions, it is said that it is necessary to see one's own appearance. However, with only conventional technologies including Patent Document 1, it is difficult to perform actions such as walking while looking at one's own appearance.

[0005] The present invention has been made in view of the above circumstances, and aims to provide a technique for performing movements such as walking while viewing one's own appearance.

[0006] In order to achieve the above object, an information processing system according to one aspect of the present invention includes: an IMU (Inertial Measurement Unit) sensor group including a plurality of IMU sensors that can be placed at the positions of a plurality of skeletal parts of a user; a 3DCG (3 Dimensional Computer Graphics) projection device that projects a 3DCG image into a real space in which the user is present in a state that is visible to the user; and a control device that communicates with the IMU sensor group and the 3DCG projection device to perform overall control, wherein the control device measures skeletal information of each of the plurality of skeletons based on time-series data acquired from the plurality of IMU sensors that are placed at the positions of the plurality of skeletal parts of the user, generates a 3DCG image that shows a first avatar assuming a predetermined posture based on the measurement results, and controls the projection of the 3DCG image from the 3DCG projection device.

[0007] An information processing method and a program according to one aspect of the present invention are respectively a method and a program corresponding to the information processing system according to one aspect of the present invention described above.

[0008] According to the present invention, it is possible to perform actions such as walking while watching one's own appearance.

[0009] 7 is a diagram illustrating an overview of a method for improving human posture during movement, which is made possible by implementing an embodiment of an information processing system of the present invention. It is a schematic diagram illustrating an example of the configuration of an embodiment of an information processing system of the present invention. It is a block diagram illustrating an example of the configuration of an embodiment of an information processing system of the present invention. It is a block diagram illustrating an example of the hardware configuration of a control device in the information processing systems of FIGS. 2 and 3. It is a diagram illustrating an example of the external configuration and sensor positions of a wireless motion capture suit in the information processing systems of FIGS. 2 and 3. It is a diagram illustrating an example of the external configuration of MR goggles in the information processing systems of FIGS. 2 and 3. It is a functional block diagram illustrating an example of the functional configuration of a control device having the hardware configuration of FIG. 4. It is a diagram illustrating an example of a coordinate system for IMU sensor data measured by a control device having the functional configuration of FIG. 7. It is a diagram illustrating an overview of an example that utilizes a first feature based on a basic system, among the features possessed by the information processing system of FIG. 2 including the ..., which is different from FIG. 9. It is a diagram illustrating an example that utilizes a first feature based on a basic system, among the features possessed by the information processing system of FIG. 2 including the control device having the functional configuration of FIG. 7, which is different from FIG. 9 and 14 is a diagram showing an example different from that shown in FIGS. 9 to 11 , which utilizes a first feature based on a basic system among the features of the information processing system of FIG. 2 including a control device having the functional configuration of FIG. 7 . FIG. 15 is a diagram showing an example different from that shown in FIGS. 9 to 11 , which utilizes a second feature of displaying an alert based on the basic system among the features of the information processing system of FIG. 2 including a control device having the functional configuration of FIG. 7 . FIG. 16 is a diagram showing an example different from that shown in FIG. 14 , which utilizes a third feature of displaying 3DCG of multiple people based on the basic system among the features of the information processing system of FIG. 2 including a control device having the functional configuration of FIG. 7 .10 is a diagram showing an overview of an example utilizing a fourth feature of presenting rhythmic sounds on the premise of a basic system, among the features of the information processing system of FIG. 2 including a control device having the functional configuration of FIG. 7 . FIG. 11 is a diagram showing an example of object detection and object recognition results used in an example different from FIGS. 9 to 14 , which utilizes features of the information processing system of FIG. 2 including a control device having the functional configuration of FIG. 7 . FIG. 12 is a diagram showing an overview of mode switching in an example using the object detection and object recognition results shown in FIG. 17 , among the examples utilizing features of the information processing system of FIG. 2 including a control device having the functional configuration of FIG. 7 . FIG. 13 is a diagram showing an overview of an example different from FIGS. 9 to 18 , which utilizes features of the information processing system of FIG. 2 including a control device having the functional configuration of FIG. 7 . FIG. 14 is a diagram showing an overview of mode switching in the example shown in FIG. 19 , among the examples utilizing features of the information processing system of FIG. 2 including a control device having the functional configuration of FIG. 7 . FIG. 15 is a diagram showing an overview of an example different from FIGS. 9 to 20 , which utilizes features of the information processing system of FIG. 2 including a control device having the functional configuration of FIG. 7 .

[0010] First, before describing an embodiment of an information processing system according to the present invention, the background that led the inventors to develop the present invention will be described.

[0011] The impact of aging on human posture is known to be one of the factors that contribute to falls, fractures, and brain diseases. The inventors have previously investigated the relationship between posture changes and aging, showing that aging has a greater impact on movement than on resting. Furthermore, by examining the posture, age, gender, bone mineral density, height, weight, body fat, arm strength, leg strength, and history of falls of 272 elderly people aged 65 or older, the inventors have shown that there are individual differences in the health status of elderly people and that health status can be classified based on parameters such as muscle strength, body fat, and posture. The inventors have also revealed that the impact of posture is more pronounced during activities such as standing and walking, such as leaning forward more easily than when standing upright and stationary. Based on these findings, the inventors aim to propose a technology to improve posture during movement, focusing on individual differences in the health status of elderly people. Conventional posture improvement methods typically involve measuring the degree of spinal curvature at rest and providing advice based on the results. However, even with these conventional methods, it is difficult to translate measurements of posture at rest into posture improvements during movement. Furthermore, basic postural control consists of postural reflexes that maintain posture in response to disturbances and unconscious postural adjustments that allow for voluntary movement of body parts such as the upper limbs. During activities such as walking in a real environment, it is necessary to pay attention to the surroundings and disturbances for safe movement. Responding to disturbances while consciously improving posture during movement can place a burden on the user and raise safety concerns. Meanwhile, with recent advances in computer technology, rehabilitation using virtual reality (VR) has been proposed, and it has been shown to effectively improve gait and balance after stroke, for example. Using VR technology, rehabilitation is now being carried out in situations and environments that are difficult to train in reality. However, VR-based training has problems, such as limited types of exercises.

[0012] Against this background, the present inventor has devised a method for improving human posture during movement, as shown in Fig. 1. Fig. 1 is a diagram for explaining an outline of a method for improving human posture during movement, which is made possible by implementing an embodiment of an information processing system of the present invention.

[0013] The present inventor has invented the following information processing system that utilizes mixed reality (hereinafter referred to as "MR") technology in order to realize a method for improving human posture during movement. That is, as shown in FIG. 1(A), the present inventor has invented an information processing system that can create a state in which an avatar AU that assumes a predetermined posture (e.g., the same posture) based on the posture of a user U moves in real time in real space together with the user U. Specifically, as shown in FIG. 1(B), the present inventor has invented an information processing system that can project an avatar AU that assumes the same posture as the user U as a 3DCG (3-dimensional computer graphics) (3D hologram) image in real time onto MR goggles worn by the user U.

[0014] Although details will be described later with reference to Figure 2 and subsequent drawings, according to one embodiment of the information processing system of the present invention, the 3DCG image of the avatar AU projected on the MR goggles is generated based on data acquired by an IMU (Inertial Measurement Unit) sensor built into a suit-type motion capture (hereinafter referred to as a "wireless motion capture suit") worn by the user U. The data acquired by the wireless motion capture suit is transferred to a control device constituted by a PC (Personal Computer) or the like via a wireless LAN (Local Area Network). The control device calculates skeletal position coordinates and joint angles based on the data, generates in real time a 3DCG image of the avatar AU assuming a posture based on the calculation results, and transfers the image to the MR goggles for projection. This allows the user U to perform actions such as walking while visually checking the posture of the avatar AU (i.e., his / her own posture) projected as a 3D hologram in the real space visible from the user's viewpoint, and also makes it possible to visualize the individual posture characteristics of free movements specific to that individual (user U).

[0015] Conventionally, walking support technologies using CG or MR technology have been proposed, primarily for guiding the user's lower limb movements by controlling the distance and movement from the presented image, or for use in walking ability evaluation tests. However, none of these conventional methods, like the "method for improving human posture during movement" (hereinafter abbreviated as "this method") shown in Figure 1 to which an embodiment of the information processing system of the present invention is applied, record the user's whole-body movements while walking freely by measuring them using a wireless motion capture suit, and the user visually confirms their own whole-body movements while walking.

[0016] In other words, in terms of "improving human posture during movement," the present method is more suitable than these conventional methods. Furthermore, the reasons why the present method is more suitable than the conventional techniques will be explained in detail below.

[0017] Conventional health maintenance, especially exercises and rehabilitation for improving posture, requires conscious effort on the part of the user to measure posture at rest and receive advice from a therapist, etc. However, this method utilizes sensory technology to detect changes in posture not only at rest but also during movement, which is difficult to detect visually, and allows the user U to measure by themselves without the assistance of a physical therapist, etc.

[0018] Furthermore, this method takes into account the fact that humans are social beings and naturally improve their posture by looking at their own reflection in a mirror. In other words, conventional methods have not focused on this point. Specifically, this method uses an engineering method to digitally display one's own image (the avatar AU for the user U) in real space, allowing the user U to check their posture while moving in real space in real time. By applying this method, the user U can improve their posture unconsciously, rather than being told to "improve their posture." Regarding this, the inventor has demonstrated through effectiveness evaluation experiments that this method is effective in improving walking posture.

[0019] In summary, unlike conventional techniques, the present technique shown in FIG. 1 displays the user's own posture (e.g., an avatar AU in the same posture as the user U) in 3DCG video, allowing the user U to see his or her own back, which cannot be seen in a mirror in real space. That is, as shown in FIG. 1 , the user U can perform his or her own movements while viewing the user U (the avatar AU) as a 3DCG (hologram) in real space. As will be described later, the avatar AU is not limited to the example shown in FIG. 1 , and its body shape can be changed based on weight and bone density data while keeping the skeleton intact. The avatar AU can also be dressed in clothing and decorated according to the purpose. Furthermore, weights w can be assigned to position coordinates and joint angles as needed, making it possible to control the body movements of the user U (the avatar AU) according to the purpose.

[0020] Next, an example of the configuration of an embodiment of an information processing system that can realize the method of Fig. 1, i.e., an information processing system of the present invention, will be described. Fig. 2 is a schematic diagram showing an example of the configuration of an embodiment of an information processing system of the present invention. Fig. 3 is a block diagram showing an example of the configuration of an embodiment of an information processing system of the present invention.

[0021] The information processing system of this embodiment is configured, for example, as shown in Figures 2 and 3, by connecting a control device 1, a wireless motion capture suit 2, and MR goggles 3 to a network N such as a wireless LAN using Wi-Fi (registered trademark).

[0022] The control device 1 is an information processing device configured by a PC or the like as described above, and has a configuration as shown in Fig. 4. Fig. 4 is a block diagram showing an example of the hardware configuration of the control device in the information processing system of Figs. 2 and 3.

[0023] The control device 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an input unit 16, an output unit 17, a memory unit 18, a communication unit 19, and a drive 20.

[0024] The CPU 11 executes various processes according to programs recorded in the ROM 12 or programs loaded from the storage unit 18 into the RAM 13. The RAM 13 also stores data and the like necessary for the CPU 11 to execute various processes.

[0025] The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface 15 is also connected to this bus 14. An input unit 16, an output unit 17, a storage unit 18, a communication unit 19, and a drive 20 are connected to the input / output interface 15.

[0026] The input unit 16 is configured with, for example, a keyboard and outputs various information. The output unit 17 is configured with, for example, a display such as an LCD, a speaker, and the like and outputs various information as images and sounds. The storage unit 18 is configured with, for example, a DRAM (Dynamic Random Access Memory) and stores various data. The communication unit 19 communicates with other devices (for example, the wireless motion capture suit 2 and MR goggles 3 in FIGS. 2 and 3 ) via a network N including a wireless LAN using Wi-Fi (registered trademark).

[0027] Removable media 31, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is appropriately attached to the drive 20. Programs read from the removable media 31 by the drive 20 are installed in the storage unit 18 as needed. The removable media 31 can also store various data stored in the storage unit 18 in the same way as the storage unit 18.

[0028] 2 and 3 , the wireless motion capture suit 2 is configured as shown in FIG. 5 . FIG. 5 is a diagram illustrating an example of the external configuration and sensor positions of the wireless motion capture suit in the information processing systems of FIGS. 2 and 3 . When the wireless motion capture suit 2 is worn over the entire body of the user U, it acquires IMU sensor data about the user U and transmits the data to the control device 1 via a network N such as a wireless LAN. As will be described in detail later, the control device 1 measures time-series data for the position coordinates (x, y, z) and Euler angles (φ, θ, ψ) of each axis of 36 skeletal positions (such as the circular positions in the diagram on the right side of FIG. 5 ) based on the IMU sensor data transmitted from the wireless motion capture suit 2. Based on this measured data, a 3DCG image (e.g., in FBX format) showing the entire body of the avatar AU in a predetermined posture (e.g., the same posture) based on the posture of the user U is generated in real time and transmitted to the MR goggles 3.

[0029] 2 and 3 , the MR goggles 3 are configured as shown in FIG. 6 . FIG. 6 is a diagram showing an example of the external configuration of the MR goggles in the information processing systems of FIGS. 2 and 3 . As shown in FIG. 6 , the MR goggles 3 are worn on the head of the user U, and project a 3D CG (avatar AU) transmitted from the control device 1 onto a semi-transparent display unit based on detection information from onboard sensors such as a visible light camera, an infrared camera, a depth sensor, an IMU, and a camera (not shown). This allows the user U to visually perceive the avatar AU (3D hologram) as if it were present in real space visible beyond the semi-transparent display unit. The MR goggles 3 also have an audio output unit including earphones and speakers, and output rhythmic sounds (see FIG. 16 ), which will be described later, from the audio output unit.

[0030] As described above, the control device 1 executes a series of processes, through cooperation between various hardware and various software, to generate 3DCG images (avatar AU) in real time based on the IMU sensor data transmitted from the wireless motion capture suit 2, and transfers the images to the MR goggles 3 for display. An example of the functional configuration of the control device 1 for executing such a series of processes is shown in Fig. 7. Fig. 7 is a functional block diagram showing an example of the functional configuration of a control device having the hardware configuration of Fig. 4.

[0031] The CPU 11 of the control device 1 functions as a measurement unit 51, a data processing unit 52, a 3DCG generation unit 53, a rhythm sound generation unit 54, and a 3DCG display and audio output unit 55. The 3DCG generation unit 53 is provided with a self 3D generation unit 531, an alert generation unit 532, and an other person 3D generation unit 533.

[0032] Based on the IMU sensor data transmitted from the wireless motion capture suit 2, the measurement unit 51 measures time-series data for the position coordinates (x, y, z) and Euler angles (φ, θ, ψ) of each axis of the coordinate data for 36 skeletal positions of the user U (e.g., the circle positions in the right diagram of FIG. 5 ). The data measured by the measurement unit 51 in this manner is hereinafter referred to as "measured IMU sensor time-series data." FIG. 8 is a diagram illustrating an example of a coordinate system for IMU sensor data measured by a control device having the functional configuration of FIG. 7 . FIG. 8A illustrates an example of three axes (x-axis, y-axis, and z-axis) for the position coordinates (x, y, z) of the coordinate data for 36 skeletal positions (e.g., the circle positions in the right diagram of FIG. 5 ). FIG. 8B illustrates an example of the Euler angles (φ, θ, ψ) of the 36 skeletal positions (e.g., the circle positions in the right diagram of FIG. 5 ).

[0033] The data processing unit 52 performs data processing required to generate 3DCG using the measured IMU sensor time-series data of the user U. Specifically, for example, as a first data processing, the data processing unit 52 performs calculation processing required to generate 3DCG showing an avatar AU in the same posture as the user U (an avatar AU in which the user U is faithfully reproduced) based on the measured IMU sensor time-series data of the user U. Furthermore, as a second data processing as needed, the data processing unit 52 assigns a weight w to the position coordinates of specific skeletal positions and joint angles (Euler angles) obtained from the measured IMU sensor time-series data of the user U. In this way, by assigning the weight w to the position coordinates of specific skeletal positions and joint angles, it becomes possible to control the body movement of the avatar AU according to the purpose. Furthermore, as a third data processing, the data processing unit 52 calculates a difference e between the left and right posture of the user U and a sway s of the head, and when at least one of the difference e and the sway s exceeds a threshold θ, executes calculation processing required to generate a 3DCG indicating an alert and issues a command (hereinafter referred to as an "alert display command") to display the alert together with an avatar AU of the user U. Furthermore, as a fourth data processing, the data processing unit 52 calculates a difference between the left and right foot movements of the user U (a difference in rhythm between the left and right feet and a difference in the timing at which the left and right feet land on the ground).

[0034] The 3DCG generation unit 53 converts the data obtained as a result of the data processing by the data processing unit 52 into data necessary for generating a hologram, thereby generating 3DCG.

[0035] Specifically, the self 3D generation unit 531 generates a 3DCG image of the avatar AU in the same posture as the user U, based on the result of the first data processing by the data processing unit 52. Furthermore, by using the result of the second data processing by the data processing unit 52, the self 3D generation unit 531 can also control the body movements of the avatar AU and generate a 3DCG image of the avatar AU whose posture has been changed based on this control.

[0036] The alert generation unit 532 functions when an alert display command is issued by the data processing unit 52, and generates a 3DCG image of the alert based on the result of the second data processing by the data processing unit 52. Although details will be described later with reference to Fig. 13 , by displaying the alert together with the avatar AU of the user U, the user U can correct his / her posture in response to the alert.

[0037] The other person 3D generation unit 533 generates a 3DCG image of an avatar (hereinafter referred to as "other person avatar") of a person different from the user U assuming a predetermined posture based on the results of the first data processing performed by the data processing unit. For example, if the other person is a model for the user U and the predetermined posture is an ideal posture, a model avatar assuming the ideal posture is generated as the other person avatar. As will be described in detail later with reference to FIGS. 14 and 15 , by displaying the other person avatar together with the user U's avatar AU, the user U can correct their posture by comparing themselves with the other person (e.g., an ideal person). Furthermore, the other person 3D generation unit 533 can use the results of the second data processing performed by the data processing unit 52 to control the body movement of the other person avatar and generate a 3DCG image of the other person avatar with a changed posture based on the control.

[0038] Based on the result of the fourth data processing by the data processing unit, the rhythmic sound generation unit 54 generates a "correct rhythmic sound" to be presented together with the 3DCG of the avatar AU and alert of the user U. Details will be described later with reference to Fig. 16 , but by outputting the correct rhythmic sound, the user U can correct his or her walking rhythm.

[0039] The 3DCG display and audio output control unit 55 transfers an image of the 3DCG (including at least the avatar AU of the user U, and further including alerts and other people's avatars as necessary) generated by the 3DCG generation unit 53 to the MR goggles 3, thereby executing control to display the 3DCG on the display unit of the MR goggles 3. Furthermore, the 3DCG display and audio output control unit 55 transfers the rhythmic sounds generated by the rhythmic sound generation unit 54 to the MR goggles 3, thereby executing control to output the rhythmic sounds from the audio output unit (earphones or speakers) of the MR goggles 3.

[0040] In summary, the information processing system of this embodiment, which includes the control device 1 of FIG. 4 , the wireless motion capture suit 2 of FIG. 5 , and the MR goggles 3 of FIG. 6 , has the functional configuration of FIG. 7 . It utilizes mixed reality technology and is based on a mechanism that enables a user U wearing the MR goggles 3 to perform movements such as walking while viewing a self-image (3DCG of an avatar AU) visualized in real space in real time. The projected 3DCG of the avatar AU is reproduced in real time based on the position coordinates and joint angles of the skeletal structure measured using sensory technology. Using this basic mechanism, the information processing system of this embodiment can realize self-body visualization and interaction technology in a mixed reality environment, as well as skill acquisition and functional recovery methods that accommodate individual differences in the mixed reality environment. The inventors have verified that applying the information processing of this embodiment results in improvements in the user U's gait and walking pattern, and this system is expected to be useful in skill acquisition, rehabilitation, and other areas for the user U in the future.

[0041] That is, the information processing system of this embodiment has the following first to fourth features, and these features can be used in a variety of different ways.

[0042] The first feature is a feature based on the above-described basic mechanism (basic system). According to the first feature, unlike conventional methods, by displaying the user (avatar AU) in 3DCG, the user U can see the backside of the user U, which cannot be seen in a real mirror. The user U can perform his / her own movements while viewing the 3DCG (hologram) version of the user (avatar AU) in real space. Here, the avatar AU's body shape can be changed based on weight and bone density data while keeping the skeleton intact. The avatar AU can also wear clothing and decorations according to the purpose. Furthermore, weights w can be assigned to position coordinates and joint angles as needed, making it possible to control the user (avatar AU)'s physical movements according to the purpose.

[0043] The second feature is that, based on the above-described basic mechanism (basic system), an alert can be displayed. The information processing system of this embodiment having the second feature can evaluate the user's posture during movement in real time and display a 3DCG alert on the MR goggles 3 together with the 3DCG of the user U's avatar AU as feedback for corrections. For example, it becomes possible to display an alert for a difference in shoulder position, head shaking, etc.

[0044] The third feature is that, based on the above-described basic mechanism (basic system), multiple people (multiple avatars) can be presented. The information processing system of this embodiment, which has the third feature, can simultaneously project an ideal self-image or an image of a model other avatar, along with the current self (user U's avatar AU), onto the MR goggles 3. User U can improve his / her posture by comparing the images of himself / herself (avatar AU) and the other (other avatar). Specifically, for example, the movements of a professional soccer player can be recorded, and a 3DCG of the other avatar representing the soccer player can be generated as a "model movement." Images of the other avatar and the user (user U's avatar AU) can be displayed side by side in real time on the MR goggles 3. This allows user U to objectively compare his / her own movements with the model's movements.

[0045] The fourth feature is that rhythmic sounds can be presented based on the above-described basic mechanism (basic system). The information processing system of this embodiment, which has the fourth feature, can calculate the difference between the left and right foot movements of the user U (the difference in rhythm between the left and right feet, the difference in the timing of foot contact between the left and right feet), and present the correct rhythmic sounds together with 3DCG visual information (avatar AU and alert).

[0046] Hereinafter, examples in which the first to fourth features of the information processing system of this embodiment are utilized will be described with reference to the drawings as appropriate.

[0047] The first example is, as described above with reference to FIG. 1A , a 3DCG (avatar AU) of the user U walking in real space is displayed in front of the user U in the same posture. By displaying the user U (avatar AU) in 3DCG, the user U can see the back side of the user U, which cannot be seen in a mirror in real life. The user U can perform his or her own movements while looking at the 3DCG (hologram) version of the user U in real space. The size and sense of distance of the 3DCG can be adjusted.

[0048] The second example is a case in which the body shape of a self-model (user U's avatar AU) can be changed, as shown in FIG. 9 . FIG. 9 is a diagram illustrating an overview of an example utilizing the first feature based on the basic system, among the features of the information processing system of FIG. 2 including the control device having the functional configuration of FIG. 7 . As shown in FIG. 9 , the body shape of the avatar AU can be freely changed to avatar AU1, avatar AU2, and avatar AU3 without changing the position coordinates and joint angles of the skeleton. For example, the avatar AU can be visualized in a state where the body shape has been changed based on weight / body fat data. Furthermore, it is possible to freely dress and decorate the 3D model (user U's avatar AU or another person's avatar) using existing software, etc.

[0049] The third example is shown in FIG. 10 , in which a user's own 3DCG image (avatar AU) performs a target movement. FIG. 10 is a diagram outlining an example different from FIG. 9 , which utilizes a first feature based on the basic system, among the features of the information processing system of FIG. 2 including the control device having the functional configuration of FIG. 7 . As shown in FIG. 10 , the user U can practice baseball form. Specifically, for example, the information processing system of this embodiment applies skeletal information and motion information of an ideal professional player to the 3DCG of the user U's own avatar AU. That is, the user U's ideal movements are presented by the avatar AU. The user U performs movements based on the movements visualized using the 3DCG of the avatar AU in which the user's skeleton is reproduced. That is, the user U can actually practice baseball image training using the 3DCG of the user U's avatar AU, taking into account the user's own skeleton.

[0050] The fourth example is a case in which the image of the user's own 3DCG (avatar AU) is made to perform an action that the user wishes to avoid, as shown in FIG. 11 . FIG. 11 is a diagram showing an overview of an example that utilizes a first feature based on the basic system, among the features of the information processing system of FIG. 2 including the control device having the functional configuration of FIG. 7 , and is different from FIGS. 9 and 10 . As shown in FIG. 11 , it is possible to reproduce an image of the user U falling (presenting an image of the user's own avatar AU falling), in other words, to display the risk of muscle weakness, etc. The information processing system of this embodiment reproduces a fall by presenting to the user U an image of the 3DCG of the avatar AU, in which the user U's skeleton is reproduced, falling. This allows the user U to respond to the risk of falling.

[0051] The fifth example is a case in which a weight w is assigned to the position coordinates and joint angles of a self-model (user U's avatar AU), as shown in FIG. 12 . FIG. 12 is a diagram illustrating an example that utilizes a first feature based on the basic system, among the features of the information processing system of FIG. 2 including the control device having the functional configuration of FIG. 7 , and is different from the examples of FIGS. 9 to 11 . The information processing system of this embodiment uses a weight w for position coordinates and joint angles to display an avatar AU with a posture that is worse than the user U's current actual posture (posture based on the current position coordinates and joint angles), or conversely, to display an avatar AU with a better posture (ideal appearance). That is, the information processing system of this embodiment uses a weight w for the position coordinates and joint angles of a specific skeletal position to display an avatar AU with poor posture emphasized, or conversely, to display an avatar AU with good posture emphasized. Such emphasis allows the user U to make quick corrections and achieve psychological benefits. For example, emphasizing good posture can help the user U set goals and increase their motivation.

[0052] The sixth example is a case in which an alert is displayed together with a self-model (user U's avatar AU), as shown in FIG. 13 . FIG. 13 is a diagram illustrating an overview of an example utilizing the second feature of the information processing system of FIG. 2 , which includes a control device having the functional configuration of FIG. 7 , namely, displaying an alert based on a basic system. The information processing system of this embodiment can evaluate the user's posture during movement in real time and display an alert indicating corrections as feedback superimposed on the corrections of the user's avatar AU using 3DCG. For example, if there is a difference in the left and right shoulders, the shoulders are the corrections that need to be made, and an alert is displayed at the shoulder of the avatar AU. Furthermore, if there is head shaking, the head is the corrections that need to be made, and an alert is displayed at the head of the avatar AU. Specifically, the example of FIG. 13 illustrates a case in which the right shoulder is higher than the left shoulder by more than the threshold θ. Therefore, an alert AM indicating that the left shoulder of the avatar AU is the correction that needs to be made is displayed at the left shoulder, and an alert AM calling attention is displayed above it. Such visualization of the alert AM provides visual feedback to the user U. This enables the user U to perform a corrected action, for example, to perform an action of dropping the right shoulder and aligning it with the left shoulder in the example of FIG.

[0053] The seventh example, as shown in FIG. 14 , is the first example of displaying 3DCG of multiple people. FIG. 14 is a diagram illustrating an overview of an example utilizing the third feature of the information processing system of FIG. 2 , which includes a control device having the functional configuration of FIG. 7 , namely, displaying 3DCG of multiple people based on the basic system. The information processing system of this embodiment can present the user (user U's avatar AU) and a model (other avatar) as a model to emulate side by side. The example of FIG. 14 illustrates a case in which a professional golfer is used as the model to emulate. As shown in FIG. 14 , the information processing system of this embodiment displays a 3DCG of an avatar AU that represents the user U's own movements alongside a 3DCG of another avatar AG that represents the movements of a professional golfer. The user U objectively compares the movements and adjusts his or her own movements. That is, the user U can become aware of corrections by visually comparing the movements of the avatar AU and the other avatar AG. The user U can then perform the corrected movements. 14 is merely an example, and can be widely applied to sports other than golf, dance, etc. Furthermore, the acquisition of skills other than sports can also be easily realized by employing a model (another person's avatar) that serves as a model for the skill to be acquired.

[0054] The eighth example, as shown in FIG. 15 , is a third example of displaying 3DCG images of multiple people. FIG. 15 is a diagram outlining an example different from that shown in FIG. 14 , utilizing the third feature of the information processing system of FIG. 2 , including the control device having the functional configuration of FIG. 7 , namely, displaying 3DCG images of multiple people based on the basic system. The example of FIG. 15 illustrates a case in which an avatar representing the ideal appearance of user U is used as a model (other avatar). As such, the other avatar can naturally represent a person different from user U in real space, or it can be a copy of user U. As shown in FIG. 15 , the information processing system of this embodiment displays a 3DCG image of an avatar AUG representing the current movements of user U, alongside a 3DCG image of an other avatar AUR representing the ideal appearance of user U. Here, the other avatar AUR representing the ideal appearance of user U is an avatar with the same skeleton as user U, in which the movements of an ideal professional golfer in real space or an ideal humanoid object in virtual space are applied to an avatar with the same skeleton as user U.

[0055] The ninth example is a rhythmic sound presentation example, as shown in FIG. 16 . FIG. 16 is a diagram outlining an example utilizing the fourth feature of the information processing system of FIG. 2 , which includes a control device having the functional configuration of FIG. 7 , that presents rhythmic sounds based on a basic system. The data processing unit 52 of the control device 1 calculates the difference between the left and right foot movements of the user U (the difference in rhythm between the left and right feet and the difference in foot contact timing between the left and right feet). The rhythmic sound generation unit 54 generates the correct rhythmic sound based on the calculation results. The 3DCG display / audio output control unit 55 presents the correct rhythmic sound along with a 3DCG visual alert. If the user U has a difference in shoulder height between the left and right and a discrepancy in the timing of the left and right foot contact, the user U listens to the rhythmic sound and adjusts the timing of the left and right foot contact. In this case, by displaying the 3DCG arm swing of the avatar AU in a larger scale, the user U's own arm movements also become larger, making it easier to keep up with the rhythm. In this way, audiovisual feedback can be used to correct movement control so that it can be controlled correctly.

[0056] Furthermore, below, with reference to the drawings as appropriate, examples that utilize the first to fourth features of the information processing system of this embodiment, which are different from the first to ninth examples described above, will be described.

[0057] The tenth case is a walking rehabilitation system (one embodiment of the information processing system in Figure 2) that uses AI object recognition and responds to the results of that object recognition. In other words, the tenth case is an example of walking rehabilitation that not only grasps the situation in the real environment and calls attention to it, but also is suited to that real environment and adapts to changes in walking posture (gait) and walking rhythm.

[0058] The basic algorithm of the tenth example is as follows. The user U wears the MR goggles 3 as shown in FIG. 6 and the wireless motion capture suit 2 as shown in FIG. 5. As described above, the measurement unit 51 (FIG. 7) of the control device 1 measures time-series data of the position coordinates (x, y, z) and Euler angles (φ, θ, ψ) of each axis of the coordinate data of 36 skeletal positions (such as the circular positions in the right diagram of FIG. 5) based on the IMU sensor data transmitted from the wireless motion capture suit 2. The data processing unit 52 and the self 3D generation unit 533 (FIG. 7) of the control device 1 construct a 3DCG image (e.g., in FBX format) showing the entire body of the avatar AU in a predetermined posture (e.g., the same posture) based on the posture of the user U, based on the data measured by the measurement unit 51. Here, the avatar AU may be one person or two people (with different postures). Alternatively, there may be two people: the avatar AU and another avatar. In this case, the other person 3D generation unit 533 constructs a 3DCG image (e.g., in FBX format) showing the entire body of the other person avatar. That is, one or two people are constructed as humanoid holograms. Furthermore, in the tenth example, the control device 1 performs "object detection" and "object recognition" as shown in FIG. 17 on objects in front of the control device 1's eyes in real space via the camera of the MR goggles 3. The processes related to these "object detection" and "object recognition" are executed by a predetermined functional block (not shown in FIG. 7 ) that functions in the CPU 11 of the control device 1. Hereinafter, such a predetermined functional block will be referred to as the "object detection / recognition unit." FIG. 17 illustrates an example of the results of object detection and object recognition, which utilizes features of the information processing system of FIG. 2 including the control device having the functional configuration of FIG. 7 and is used in an example different from those of FIGS. 9 to 14 . In the example of FIG. 17 , three objects present in real space are detected and recognized as a dog, a bicycle, and a car, respectively. The methods of "object detection" and "object recognition" may be any method that uses images captured by a camera, and any of the various methods that exist at the time of filing this application or that will be introduced in the future may be applied.

[0059] The walking rehabilitation system of the tenth example (one form of the information processing system of FIG. 2) has the following first to sixth functions.

[0060] As shown in FIG. 18 , the first function is a function of switching an object recognition mode appropriate for the user U's posture based on the posture measurement results, in accordance with the user U's motion posture information (walking, stationary, crouching, etc.). This function is performed by the object detection and recognition unit of the control device 1. FIG. 18 is a diagram showing an overview of mode switching in a case where the results of object detection and object recognition as shown in FIG. 17 are used, among cases utilizing the features of the information processing system of FIG. 2 including the control device having the functional configuration of FIG. 7 . In FIG. 18 , the posture measurement results refer to time-series data of the position coordinates (x, y, z) of each skeleton position of the user U and the Euler angles (φ, θ, ψ) of each axis, obtained by the measurement unit 51 ( FIG. 7 ).

[0061] Specifically, for example, when the user U's motion posture is walking, the user U himself / herself is moving, and therefore the object detection / recognition unit switches to an object recognition mode in which the following actions (a) to (c) are taken, regardless of whether the object is moving or not. Note that this mode in which all objects are recognized is hereinafter referred to as the "first object recognition mode." (a) The following action is taken for the object type (label). For example, if a specified object (such as a car or a bicycle) is recognized, the alert generation unit 532 ( FIG. 7 ) is activated to display an alert message. Note that people and specified specific objects are exceptions. (b) The following action is taken for the object size. For example, if the object size is equal to or greater than a first value, the alert generation unit 532 ( FIG. 7 ) is activated to display an alert message. (c) The following action is taken for the object movement. For example, if a person or a specified specific object is recognized, the alert generation unit 532 ( FIG. 7 ) is activated to display an alert message. Furthermore, if an object is moving and the specified object is recognized, the alert generation unit 532 (FIG. 7) is activated to display an alert to that effect.

[0062] Furthermore, for example, when the user U's motion posture is upright (stationary), the user U is stationary, and the object detection / recognition unit switches to an object recognition mode that performs the following actions (a) to (c) to recognize moving objects. Note that this mode for recognizing moving objects is hereinafter referred to as the "second object recognition mode." (a) The following action is taken with respect to the type (label) of the object. For example, if a specified object (such as a car or a bicycle) is recognized, the alert generation unit 532 ( FIG. 7 ) is activated to display an alert message. Note that people and specified specific objects are exceptions. (b) The following action is taken with respect to the size of the object. For example, if the size of the object is equal to or greater than a second value (a value greater than the first value), the alert generation unit 532 ( FIG. 7 ) is activated to display an alert message. (c) The following action is taken with respect to the movement of the object. For example, if a person or a specified specific object is recognized, the alert generation unit 532 ( FIG. 7 ) is activated to display an alert message. Furthermore, if an object is moving and the specified object is recognized, the alert generation unit 532 (FIG. 7) is activated to display an alert to that effect.

[0063] Furthermore, for example, if the user U's motion posture is a crouching posture or another specific posture, the object detection / recognition unit determines that this is abnormal and stops the object recognition mode.

[0064] The first function of the walking rehabilitation system of the tenth example (one embodiment of the information processing system of FIG. 2) has been described above. Next, the second function of the walking rehabilitation system of the tenth example (one embodiment of the information processing system of FIG. 2) will be described.

[0065] The second function is a function for (hazard prediction), calling attention, and presenting an alert (auditory and visual) when there is a large change in the user U's movement posture (the amount of change in the position coordinates of each skeleton in the time-series data) based on the results of posture measurement of the user U. The second function is performed by the measurement unit 51, data processing unit 52, alert generation unit 532, rhythmic sound generation unit 54, and 3DCG display / audio output control unit 55 of the control device 1, which are shown in FIG. 7 above. In other words, when a person (user U) senses an "obstacle" or "dangerous object" in front of them, their behavior is likely to be affected by the object. Here, the measurement device 1 is capable of measuring the posture of the user U and can detect changes in the user U's posture faster than the camera of the MR goggles 3 can detect objects. Therefore, as a second function, if the measurement unit 51 and the data processing unit 52 determine that the user U has a large change in posture or that the user U is not walking steadily (if the walking rhythm is unstable), the alert generation unit 532 or the rhythmic sound generation unit 54 generates data for calling attention or providing an alert (auditory or visual).The 3DCG display and audio output control unit 55 transfers the data to the MR goggles 3, thereby presenting the attention or providing an alert (auditory or visual) to the user U through the MR goggles 3.

[0066] The second function of the walking rehabilitation system of the tenth example (one embodiment of the information processing system of FIG. 2) has been described above. Next, the third function of the walking rehabilitation system of the tenth example (one embodiment of the information processing system of FIG. 2) will be described.

[0067] The third function is a function that, if a problem is found in the stability of the user U's walking rhythm based on the results of posture measurement (a stable gait results in a constant cycle, but if this is not the case, for example, if there is a deviation to the left or right, or if dragging of the foot is observed), determines that assistance with walking is necessary, and supports the walker by recognizing obstacles and dangerous objects with a high level of accuracy and communicating the obstacle status by voice. In other words, the third function can be understood as a function that substitutes the role of an assistant with voice. The third function is performed by the measurement unit 51, data processing unit 52, alert generation unit 532, 3DCG display / voice output control unit 55, object detection / recognition unit, and voice generation unit (not shown) of the control device 1, all of which are shown in FIG. 7 above.

[0068] The third function of the walking rehabilitation system of the tenth example (one embodiment of the information processing system of FIG. 2 ) has been described above. Next, the fourth function of the walking rehabilitation system of the tenth example (one embodiment of the information processing system of FIG. 2 ) will be described.

[0069] The fourth function is a function of switching the walking rehabilitation mode when a moving object is detected based on the object detection result (in the case of an environmental change). The fourth function is performed by the measurement unit 51, data processing unit 52, self 3D generation unit 631, alert generation unit 532, other person 3D generation unit 533, rhythmic sound generation unit 54, and 3DCG display / audio output control unit 55, as well as the object detection / recognition unit and audio generation unit (not shown) of the control device 1, all of which are shown in FIG. 7 above.

[0070] Here, the walking rehabilitation mode includes the following first to fifth modes. The first mode is a mode for displaying one person, in which an avatar AU of the user U's own posture (skeletal coordinates) is displayed as is. The second mode is a mode for displaying one person, in which an avatar AU in which the user U's posture has been corrected based on the user U's own skeleton. The third mode is a mode for displaying one person, in which another avatar serves as a walking example. The fourth mode is a mode for displaying two people, in which an avatar AU of the user U's own posture (skeletal coordinates) (avatar AU of the first mode) is displayed in combination with an avatar AU in which the posture has been corrected based on the user U's own skeleton (avatar AU of the second mode). The fifth mode is a mode for displaying two people, in which an avatar AU of the user U's own posture (skeletal coordinates) (avatar AU of the first mode) is displayed in combination with another avatar serving as a walking example (another avatar of the third mode). The sixth mode is a mode in which two people are displayed, and displays a combination of an avatar AU (second mode avatar AU) in which the posture distortion of the user U has been corrected based on the skeleton of the user U, and another person's avatar (third mode avatar) that serves as a walking example. Furthermore, for each of the first to sixth modes, a distinction may be made between cases in which rhythmic footstep sounds are added and cases in which they are not added.

[0071] Specifically, as the fourth function, for example, when an object is moving but the direction of the movement is not facing the user U, the 3DCG display / audio output control unit 55 continues the rehabilitation mode as is if the rehabilitation mode is the first mode, or changes the rehabilitation mode to the first mode if the rehabilitation mode is one of the second to sixth modes, and then transfers an image of an avatar corresponding to the rehabilitation mode to the MR goggles 3, thereby executing control to display the 3DCG on the display unit of the MR goggles 3. At this time, a predetermined first melody sound is generated by a melody sound generation unit (not shown) functioning in the CPU 11 of the control device 1. The 3DCG display / audio output control unit 55 transfers the first melody sound to the MR goggles 3, thereby executing control to output the first melody sound from the audio output unit (earphones or speakers) of the MR goggles 3. Furthermore, for example, when an object is moving and the direction of the movement is facing the user, the 3DCG display / audio output control unit 55 stops the rehabilitation mode and prohibits the transfer of the image of the avatar to the MR goggles 3. As a result, the display of the 3DCG avatar is erased from the display unit of the MR goggles 3. At this time, a predetermined second melody sound is generated by a melody sound generation unit (not shown) functioning in the CPU 11 of the control device 1. The 3DCG display and audio output control unit 55 transfers the second melody sound to the MR goggles 3, thereby executing control to output the second melody sound from the audio output unit (earphones or speakers) of the MR goggles 3. Note that in any of the above cases, if the recognized object is an already registered specific object or a person, the above series of processes is not executed.

[0072] The fourth function of the walking rehabilitation system of the tenth example (one embodiment of the information processing system of FIG. 2 ) has been described above. Next, the fifth function of the walking rehabilitation system of the tenth example (one embodiment of the information processing system of FIG. 2 ) will be described.

[0073] The fifth function is a function of switching the walking rehabilitation mode when an object is detected in the short distance based on the object detection result and when no change is detected in the walking posture (gait) and walking rhythm of the user U. The fifth function is performed by the measurement unit 51, data processing unit 52, self 3D generation unit 631, alert generation unit 532, other person 3D generation unit 533, rhythmic sound generation unit 54, and 3DCG display / audio output control unit 55, as well as the object detection recognition unit and audio generation unit (not shown) of the control device 1, all of which are shown in FIG. Specifically, for example, if an object is detected in the short distance based on the object detection result and no change is detected in the walking posture (gait) and walking rhythm of the user U, the 3DCG display / audio output control unit 55 continues the rehabilitation mode as is if the rehabilitation mode is the first mode, or changes the rehabilitation mode to the first mode if the rehabilitation mode is one of the second to sixth modes, and then transfers an image of an avatar corresponding to the rehabilitation mode to the MR goggles 3, thereby executing control to display the 3DCG on the display unit of the MR goggles 3. The 3DCG display / audio output control unit 55 also transfers data prompting attention to the MR goggles 3, thereby prompting the user U to pay attention via the MR goggles 3. Here, if the data prompting attention is data prompting auditory attention, a melody sound for gait stability generated by a melody sound generation unit (not shown) is used. On the other hand, if the data prompting attention is data prompting visual attention, an alert image generated by the alert generation unit 532 is used.

[0074] The fifth function of the walking rehabilitation system of the tenth example (one embodiment of the information processing system of FIG. 2 ) has been described above. Next, the sixth function of the walking rehabilitation system of the tenth example (one embodiment of the information processing system of FIG. 2 ) will be described.

[0075] The sixth function is a function for stopping the walking rehabilitation system if, based on the object detection result, no object is detected in the close range but a change in walking posture (gait) and walking rhythm is detected. The sixth function is performed by the measurement unit 51, data processing unit 52, self 3D generation unit 631, alert generation unit 532, other person 3D generation unit 533, rhythmic sound generation unit 54, 3DCG display / audio output control unit 55, object detection recognition unit, and audio generation unit (not shown) of the control device 1, all of which are shown in FIG. 7 . Specifically, for example, if no object is detected in the close range but a change in walking posture (gait) and walking rhythm is detected, the 3DCG display / audio output control unit 55 stops the rehabilitation mode and prohibits the transfer of the avatar image to the MR goggles 3. As a result, the display of the 3DCG avatar is erased from the display unit of the MR goggles 3. Furthermore, the 3DCG display and audio output control unit 55 transfers data for calling attention to the MR goggles 3, thereby calling the attention of the user U via the MR goggles 3. Here, if the data for calling attention is data for calling attention auditorily, a melody sound for stabilizing walking, generated by a melody sound generating unit (not shown), is adopted. On the other hand, if the data for calling attention is data for calling attention visually, an alert video generated by the alert generating unit 532 is adopted.

[0076] The tenth example has been described above as an example utilizing the first to fourth features of the information processing system of this embodiment. Next, an eleventh example will be described as an example utilizing the first to fourth features of the information processing system of this embodiment.

[0077] The eleventh example is a case in which a virtual display of an obstacle is performed, as shown in FIG. 19 . Note that such a virtually displayed obstacle (digital information) will be referred to as a “digital obstacle” hereinafter to distinguish it from a real obstacle. FIG. 19 is a diagram illustrating an example that utilizes features of the information processing system of FIG. 2 , which includes a control device having the functional configuration of FIG. 7 , and is different from the examples of FIGS. 9 to 18 . The left side of FIG. 19 shows the state of real space RK (as seen by a user U not wearing MR goggles 3). No real obstacles (stones, etc.) exist in this real space RK. The right side of FIG. 19 shows the state of real space KK as seen through a translucent display unit by a user U not wearing MR goggles 3. Digital obstacles VS1 and VS2 (3D holograms) of different sizes exist in this real space KK. For ease of explanation, one or two avatars (user U's avatar AU or another person's avatar) are not depicted in the real space KK on the right side of FIG. 19 , but in reality, one or two avatars are present in front of user U. In this eleventh example, the 3DCG generation unit 53 ( FIG. 7 ) is provided with an obstacle generation unit (not shown in FIG. 7 ). The obstacle generation unit generates 3DCG images (digital information) of digital obstacles of different sizes (e.g., digital obstacles VS1 and VS2). The 3DCG display and audio output control unit 55 transfers the images of the 3DCG (one or two avatars (user U's avatar AU or another person's avatar) not shown in FIG. 19 , as well as digital obstacles VS1 and VS2, etc.) generated by the 3DCG generation unit 53 to the MR goggles 3, thereby controlling the display of the 3DCG on the display unit of the MR goggles 3. Although not shown, it is assumed that the user U can control the placement position and size of each digital obstacle with his or her hands.

[0078] Furthermore, by combining this with the tenth case, it is also possible to switch to an appropriate rehabilitation mode based on the walking situation (posture, walking rhythm) when an obstacle is digitally displayed, as shown in Fig. 20. Fig. 20 is a diagram showing an overview of mode switching in the case shown in Fig. 19, among cases that utilize the features of the information processing system of Fig. 2, which includes a control device having the functional configuration of Fig. 7. In Fig. 20, the posture measurement result means time-series data of the position coordinates (x, y, z) of each skeleton position of the user U and the Euler angles (φ, θ, ψ) of each axis, obtained by the measurement unit 51 (Fig. 7) in Fig. 7.

[0079] By adopting the eleventh example shown in FIG. 20 , it is possible to realize rehabilitation that is closer to a real environment. That is, the risk of falling is influenced not only by the muscle strength of a person (user U), but also by the person's (user U's) visual attention inhibition function, attention control, and the like. Therefore, a method that also trains the visual attention inhibition function, attention control, and the like while walking is effective. In particular, by not only training "visual attention" and "motor function," but also combining these in a comprehensive manner, it is possible to realize rehabilitation that is closer to a real environment.

[0080] Further, a specific flow of the eleventh case will be described below. First, in step S1, the user U specifies the size and position of a digital obstacle by himself / herself. When the size and position of the digital obstacle are specified by the user U, the control device 1 acquires the change in walking posture (gait) at that time (the difference in the change in the position coordinates (x, y, z) of the coordinate data of each skeletal position of the user U from time t to time t+1), and records the digital obstacles that caused a large change in posture. The recorded information is the size of the digital obstacles and their position coordinates. The control device 1 groups the digital obstacles that caused a large change in the posture of the user U based on the magnitude of their influence on the posture change, and creates a "difficulty" index tailored to the characteristics of the user U (individual).

[0081] In step S2, under the control of the control device 1, the user U performs walking rehabilitation in accordance with the difficulty index created in step S1. The type (mode) of walking rehabilitation is not particularly limited, but for example, the first to fifth modes described above in the fourth function of the tenth example can be adopted. Furthermore, if the user U's posture tends to deteriorate in response to the presentation of visual digital obstacles, muscle weakness may be observed. However, since muscle strength and posture are correlated, drastic changes in posture during movement or walking can indicate muscle weakness. If muscle strength needs improvement based on the posture results, the control device 1 can issue a notification to improve muscle strength. In other cases, the control device 1 can similarly determine whether the cause is visual information processing or walking in response to digital obstacles (coordination of vision and movement).

[0082] Here, when a digital obstacle is displayed, the control device 1 can issue an alert to call the user's attention if there is a large change in the user's posture. For example, the control device 1 can present a melody sound to the user U as the alert to call the user's attention.

[0083] Furthermore, the digital obstacles can be animated and can be changed into animals or characters. Such size and movement of the obstacles can only be realized with digital obstacles created using 3DCG. In other words, digital obstacles are obstacles that cannot be experienced in reality. The eleventh example is characterized in that walking rehabilitation can be performed using such digital obstacles. Another feature of the eleventh example is that the digital obstacles can be changed into animals or characters. By using characters such as pets that are familiar to the user U (individual), it becomes possible to train visual attention control during walking movements, like in a game, rather than as obstacles.

[0084] The eleventh example has been described above as an example utilizing the first to fourth features of the information processing system of this embodiment. Next, the twelfth example will be described as an example utilizing the first to fourth features of the information processing system of this embodiment.

[0085] The twelfth case is similar to the eleventh case in that it involves the virtual display of obstacles, but it also involves changing the environment in real space. That is, the twelfth case is a case in which, under the same conditions as the eleventh case, the environment in real space is changed, and walking rehabilitation is performed while digital obstacles (such as obstacles VS1 and VS2 in FIG. 19 ) corresponding to the changed environment are presented.

[0086] Here, the environment is constructed using a plurality of projectors as shown in Fig. 21. Fig. 21 is a diagram showing an overview of an example that utilizes features of the information processing system of Fig. 2, which includes a control device having the functional configuration of Fig. 7, and is different from Figs. 9 to 20. In the example of Fig. 21, a projector PF is used to project an environment in front of a user U, a projector PL is used to project an environment on the left side, and a projector PR is used to project an environment on the right side. Note that the number of projectors is not limited to three, and any number greater than three may be used.

[0087] In the twelfth case, similarly to the eleventh case, the 3DCG generation unit 53 (FIG. 7) is provided with an obstacle generation unit (not shown in FIG. 7), as well as an environmental image generation unit and an environmental image display control unit. The environmental image generation unit generates, for example, the following first to fourth patterns of environmental images as environmental images to be displayed on each of the projectors PF, PL, and PR. The first pattern is a large, empty sports field. The second pattern is a lecture with people passing by. The third pattern is a city with heavy traffic. The fourth pattern is a place that the user U has never been to (an unfamiliar place). The obstacle presentation conditions are the same as those in the eleventh case. The walking rehabilitation mode is the same as those in the tenth and eleventh cases.

[0088] By adopting the twelfth example, it becomes possible to provide rehabilitation tailored to each individual. That is, an index of the "difficulty" of walking can be constructed based on a combination of "environmental patterns" and "obstacles," and this index can be used to recommend walking rehabilitation based on a combination of "environmental patterns" and "obstacles." After each user has undergone sufficient rehabilitation under the environmental and obstacle conditions of a group deemed to be low in difficulty, the difficulty level can be gradually increased, making it possible to provide rehabilitation tailored to each individual.

[0089] Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope of achieving the object of the present invention are included in the present invention.

[0090] For example, in the above-described embodiment, a wireless motion capture suit 2 was employed, but this is not particularly limited, and any implementation form is possible as long as there are multiple IMU sensors placed at each of the multiple skeletal positions of the user U. However, as in this embodiment, it is preferable to employ a wireless motion capture suit 2 in which IMU sensors are pre-fixed at each of the positions that come into contact with each of the multiple skeletal positions of the user U, because the IMU sensors can be appropriately placed at the user U's skeletal positions simply by wearing the suit.

[0091] Furthermore, for example, in the above-described embodiment, MR goggles 3 are employed, but the present invention is not limited to this, and any device capable of projecting 3DCG (3D holograms) into real space will suffice. However, employing MR goggles 3, as in the present embodiment, is preferable because the user U can have 3DCG projected in front of their eyes simply by wearing the MR goggles on their head.

[0092] In the above-described embodiment, the communication between the control device 1, the wireless motion capture suit 2, and the MR goggles 3 is performed wirelessly, but this is not particularly limited, and at least some of the communication may be performed wired. However, as in the present embodiment, the communication is performed entirely wirelessly (wireless communication), which is preferable because it allows the user U to move freely.

[0093] For example, the above-described series of processes can be executed by hardware or software. In other words, the functional configuration of FIG. 7 is merely illustrative and is not particularly limited. That is, it is sufficient for the information processing system to have the functionality to execute the above-described series of processes as a whole, and the functional blocks and databases used to realize these functions are not particularly limited to the example of FIG. 7. Furthermore, the locations of the functional blocks and databases are not particularly limited to those shown in FIG. 7 and may be arbitrary. For example, at least some of the functional blocks and databases required to execute various processes may be transferred to the wireless motion capture suit 2 or the MR goggles 3. Furthermore, one functional block and database may be configured as hardware alone, software alone, or a combination thereof.

[0094] When a series of processes is executed by software, the programs constituting the software are installed onto a computer or the like from a network or a recording medium. The computer may be a computer incorporated into dedicated hardware. The computer may also be a computer capable of executing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0095] The recording medium containing such a program may be composed of not only a removable medium (not shown) that is distributed separately from the device main body in order to provide the program to users, etc., but also a recording medium that is provided to users, etc. in a state where it is pre-installed in the device main body.

[0096] In this specification, the steps describing the program recorded on the recording medium include not only processes that are performed in chronological order, but also processes that are not necessarily performed in chronological order but are performed in parallel or individually. Also, in this specification, the term "system" means an overall device composed of multiple devices or multiple means, etc.

[0097] In other words, an information processing system to which the present invention is applied is only required to have the following configuration, and can take various forms. That is, the information processing system to which the present invention is applied is an information processing system including: an IMU sensor group (e.g., the wireless motion capture suit 2 in FIGS. 2 and 5) including a plurality of IMU (Inertial Measurement Unit) sensors that can be arranged at the respective positions of a plurality of skeletons (e.g., the circular positions in the right diagram in FIG. 5) of a user (e.g., user U in FIG. 1), a 3DCG (3 Dimensional Computer Graphics) projection device (e.g., the MR goggles 3 in FIGS. 2 and 6) that projects a 3DCG (3D Computer Graphics) image into the real space in which the user exists in a state that the image can be seen by the user, and a control device (e.g., the control device 1 in FIGS. 2, 4, and 7) that communicates with the IMU sensor group and the 3DCG projection device to perform overall control, wherein the control device It is sufficient that the information processing system measures skeletal information of each of the plurality of skeletons ("measured IMU sensor time series data" consisting of the position coordinates and joint angles of the above-mentioned skeletons) based on time series data acquired from the plurality of IMU sensors arranged at the respective positions of the plurality of skeletons of the user, generates a 3DCG image showing a first avatar (e.g., avatar AU in Figure 1) taking a predetermined posture based on the measurement results, and executes control to project the 3DCG image from the 3DCG projection device (e.g., control to create a state for user U as shown in Figure 1 (A)).

[0098] This allows a user (e.g., user U) to perform his / her own movements while visually recognizing a first avatar (e.g., avatar AU) that reproduces his / her skeletal information.

[0099] The control device comprises: a measurement means (e.g., the measurement unit 51 in FIG. 7 ) that measures, as the skeletal information, time-series data of position coordinates of coordinate data of each of the user's multiple skeletons and Euler angles of joint axes, based on the time-series data transmitted from the IMU sensor group; a data processing means (e.g., the data processing unit 52 in FIG. 7 ) that executes, as first data processing, calculation processing required to generate 3DCG showing a first avatar (e.g., avatar AU in FIG. 1 in which user U is faithfully reproduced) taking the predetermined posture (e.g., the same posture) based on the posture of the user, based on the skeletal information of the user; and a 3DCG generation means (e.g., the 3DCG generation unit 53 having a self 3D generation unit 531 in FIG. 7 ) that generates a 3DCG image showing the first avatar taking the predetermined posture, based on a result of the first data processing by the data processing means. and a 3DCG projection means (for example, the 3DCG display and audio output control unit 55 in FIG. 7) that controls the projection of a 3DCG image representing the first avatar generated by the 3DCG generation means from the 3DCG device.

[0100] This makes it possible to build a mechanism, i.e., a basic system, that enables a user (e.g., user U) to perform actions such as walking while viewing a self-visualized self (e.g., a 3DCG representation of a first avatar such as avatar AU) visualized in real time in real space. As a result, the information processing system has the above-mentioned first feature based on the basic system. That is, unlike conventional methods, by displaying the self (first avatar) in 3DCG, the user can see the backside, which cannot be seen in a real mirror. The user can perform his or her own actions while viewing the 3DCG self (first avatar) in real space. Here, the first avatar's body shape can be changed based on weight and bone density data while keeping the skeleton unchanged. Furthermore, the first avatar can also wear clothing and decorations according to the purpose.

[0101] Furthermore, as a second data processing, the data processing means may assign weights to position coordinates of specific skeletal positions and Euler angles of joints obtained from the skeletal information of the user U, and the 3DCG generation means may use the results of the second data processing by the data processing means to control the body movements of the first avatar and generate a 3DCG image of the first avatar with the predetermined posture changed based on that control.

[0102] In this way, by assigning weights to the position coordinates and joint angles of specific skeletal positions, it becomes possible to control the body movements of the first avatar (e.g., avatar AU) according to the purpose (see, for example, Figure 12).

[0103] Furthermore, as a third data processing, the data processing means calculates the difference between the left and right posture of the user and the shaking of the head, and when at least one of the difference between the left and right posture and the shaking exceeds a threshold, executes the calculation processing required to generate 3DCG indicating an alert and issues a command (for example, referred to as the above-mentioned "alert display command") to display the alert together with the first avatar of the user, and the 3DCG generation means (for example, the alert generation unit 532 in Figure 7) can further generate a 3DCG image of the alert based on the result of the third data processing of the data processing means when the command is issued by the data processing means.

[0104] As a result, the information processing system can have a second feature of being able to display an alert, based on the above-mentioned basic system. The information processing system having the second feature can evaluate its own posture during movement in real time, and project the 3DCG of the alert together with the 3DCG of the user's first avatar onto a 3DCG projection device as feedback for correction (see, for example, FIG. 13 ).

[0105] Furthermore, the 3DCG generation means (e.g., the other person 3D generation unit 533 in FIG. 7) can generate a 3DCG image of a second avatar in which another person different from the user assumes a predetermined posture based on at least a result of the first data processing by the data processing means, and the 3DCG projection means can execute control to simultaneously project the 3DCG of the second avatar together with the 3DCG of the first avatar onto the 3DCG projection device.

[0106] This allows the information processing system, based on the basic system, to have a third feature of being able to present multiple people (multiple avatars). The information processing system having the third feature can simultaneously project, on a 3DCG projection device, an image of a second avatar, such as an ideal self-image or a model other avatar, along with the current self (a first avatar such as avatar AU). User U can improve his / her posture by comparing the image of the self (first avatar) with the image of the other (second avatar) (see, for example, FIG. 14 ). Note that the "other" indicated by the second avatar is a broad concept that includes not only a person different from the user in real space, but also a so-called copy of the user (see FIG. 15 ), as described above.

[0107] Furthermore, the data processing means may calculate a difference in the movement of the user's feet between the left and right sides as a fourth data processing operation, and the control device may further include rhythmic sound generation means (e.g., rhythmic sound generation unit 54 in FIG. 7) that generates a predetermined rhythmic sound to be presented together with the 3DCG based on the result of the fourth data processing operation of the data processing means.

[0108] This allows the information processing system, based on the basic system, to have a fourth feature of being able to present rhythmic sounds. The information processing system having the fourth feature can calculate the difference between the left and right movements of the user's feet (for example, the difference in rhythm between the left and right feet, or the difference in the timing of foot contact between the left and right feet), and present the correct rhythmic sounds together with 3DCG visual information (the first avatar or an alert) (see FIG. 16 ).

[0109] The control device may further include an object detection and recognition means for detecting and recognizing an object present in the real space based on an image of the real space acquired by the 3DCG projection device (for example, detecting and recognizing an object as shown in FIG. 17 ), and a first mode switching means for switching a mode of projection of the 3DCG image by the 3DCG projection means based on the result of the recognition of the object by the object detection and recognition means, or for switching a mode of recognition of the object by the object detection and recognition means based on the skeletal information of the user measured by the measurement means (for example, switching a mode as shown in FIG. 18 ).

[0110] Furthermore, the 3DCG generating means can further generate a 3DCG image showing an object (for example, digital obstacles VS1 and VS2 in FIG. 19) showing a predetermined target different from the first avatar, and the 3DCG projecting means can execute control to project, from the 3DCG device, the 3DCG images showing the first avatar and the object generated by the 3DCG generating means.

[0111] The information processing system may further include a display device (e.g., projectors PF, PL, PR in FIG. 21) that displays an environmental image showing a predetermined environment in real space, and the control device may further include: an environmental image generation means that generates the environmental image; and a display control means that executes control to display the generated environmental image on the display device, and the 3DCG generation means may further generate a 3DCG image showing the first avatar and the object that changes according to the predetermined environment represented by the environmental image.

[0112] DESCRIPTION OF SYMBOLS 1: Control device, 2: Wireless motion capture suit, 3: MR goggles, 11: CPU, 12: ROM, 13: RAM, 14: Bus, 15: Input / output interface, 16: Input unit, 17: Output unit, 18: Storage unit, 19: Communication unit, 20: Drive, 31: Removable media, 51: Measurement unit, 52: Data processing unit, 53: 3DCG generation unit, 54: Rhythm sound generation unit, 55: 3DCG display / audio output control unit, 531: Self 3D generation unit, 532: Alert generation unit, 533: Other person 3D generation unit, U: User, AU: Avatar, VS1, VS2: Digital obstacles

Claims

1. An information processing system including: an IMU sensor group including a plurality of IMU (Inertial Measurement Unit) sensors that can be placed at the positions of each of a user's multiple skeletal components; a 3DCG projection device that projects a 3DCG (3 Dimensional Computer Graphics) image into a real space in which the user exists in a state that the user can view the image; and a control device that communicates with the IMU sensor group and the 3DCG projection device and performs overall control, wherein the control device measures skeletal information of each of the multiple skeletons based on time-series data acquired from the multiple IMU sensors that are placed at the positions of each of the user's multiple skeletons, generates a 3DCG image showing a first avatar taking a predetermined posture based on the measurement results, and executes control to project the 3DCG image from the 3DCG projection device.

2. The information processing system according to claim 1, wherein the control device comprises: a measuring means for measuring, as the skeletal information, time series data of position coordinates of coordinate data of the position of each of the user's multiple skeletons and Euler angles of joint axes, based on the time series data transmitted from the IMU sensor group; a data processing means for executing, as first data processing, calculation processing required to generate 3DCG showing the first avatar in the predetermined posture based on the posture of the user, based on the skeletal information of the user; a 3DCG generation means for generating, based on a result of the first data processing by the data processing means, the 3DCG image showing the first avatar in the predetermined posture; and a 3DCG projection means for executing control to project, from the 3DCG device, the 3DCG image showing the first avatar, generated by the 3DCG generation means.

3. The information processing system according to claim 2, wherein the data processing means further performs second data processing by weighting position coordinates of specific skeletal positions and Euler angles of joints obtained from the skeletal information of the user U, and the 3DCG generation means uses a result of the second data processing by the data processing means to control the body movement of the first avatar and generate a 3DCG image of the first avatar with the specified posture changed based on that control.

4. The information processing system of claim 2, wherein the data processing means further calculates, as a third data processing, a difference between the left and right sides of the user's posture and a swaying of the head, and when at least one of the difference between the left and right sides and the swaying exceeds a threshold, executes the calculation processing required to generate a 3DCG indicating an alert and issues a command to display the alert together with the first avatar of the user; and when the command is issued by the data processing means, the 3DCG generation means further generates a 3DCG image of the alert based on a result of the third data processing of the data processing means.

5. The information processing system according to claim 2, wherein the 3DCG generation means further generates a 3DCG image of a second avatar of a person different from the user taking a predetermined posture based on at least a result of the first data processing by the data processing means, and the 3DCG projection means executes control to simultaneously project the 3DCG of the second avatar together with the 3DCG of the first avatar onto the 3DCG projection device.

6. The information processing system of claim 2, wherein the data processing means further calculates a difference between the left and right movements of the user's feet as a fourth data processing step, and the control device further comprises rhythmic sound generation means for generating a predetermined rhythmic sound to be presented together with the 3DCG based on the result of the fourth data processing by the data processing means.

7. The information processing system of claim 2, further comprising: an object detection and recognition means for detecting and recognizing an object present in a real space based on an image of the real space acquired by the 3DCG projection device; and a first mode switching means for switching a mode of projection of the 3DCG image by the 3DCG projection means based on a result of the recognition of the object by the object detection and recognition means, or switching a mode of recognition of the object by the object detection and recognition means based on the skeletal information of the user measured by the measurement means.

8. The information processing system according to claim 2, wherein the 3DCG generation means further generates a 3DCG image showing an object showing a predetermined target different from the first avatar, and the 3DCG projection means executes control to project, from the 3DCG device, the 3DCG images showing each of the first avatar and the object generated by the 3DCG generation means.

9. The information processing system according to claim 8, further comprising a display device for displaying an environmental image showing a predetermined environment in real space, wherein the control device further comprises: an environmental image generation means for generating the environmental image; and a display control means for executing control for displaying the generated environmental image on the display device, and wherein the 3DCG generation means further generates a 3DCG image showing the first avatar and the object, the image varying according to the predetermined environment represented by the environmental image.

10. An information processing method executed by a control device in an information processing system including an IMU sensor group including a plurality of IMU sensors that can be placed at the positions of each of a user's multiple skeletons; a 3DCG projection device that projects a 3DCG image into a real space in which the user is present in a state that the user can view the image; and a control device that communicates with the IMU sensor group and the 3DCG projection device and performs overall control, the information processing method including the steps of measuring skeletal information of each of the multiple skeletons based on time-series data acquired from the multiple IMU sensors placed at the positions of each of the user's multiple skeletons, generating a 3DCG image showing a first avatar taking a predetermined posture based on the measurement results, and executing control to project the 3DCG image from the 3DCG projection device.

11. In an information processing system including an IMU sensor group including a plurality of IMU sensors that can be placed at the positions of each of a user's multiple skeletal components; a 3DCG projection device that projects a 3DCG image into a real space in which the user is present in a state that the user can view the image; and a control device that communicates with the IMU sensor group and the 3DCG projection device and performs overall control. A program that causes a computer that controls the control device to execute a control process including the steps of measuring skeletal information of each of the multiple skeletons based on time-series data acquired from the multiple IMU sensors that are placed at the positions of each of the user's multiple skeletons, generating a 3DCG image showing a first avatar taking a predetermined posture based on the measurement results, and executing control to project the 3DCG image from the 3DCG projection device.

Citation Information

Patent Citations

  • Body motion training support system

    JP2016047219A

  • Information processing device, information processing method and program

    JP2017080197A

  • Rehabilitation support device and program

    JP2021003379A

  • Systems and methods for augmented neurologic rehabilitation

    US20210345947A1

  • Avatar tracking and rendering in virtual reality

    US20210349529A1