Skill acquisition support system, information processing device, control method, and program

Abstract

We provide a system that reduces cognitive load and allows users to learn movements. [Solution] A skill acquisition support system comprising an activity information acquisition unit that acquires activity information related to the activity of a first living organism, a perceptual information output unit configured to output perceptual information in a perceptible area of ​​a second living organism, and a control unit that outputs the perceptual information to the perceptual information output unit, wherein the perceptual information output unit includes a first perceptual information output unit and a second perceptual information output unit, and when an output criterion is satisfied, the control unit causes the first perceptual information output unit to output first perceptual information corresponding to a first activity among the activity information of the first living organism, and causes the second perceptual information output unit to output second perceptual information corresponding to a second activity among the activity information of the first living organism, the second perceptual information being in an information format different from the first perceptual information.

Description

[Technical Field]

[0001] The present technology relates to a skill acquisition support system, an information processing device, a control method, and a program. [Background technology]

[0002] To help people live longer and healthier lives, effective interventions using digital technology to improve physical and mental functions are anticipated. To achieve healthy longevity, it is important to take a comprehensive approach to preventing frailty, which is the decline in physical and mental functions associated with aging, including not only physical aspects but also social, cognitive, psychological, and spiritual aspects. For example, when people learn new movements that suit their diverse preferences, it would be ideal to have technology that supports the acquisition of those movements.

[0003] An example of a technique for passing on work movements to other workers is disclosed in Patent Document 1. Patent Document 1 provides a technique for automatically generating an optimal composite movement list for performing work efficiently, and discloses that the optimal movements defined in the composite movement list can be graphed and output as movements in a horizontal plane (XY plane), for example. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6583537 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, interventions for frailty require cognitive, psychological, and mental support to enhance self-efficacy and enable continued physical activity. However, the technology disclosed in Patent Document 1 has the problem that it is difficult to grasp the difference between optimal movements and one's own physical movements, and the cognitive load tends to be high when learning complex movements. Furthermore, movements that require a high cognitive load tend to be difficult to learn, not only in frailty prevention but also in the transmission of traditional skills, surgical skills, and early childhood education.

[0006] In view of the technical challenges described above, one aspect of the present technology aims to provide a mechanism that allows a user to learn a movement while reducing cognitive load. [Means for solving the problem]

[0007] In order to solve the above problems, the present technology discloses the following configuration. [Configuration 1] The skill acquisition support system related to this technology is: an activity information acquisition unit that acquires activity information related to the activity of the first living organism; a sensory information output unit configured to output sensory information in a perceptible region of the second living body; a control unit that outputs sensory information to the sensory information output unit; Equipped with the sensory information output unit includes a first sensory information output unit and a second sensory information output unit, When the output standard is satisfied, the control unit causing the first sensory information output unit to output first sensory information corresponding to a first activity among the first biological activity information; The second sensory information output unit is caused to output second sensory information corresponding to a second activity among the activity information of the first living body, the second sensory information being in an information format (modality) different from that of the first sensory information.

[0008] When information about multiple biological activities is presented using the same modality, the user receiving the information becomes confused and unable to organize the information, making it difficult to connect each piece of information with their own movements. In other words, the cognitive load associated with each piece of information tends to be high. According to the above configuration, first sensory information corresponding to a first activity among the activity information of a first biological organism and second sensory information corresponding to a second activity can be presented in different modalities in the perceptible area of ​​the second biological organism (i.e., the user). This allows the information to be transmitted and presented while reducing the user's cognitive load. As a result, the user can intuitively perform actions based on the first and second sensory information, and can learn the activity relatively easily, even for complex movements, compared to when information is presented using the same modality.

[0009] [Configuration 2] In the above configuration 1, when the activity information of the first living organism satisfies a first criterion, the control unit causes the first perceptual information output unit to output the first perceptual information at a first perceptual value, and when the activity information of the first living organism satisfies a second criterion, causes the first perceptual information output unit to output the first perceptual information at a second perceptual value different from the first perceptual value.

[0010] [Configuration 3] In the above-mentioned configuration 1 or 2, the control unit causes the second perceptual information output unit to output the second perceptual information at a first perceptual value when the activity information of the first living organism satisfies a first criterion, and causes the second perceptual information output unit to output the second perceptual information at a second perceptual value different from the first perceptual value when the activity information of the first living organism satisfies a second criterion.

[0011] [Configuration 4] In any of the above configurations 1 to 3, the control unit causes the first sensory information output unit to output the first sensory information as tactile information, and causes the second sensory information output unit to output the second sensory information as visual information.

[0012] [Configuration 5] In the above-mentioned configuration 4, the activity information of the first living organism is information regarding the movement of the first living organism, and the control unit causes the first sensory information output unit to output, as the first sensory information, temporal elements of the movement of the first living organism as information regarding the sense of touch, and causes the second sensory information output unit to output, as the second sensory information, spatial elements of the movement of the first living organism as information regarding the sense of sight.

[0013] [Configuration 6] In any one of the above configurations 1 to 5, at least one of the first sensory information and the second sensory information includes information representing at least one of the direction, speed, strength, and changes therein of the movement of the first living organism.

[0014] [Configuration 7] In any of the above configurations 1 to 6, the activity information of the first living organism is information regarding the movement of the same part of the first living organism, and the control unit causes the first sensory information output unit and the second sensory information output unit to output the information regarding the movement of the one part of the first living organism as the first sensory information and the second sensory information, which are different from each other.

[0015] [Configuration 8] In the above configuration 7, the control unit outputs information regarding the strength of the movement of the same part of the first living body to the first sensory information output unit as information regarding the sense of touch, and outputs information regarding the direction of the movement of the same part of the first living body to the second sensory information output unit as information regarding the sense of vision.

[0016] [Configuration 9] In any of the above configurations 1 to 8, the activity information of the first living organism is information regarding the movement of two or more different parts of the first living organism, and the control unit causes the first sensory information output unit to output the information regarding the movement of the first part of the first living organism as the first sensory information, and causes the second sensory information output unit to output the information regarding the movement of the second part of the first living organism as the second sensory information.

[0017] [Configuration 10] In any of the above configurations 1 to 9, the control unit accepts specification of the information format of the first sensory information to be output by the first sensory information output unit and the information format of the second sensory information to be output by the second sensory information output unit, and outputs the first sensory information and the second sensory information according to the specified information format from the first sensory information output unit and the second sensory information output unit, respectively.

[0018] [Configuration 11] In one embodiment, the present technology provides an information processing device comprising: an activity information acquisition unit that acquires activity information related to an activity of a first living organism; a sensory information output unit configured to output sensory information in a perceptible area of ​​a second living organism; and a control unit that outputs the sensory information to the sensory information output unit, wherein the sensory information output unit includes a first sensory information output unit and a second sensory information output unit, and when an output criterion is satisfied, the control unit causes the first sensory information output unit to output first sensory information corresponding to a first activity among the activity information of the first living organism, and causes the second sensory information output unit to output second sensory information corresponding to a second activity among the activity information of the first living organism, the second sensory information being in an information format different from the first sensory information.

[0019] [Configuration 12] In one embodiment, the present technology provides a control method in which a computer executes the steps of acquiring first sensory information corresponding to a first activity from activity information related to a first living organism that has been acquired in advance, acquiring second sensory information corresponding to a second activity from the activity information of the first living organism, the second sensory information being in an information format different from that of the first sensory information, and outputting the first sensory information to a first sensory information output unit and outputting the second sensory information to a second sensory information output unit.

[0020] [Configuration 13] In one embodiment, the present technology provides a program for causing a computer to execute the steps of: acquiring first sensory information corresponding to a first activity from activity information related to a first living organism that has been acquired in advance; acquiring second sensory information corresponding to a second activity from the activity information of the first living organism, the second sensory information being in an information format different from that of the first sensory information; and outputting the first sensory information to a first sensory information output unit and outputting the second sensory information to a second sensory information output unit.

[0021] [Configuration 14] In one embodiment, the present technology provides a program for causing a computer to execute the steps of: generating first sensory presentation information for generating first sensory information corresponding to a first activity among the activity information of a first living organism based on activity information related to the activity of the first living organism; and generating second sensory presentation information for generating second sensory information corresponding to a second activity among the activity information of the first living organism based on the activity information, the second sensory information being in an information format different from the first sensory information.

[0022] [Configuration 15] In one embodiment, the present technology provides an information processing method for assisting skill acquisition, including the steps of: generating, based on activity information related to a first living organism, first sensory presentation information for generating first sensory information corresponding to a first activity among the activity information of the first living organism; and generating, based on the activity information, second sensory presentation information for generating second sensory information corresponding to a second activity among the activity information of the first living organism, the second sensory information being in an information format different from the first sensory information. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is an example of a schematic diagram of the overall configuration of a skill acquisition support system. [Figure 2] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of the management server. [Figure 3] FIG. 3 is a schematic diagram illustrating an example of the hardware configuration of a technician terminal. [Figure 4] FIG. 4 is a schematic diagram illustrating an example of the hardware configuration of the learner terminal. [Figure 5] FIG. 5 is a conceptual diagram showing (a) activity information of the right hand and (b) activity information of the left hand according to one embodiment. [Figure 6] FIG. 6 is a conceptual diagram showing (a) the activity information of the right hand in FIG. 5 and (c) first sensory presentation information for the right hand created based on the activity information. [Figure 7] FIG. 7 is a conceptual diagram showing (b) the activity information of the left hand in FIG. 5 and (d) second sensory presentation information for the left hand created based on the activity information. [Figure 8] FIG. 8 is a diagram illustrating an example of a skill acquisition support flow. [Figure 9] FIG. 9 is a diagram illustrating another example of the skill acquisition support flow. [Figure 10] FIG. 10 is a diagram showing an example of how the first sensory information output device is worn. [Figure 11] FIG. 11 is a diagram showing an example of how another sensory information output device is worn. [Figure 12] FIG. 12 is a diagram showing an example of how another sensory information output device is worn. [Figure 13] FIG. 13 is a conceptual diagram of a case where perceptual information is presented using AR glasses. [Figure 14] FIG. 14 is a diagram showing an example of the modality reception screen. [Figure 15] FIG. 15 is a diagram showing another example of the modality reception screen. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, the present technology will be described based on each embodiment with reference to the accompanying drawings as appropriate. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0025] [Embodiment 1] 1. Overall system configuration The skill acquisition support system 1 relating to this technology acquires the movements of a skilled worker who has a specified skill as digital information, and supports the acquisition of a skill by presenting the information about the acquired movements as perceptual information when the learner of the skill practices the movement.

[0026] According to the research of the present inventors, when various pieces of information about movements are presented as perceptual information in a single format, it has become clear that the cognitive load of the information is high for the learner. Furthermore, it has been found that by presenting various pieces of information about movements as perceptual information in different formats, the cognitive load of the perceptual information for the learner can be reduced. This technology was developed based on this knowledge and is designed to assist learners in acquiring skills by presenting information about the movements of the learner in a form that places a low cognitive load on the learner.

[0027] In this embodiment, the present technology will be described using an example in which a learner learns to operate drumsticks in a drum performance by a skilled person. Here, in order to facilitate understanding of the present technology, the description will be based on a relatively simplified case of drum performance (eight-beat performance).

[0028] An overview of the skill acquisition support in this embodiment is shown in Table 1 below. When a skilled person plays the drums (activity), activity information is acquired by a motion sensor. The sensory information presented to the learner during practice is as follows: first sensory information is information indicating the timing of the right hand beat (strike), and second sensory information is information indicating the timing of the left hand beat and advance information. Here, the modality of the first sensory information is tactile information, and the modality of the second sensory information is visual information. In this way, by differentiating the modalities of the information presented to the learner, the burden on the learner when perceiving the first sensory information and the second sensory information can be reduced.

[0029] [Table 1]

[0030] FIG. 1 is a diagram illustrating an example of the configuration of a skill acquisition support system 1. The skill acquisition support system 1 includes one or more servers 101, one or more skilled worker terminals 102, one or more learner terminals 103, and sensory information output devices 104 and 105. The server 101, the skilled worker terminal 102, the learner terminal 103, and the sensory information output devices 104 and 105 are each capable of transmitting and receiving information to and from each other via a wired or wireless network. The sensory information output devices 104 and 105 are typically worn by the learner when the learner acquires a skill.

[0031] The skilled worker terminal 102 is a terminal for acquiring activities such as the movements of a skilled worker as digital information (activity information). The learner terminal 103 is a terminal for outputting sensory information corresponding to the acquired activity information to the sensory information output devices 104 and 105 . The server 101 is a device that manages sensory information and executes various processes for skill acquisition, which will be described later, in response to requests from the learner terminal 103, for example.

[0032] However, the server 101 may be configured to be able to execute the functional elements of at least one of the technician terminal 102 and the learner terminal 103. In other words, the server 101 may be configured integrally with the technician terminal 102 and / or the learner terminal 103. Furthermore, the server, each terminal, and device may each be configured as a single device, or may be configured as two or more separate partial devices.

[0033] The server 101, the technician terminal 102, and the learner terminal 103 (hereinafter, the technician terminal 102 and the learner terminal 103 may be simply referred to as "terminals") each include a processor that executes an operating system, applications, programs, etc., a main storage device such as a RAM (Random Access Memory) or a ROM (Read Only Memory), an auxiliary storage device such as a hard disk drive, an SSD (Solid State Drive), or a flash memory, a communication control unit such as a network card, a wireless communication module, or a mobile communication module, an input device such as a touch panel, a keyboard, a mouse, or a voice input, and an output device such as a monitor or a display. The output device may also be a device or terminal that transmits information to be output to an external monitor, display, printer, or other device.

[0034] The main memory stores various programs and applications (software modules), and the processor executes these programs and applications to realize each functional element of the overall system. Each module may be implemented as an independent program or application, or as a subprogram or function within a single integrated program or application. Each module may also be implemented as hardware (hardware modules) by integrating circuits or using a microcomputer. In this specification, each module is described as being the entity that performs the processing, but in reality, the processing is carried out by a processor that processes various programs, applications, etc. (modules).

[0035] Various databases (DBs) are stored in the auxiliary storage device. A "database" is a functional element (storage unit) that stores a set of data so that it can accommodate any data manipulation (e.g., extraction, addition, deletion, overwriting, etc.) from a processor or an external computer. There are no limitations on how the database is implemented; for example, it can be a database management system, spreadsheet software, or a text file such as XML or JSON.

[0036] The server 101 and each terminal may be, for example, a stationary or portable computer, or a server located on the cloud or a network. The server 101 and each terminal may be, for example, a portable terminal (mobile terminal) such as a smartphone, tablet, mobile phone, or personal digital assistant (PDA), or a wearable terminal such as glasses, a wristwatch, or clothing. The functions of the terminals may be a VR (Virtual Reality) terminal, an AR (Augmented Reality) terminal, or an MR (Mixed Reality) terminal. Alternatively, a combination of these terminals may be used. For example, a combination of one smartphone and one wearable terminal may logically function as one terminal. Other information processing terminals may also be used.

[0037] 2. Server hardware configuration FIG. 2 shows an example of the hardware configuration of the server 101 shown in FIG. The server 101 is configured, for example, by a server computer located on a cloud. The server 101 includes a main storage device 201 and an auxiliary storage device 202. The server 101 also includes a processor 203, an input device 204, an output device 205, and a communication control unit 206 as described above.

[0038] The main memory device 201 stores programs and applications such as an activity information acquisition module 211, a sensory information generation module 212, a first sensory information output module 213, a second sensory information output module 214, and an output control module 215. The processor 203 executes these programs and applications to realize each functional element of the server 101.

[0039] The activity information acquisition module 211 acquires activity information. The activity information is information related to the activity of a skilled worker (an example of a first living body) acquired via one or more of the sensors. In this embodiment, the activity information is acquired by the skilled worker terminal 102.

[0040] The sensory information generation module 212 generates sensory information based on the activity information. The sensory information generation module 212 generates at least first sensory information corresponding to a first activity of the activity information of the skilled worker and second sensory information corresponding to a second activity. Here, the sensory information generation module 212 generates the second sensory information in an information format different from that of the first sensory information. Details of the sensory information generation process by the sensory information generation module 212 will be described later.

[0041] The first sensory information output module 213 outputs the first sensory information to, for example, the first sensory information output device 104 attached to the perceptible area of ​​the learner (second living body). The second sensory information output module 214 outputs the second sensory information. For example, the second sensory information output module 214 outputs the second sensory information to the second sensory information output device 105 attached to the perceptible area of ​​the learner (second living body).

[0042] The output control module 215 controls the operations of the first perceptual information output module 213 and the second perceptual information output module 214. For example, the output control module 215 causes the first perceptual information output module 213 to output the first perceptual information and the second perceptual information output module 214 to output the second perceptual information. The output control module 215 causes the first perceptual information output module 213 and the second perceptual information output module 214 to output the first perceptual information and the second perceptual information so that the information is synchronized.

[0043] The auxiliary storage device 202 stores activity information 500, first sensory presentation information 600, and second sensory presentation information 700. The auxiliary storage device 202 includes a database necessary for storing these various types of information. The database may be implemented as a key-value store or as a relational DB. The contents of each piece of information will be described later. Part or all of the auxiliary storage device 202 may be provided separately from the main body of the server 101, for example, in the form of a cloud-based DB.

[0044] 3. Hardware configuration of technician terminals FIG. 3 shows an example of the hardware configuration of the technician terminal 102 shown in FIG. The technician terminal 102 is configured as a terminal such as a smartphone, tablet, notebook PC, or desktop PC. The technician terminal 102 includes a main memory device 301 and an auxiliary memory device 302. The technician terminal 102 also includes the processor 303, input device 304, output device 305, sensor 306, and communication control unit 307 as described above.

[0045] The main storage device 301 includes an activity information acquisition module 311. The activity information acquisition module 311 is the same functional module as the activity information acquisition module 211 of the server 101. That is, the skilled worker terminal 102 can perform the same processing as that of the server 101. The function of this module is the same as that described above, and therefore a description thereof will be omitted. Although not specifically shown, the main storage device 301 may also include a perceptual information generation module 312, a first perceptual information output module 313, a second perceptual information output module 314, and an output control module 315, and be configured to be able to perform the same processing as that of the server 101.

[0046] The auxiliary storage device 302 stores activity information 500. This information may be the same as part or all of the activity information 500 stored in the auxiliary storage device 202 of the server 101. Note that the auxiliary storage device 302 may also store other information that is the same as the information stored in the auxiliary storage device 202 of the server 101.

[0047] In addition, the main memory device 301 and auxiliary memory device 302 of the technician terminal 102 may store functional modules and information that are exactly the same as those of the main memory device 201 and auxiliary memory device 202 of the server 101, and in that case, each module provided in the technician terminal 102 may perform processing similar to that of the server 101.

[0048] The sensor 306 of the technician terminal 102 may be any of various sensors capable of detecting the movements of the technician. The sensor 306 may be a sensor capable of detecting the state of a part of the technician's body (e.g., position, acceleration, temperature, color, shape, etc.) and changes therein. Examples of sensors include, but are not limited to, a gyro sensor, an acceleration sensor, an electromyographic sensor, a pulse wave sensor, a temperature sensor, a color sensor, a contact sensor, an imaging sensor (camera), a motion sensor, etc.

[0049] In this embodiment, the sensor 306 of the technician terminal 102 is a motion sensor consisting of one or more markers attached to the technician's body and one or more cameras. The markers are equipped with, for example, piezoelectric elements or accelerometers, and the markers themselves are capable of detecting the technician's movements (continuous changes in acceleration), gravity, tilt, vibration, impact, etc. The configuration of the motion sensor is not particularly limited, and examples include sensors based on capture technologies such as optical (including image and infrared), inertial sensor, mechanical, and magnetic.

[0050] 4. Hardware configuration of the learner's terminal FIG. 4 shows an example of the hardware configuration of the learner terminal 103 shown in FIG. The learner terminal 103 is configured as a terminal such as a smartphone, tablet, notebook PC, or desktop PC.

[0051] The learner terminal 103 includes a main memory device 401 and an auxiliary memory device 402. The learner terminal 103 also includes a processor 403, an input device 404, an output device 405, a sensor 406, and a communication control unit 407, as described above.

[0052] The main storage device 401 includes a first sensory information output module 413, a second sensory information output module 414, and an output control module 415. These modules are functionally identical to the first sensory information output module 213, the second sensory information output module 214, and the output control module 215 of the server 101. That is, the learner terminal 103 can perform the same processing as that of the server 101. The function of this module is the same as that described above, and therefore a description thereof will be omitted. Although not specifically shown, the main storage device 401 may also include an activity information acquisition module 411 and a sensory information generation module 412, and be configured to be able to perform the same processing as that of the server 101.

[0053] The auxiliary storage device 402 stores first sensory presentation information 600 and second sensory presentation information 700. This information may be the same as part or all of the first sensory presentation information 600 and the second sensory presentation information 700 stored in the auxiliary storage device 202 of the server 101. Note that the auxiliary storage device 402 may also store other information that is the same as the information stored in the auxiliary storage device 202 of the server 101.

[0054] In addition, the main memory device 401 and auxiliary memory device 402 of the learner terminal 103 may store functional modules and information that are exactly the same as those of the main memory device 201 and auxiliary memory device 202 of the server 101, and in that case, each module provided in the learner terminal 103 may perform the same processing as that of the server 101.

[0055] 5. Sensory information output device The sensory information output devices 104 and 105 are devices that present (output) sensory information. Typically, the sensory information output devices 104 and 105 are connected to the learner terminal 103 via a wired or wireless connection, and present sensory information based on the sensory presentation information 600 and 700 acquired from the learner terminal 103.

[0056] Sensory information is information that can be perceived by a living organism, and includes, but is not limited to, visual information, tactile information, and auditory information. Visual information includes, for example, information regarding color, brightness, position, shape, size, and changes therein. Tactile information includes, for example, information regarding the presence or absence of contact, the contact position, contact area, pressure, pain, temperature, and changes therein. Auditory information includes, for example, information regarding the frequency, waveform, and volume of sound, and changes therein.

[0057] The sensory information output device is configured to output the sensory information in a region perceptible by the learner. The sensory information output device is configured, for example, to be attachable to the learner's body or an operation target (e.g., a tool or a treatment target). The sensory information output device may be configured appropriately depending on the modality of the sensory information to be presented to the learner. For example, when the sensory information is visual information, examples of the sensory information output device include lights such as LEDs, wearable displays, and AR or VR displays. When the sensory information is tactile information, examples of the sensory information output device include heat generators such as heaters, tactile presentation devices such as actuators and oscillators, exoskeleton devices that assist joint movement by applying moments around joints, and electrical stimulation devices such as EMS (Electrical Muscle Stimulation). When the sensory information is auditory information, examples of the sensory information output device include sound-generating devices such as earphones and headphones.

[0058] Such sensory information output devices 104, 105 are suitable for use when attached to a moving part of the body or to a tool used in that part, as this naturally links the moving part of the body with the sensory information presented in relation to the movement of that part. Furthermore, when such sensory information output devices 104, 105 are not devices that assist the learner's movements themselves, but devices that do not assist the learner's movements themselves, the movements can be performed as voluntary movements by the learner, thereby further improving the efficiency of learning.

[0059] In this embodiment, the first sensory information output device 104 is a tactile sensation providing device, such as a vibration generating device. The vibration generating device is configured to be wearable on the learner's body via an adhesive pad, a silicone belt, or the like. Fig. 10 is a diagram showing an example of how to wear the first sensory information output device 104. The first sensory information output device 104 (vibration generating device) shown in Fig. 10 has a vibration generating element on the inner surface of a silicon belt, and by wearing the silicon belt on the wrist or the like of the learner, the vibration generating element can be brought into close contact with the learner's skin.

[0060] The second sensory information output device 105 is a visual information device, such as a line LED. The line LED is configured by arranging a plurality of LEDs (five in this example) in a line, as shown in FIG. 7, for example. This line LED is configured to be attachable, for example, via an adhesive, to the drumsticks that the learner will operate. Line LEDs are useful in that they are inexpensive and lightweight, and can present a variety of sensory information based on relatively simple sensory presentation information. For example, when multiple learners are practicing drumming at the same time, it is also preferable that line LEDs for each learner can be prepared at low cost.

[0061] 6. Data Overview Next, various types of information managed by the skill acquisition support system 1 will be briefly described. The activity information 500 is information related to the physical activity of a first living organism (here, a skilled worker) when the first living organism is performing a desired movement. The activity information 500 is typically information related to the physical activity of the skilled worker acquired via one or more sensors. The information related to the physical activity may be, for example, information related to the movement of a part of the body (e.g., hand, finger, arm, leg, head, eyeball, etc.) or the entire body (e.g., position, angle (including posture), acceleration, angular velocity, etc.). The information related to the physical activity may also be, for example, biological information (e.g., body temperature, heart rate, brain waves, etc.). The activity information 500 in this embodiment is motion sensor information acquired via a motion sensor.

[0062] The first sensory presentation information 600 and the second sensory presentation information 700 are information for causing the first sensory information output device 104 and the second sensory information output device 105 to present first and second sensory information corresponding to a first activity and a second activity of the skilled worker, respectively. Here, at least the first sensory information and the second sensory information are information of different modalities, and at least the first sensory presentation information 600 and the second sensory presentation information 700 are information for presenting information of different modalities. In this embodiment, the first sensory presentation information 600 is information for causing a vibration generating device to generate vibrations. Furthermore, the second sensory presentation information 700 is information for turning on a plurality of LED lights.

[0063] 7.Skill acquisition support method Next, the skill acquisition support method and the processing of the skill acquisition support system 1 that implements this method will be described for each processing stage.

[0064] 7-1. Overall flow of skills acquisition support method 8 is a diagram showing an example of a skill acquisition assistance flow. The skill acquisition assistance method includes the following steps.

[0065] In the skill acquisition support method, first, the server 101 acquires activity information 500 (S810). The server 101 acquires first sensory presentation information 600 based on the activity information 500 (S820). The server 101 acquires second sensory presentation information 700 based on the activity information 500 (S830). Then, if the output criterion is satisfied (Yes in S840), the server 101 causes the first sensory information output unit to output the first sensory information and the second sensory information output unit to output the second sensory information based on the activity information 500 (S850). Step S820 and step S830 may be executed either first, or may be executed simultaneously or in parallel. The processing of each step will be described below.

[0066] 7-2. Acquisition of activity information (S810) When the activity information acquisition module 211 of the server 101 acquires activity information 500 about the activity of a skilled worker (first living organism) for the first time, it cooperates with, for example, the activity information acquisition module 311 of the skilled worker terminal 102 to acquire the activity information 500 from the skilled worker terminal 102. The activity information acquisition module 211 can acquire the activity information 500 stored in the auxiliary storage device 302 of the skilled worker terminal 102, or the activity information 500 acquired via the sensor 306 of the skilled worker terminal 102. The activity information acquisition module 211 stores the acquired activity information 500 in, for example, the auxiliary storage device 202.

[0067] A method for acquiring activity information 500 relating to the movements of a skilled worker while playing the drums in the skilled worker terminal 102 will be described. The motion sensor 306 is provided in (connected to), for example, the skilled worker terminal 102. The markers of the motion sensor 306 are attached to, for example, the backs of the right and left hands of the skilled worker who operate the drumsticks. If the skilled worker operates a tool (drumsticks), the markers may be attached to the tool. The skilled worker then operates the drumsticks while wearing the markers to play the drums. The activity information acquisition module 311 of the skilled worker terminal 102 can acquire activity information 500 of the skilled worker's right and left hands, for example, by recording the movement of the markers during performance using a motion sensor camera. The activity information acquisition module 311 stores the acquired activity information 500 in, for example, the auxiliary storage device 302.

[0068] When a skilled worker operates a tool (drum set), the sensor 306 is not limited to being attached to a tool (e.g., drumsticks) that the skilled worker directly operates, but may also be attached to a tool that is directly affected by the skilled worker's actions (e.g., a drum or cymbal that is the target of the striking action). For example, attaching an acceleration sensor or contact sensor to a drum or cymbal as the sensor 306 is preferable because it allows for easy acquisition of information that more clearly indicates the point in time when the skilled worker strikes the drum or cymbal.

[0069] The server 101 may acquire the activity information 500 at the same time as the skilled worker terminal 102 acquires the activity information 500, or may acquire the activity information 500 after an arbitrary time lag.

[0070] 7-3. Generation of first perceptual presentation information (S820) The sensory information generation module 212 of the server 101 generates first sensory presentation information 600 based on the activity information 500 . FIG. 5 is a conceptual diagram showing (a) activity information of the right hand and (b) activity information of the left hand according to one embodiment.

[0071] The sensory information generation module 212 first obtains the activity information 500 from the auxiliary storage device 202. Next, the sensory information generation module 212 extracts, from the movements recorded as the activity information 500, information about characteristic movements of the marker attached to the right hand that are observed when hitting a drum (e.g., a low drum). Examples of characteristic movements include when the acceleration is greater than a predetermined standard, or when the direction of the movement changes from downward to upward, etc. This makes it possible to obtain (a) right hand activity information that indicates when (the timing of) the drum was hit by manipulating the drumstick with the right hand.

[0072] FIG. 6 is a conceptual diagram showing (a) the activity information of the right hand in FIG. 5 and (c) first sensory presentation information 600 for the right hand created based on the activity information. The sensory information generation module 212 generates (c) first sensory presentation information 600 including a vibration generation signal (e.g., a square wave) corresponding to the timing of hitting the drum with the right hand, based on (a) right-hand activity information. The sensory information generation module 212 may process the first sensory presentation information 600 into an appropriate format depending on the specifications of the first sensory information output device 104 used for output. In this embodiment, the sensory information generation module 212 generates the first sensory presentation information 600 as information instructing the timing of generating vibrations in the vibration generating device, which is the second sensory information output device 105. The sensory information generation module 212 stores the generated first sensory presentation information 600 in, for example, the auxiliary storage device 202.

[0073] 7-4. Generation of Secondary Perceptual Presentation Information (S830) The sensory information generation module 212 of the server 101 generates second sensory presentation information 700 based on the activity information 500. The sensory information generation module 212 first obtains the activity information 500 from the auxiliary storage device 202. Next, the sensory information generation module 212 extracts information about the characteristic movements of the marker worn on the left hand that are observed when hitting a drum (e.g., a hi-hat cymbal) from the movements recorded as the activity information 500. This makes it possible to obtain (b) left hand activity information that indicates the timing when the drumstick is manipulated with the left hand to hit the drum, as shown in FIG. 5(b).

[0074] FIG. 7 is a conceptual diagram showing the activity information of the left hand in FIG. 5(b) and second sensory presentation information for the left hand (d) created based on the activity information of the left hand. As described above, it may be more difficult to grasp the timing of the left hand movement when playing drums than the right hand movement. Therefore, in this embodiment, the sensory information generation module 212 generates (d) the second sensory presentation information 700 so as to include information indicating the timing of the drum beat and information corresponding to the left hand movement before the beat (prior information).

[0075] The prior information is information corresponding to the movement of the left hand before the beat of the target movement (for example, a predetermined period immediately before), and for example, by recognizing the prior information, it becomes easier to predict the timing of the beat of the target movement. Furthermore, by presenting such prior information, the effect of inducing the beat of the target movement is manifested.

[0076] For example, the sensory information generation module 212 generates the second sensory presentation information 700 including a light emission signal (e.g., a square wave) that causes a plurality of LEDs in the line LED to emit light in association with the movement (movement) of the left hand when striking the drum and for a predetermined period immediately before that. The predetermined period represented by the prior information may be, for example, a period from the start of the prior movement (e.g., from the moment the arm is raised to strike), or a time period during which the learner can recognize the next movement in advance (e.g., one second).

[0077] In this case, the line LED serving as the second sensory information output device 105 includes five LEDs L1 to L5 arranged from the center toward the tip in the longitudinal direction of the stick, as shown in Fig. 7. In other words, by using the line LED, the second sensory information output device 105 can spatially present information related to movement.

[0078] Furthermore, (d) second sensory presentation information 700 includes (d1) to (d4) advance information 1 to 4 and (d5) timing information for illuminating the five LEDs L1 to L5, respectively. (d1) to (d4) advance information 1 to 4 are instruction information for sequentially illuminating the four LEDs L1 to L4 to indicate the movement of the marker before the beat, and (d5) timing information is instruction information for illuminating the L5 LED to indicate the timing of the beat. According to this second sensory presentation information 700, the line LEDs can be illuminated sequentially from L1 to L5, starting a predetermined time before the timing of the beat. This allows the learner to know the advance movement of the beat and predict the timing of the beat with the left hand. Furthermore, by being able to predict the timing of the beat, the learner can practice with an increased sense of agency and self-efficacy. Furthermore, by repeatedly comparing predictions with the actual movements during practice, the learner can efficiently master the movements.

[0079] The sensory information generation module 212 may process the second sensory presentation information 700 into an appropriate format depending on the specifications of the second sensory information output device 105 used for output. In this embodiment, the sensory information generation module 212 generates the second sensory presentation information 700 as information instructing the timing of emitting light from each LED in the light-emitting device that is the second sensory information output device 105. The sensory information generation module 212 stores the generated second sensory presentation information 700 in, for example, the auxiliary storage device 202.

[0080] 7-5. Satisfaction of output standards (S840) The output control module 215 of the server 101 evaluates whether the output criteria are met. If the output criteria are met, the output control module 215 proceeds to the next step (S850), and if the output criteria are not met, the output control module 215 waits. As the output standard, various conditions such as those listed below can be taken into consideration. When an instruction to output perceptual information is received from the learner terminal 103 When an instruction to output sensory information is received from the first sensory information output device 104 and / or the second sensory information output device 105. When an instruction to output sensory information is received from any other device When the specified output time for sensory information arrives

[0081] 7-6. Output of sensory information (S850) The learner attaches the first sensory information output device 104 and the second sensory information output device 105 to their body or tools. Specifically, for example, the learner wears a silicone belt-type vibration generating device on their right wrist and attaches line LEDs to their drumsticks via adhesive. Once the learner is ready to practice drumming in this way, they can send output instructions to the server 101 via the learner's terminal 103 as needed, for example, so that the server 101 satisfies the output criteria of step S840.

[0082] Then, the output control module 215 of the server 101 causes the first sensory information to be output from the first sensory information output device 104, and the second sensory information to be output from the second sensory information output device 105. Specifically, the output control module 215 cooperates with, for example, the output control module 415 of the learner terminal 103 to execute the following processes. That is, the first perceptual information output module 213 outputs the first perceptual presentation information 600 to the learner terminal 103 , and the second perceptual information output module 214 outputs the second perceptual presentation information 700 to the learner terminal 103 .

[0083] The output control module 415 of the learner terminal 103 stores the acquired first sensory presentation information 600 and second sensory presentation information 700 in the auxiliary storage device 402, for example. In addition, the first sensory information output module 413 of the learner terminal 103 outputs the acquired first sensory presentation information 600 to the first sensory information output device 104, causing the first sensory information output device 104 to output the first sensory information. In addition, the second sensory information output module 414 of the learner terminal 103 outputs the acquired second sensory presentation information 700 to the second sensory information output device 105, causing the second sensory information output device 105 to output the second sensory information. Here, the output control module 215 controls the driving of the first perceptual information output module 413 and the second perceptual information output module 414 so that the output of the first perceptual information by the first perceptual information output device 104 and the output of the second perceptual information by the second perceptual information output device 105 are synchronized.

[0084] The learner practices playing the drums based on the sensory information presented on the first sensory information output device 104 and the second sensory information output device 105. For example, the learner practices hitting the drums with the right hand to match the vibrations output by the vibration output device. Also, for example, the learner practices hitting the drums with the left hand to match the timing information presented by the light emission of the line LEDs, while measuring the timing of hitting the drums based on the advance information presented by the light emission of the line LEDs.

[0085] 7-7. Skill acquisition support method flow (during repetitive practice) After the first sensory presentation information 600 and the second sensory presentation information 700 are stored in the learner terminal 103, for example, skill acquisition support can be performed offline in the learner terminal 103. FIG. 9 is a diagram illustrating another example of the skill acquisition support flow.

[0086] The skill acquisition support method includes the following steps. First, the output control module 415 of the learner terminal 103 acquires the first sensory presentation information 600 from the auxiliary storage device 402 (S910), and also acquires the second sensory presentation information 700 from the auxiliary storage device 402 (S920). Then, if the output criteria are met (Yes in S930), the output control module 415 causes the first sensory information output device 104 to output the first sensory information and the second sensory information output device 105 to output the second sensory information based on the first and second sensory presentation information 600 and 700.

[0087] Step S910 and step S920 may be performed either first, or may be performed simultaneously or in parallel. The output criteria may be, for example, the same as the conditions described above in the description of step S840. This allows the learner to repeatedly practice playing the drums offline, for example, without connecting to the server 101.

[0088] 8. Composition and effects According to the above configuration, it is possible to efficiently acquire skills by reducing cognitive load. Furthermore, even if a skilled person is in a remote location, the skilled person's movements can be conveyed to the learner, thereby increasing opportunities for skill transfer. Furthermore, once acquired, the sensory presentation information 600, 700 can be reused, allowing practice for skill acquisition both online and offline.

[0089] In the above embodiment, the output control module 215 (an example of a control unit) is configured to, when an output criterion is met, output first perceptual information corresponding to the movement of the right hand of the skilled worker (an example of a first living organism) in playing the drums to the first perceptual information output module 213 (an example of a first perceptual information output unit), and output second perceptual information corresponding to the movement of the left hand of the skilled worker in playing the drums, the second perceptual information being of a different modality from the first perceptual information, to the second perceptual information output module 214 (an example of a second perceptual information output unit).

[0090] As a result, the vibration generating device, which is the first sensory information output device 104, generates vibrations at the timing of hitting a drum (e.g., a low drum), and provides vibration information indicating the timing of hitting the drum to the learner's right wrist. Synchronized with this vibration, the line LED, which is the second sensory information output device 105, illuminates LEDs L1 to L5 at the timing of hitting a drum (e.g., a hi-hat cymbal) and for a predetermined period before that, spatially presenting optical information indicating the timing of hitting the drum within the learner's field of vision. By presenting information about the movement in different modalities in this way, the learner can perceive each piece of information with reduced strain, allowing them to practice and learn the movement efficiently.

[0091] Furthermore, if the learner's right hand movement differs from the vibration information from the vibration generating device, the learner will be able to strongly recognize the difference and will be conscious of moving their right hand to reduce the difference. Furthermore, the learner can predict the movement of their left hand based on the advance information displayed by the line LED, and can move their left hand based on this prediction with a sense of mental ease. Furthermore, by checking whether the movement based on this prediction matches the timing information displayed by the line LED, the learner can efficiently learn the movement. This allows for the elderly, young children, and beginner drummers to be presented with perceptual information for the left and right arm movements required for playing the drums while reducing the cognitive load, thereby enabling them to efficiently acquire physical skills, for example.

[0092] Furthermore, with the above configuration, for example, a large amount of information about movement can be presented separately as visual information, which has a more limited presentation area, and as other visual information, such as tactile and auditory information, which has a relatively less limited presentation area. This allows a large amount of information about the movement to be conveyed to be presented with reduced cognitive load. In particular, by presenting information about the movement of a specific motion part as visual information in a position close to the part, the movement information of the part can be presented with even lower load.

[0093] In the above embodiment, the output control module 215 causes the first sensory information output module 213 to output the first sensory information as tactile information, and causes the second sensory information output module 214 to output the second sensory information as visual information. Each modality of perceptual information has different characteristics. For example, visual information excels at presenting spatial information and can present a large amount of information with a relatively low load, but it is difficult to obtain information over a wide field of view. In contrast, tactile information excels at presenting temporal information and can be perceived over a wide area, such as the skin. Therefore, by presenting the first and second perceptual information as a combination of visual and tactile information, it is possible to reduce cognitive load while taking advantage of the characteristics of each modality, and to effectively learn the desired movement.

[0094] In the above embodiment, the activity information of the skilled worker is information about the action of playing a drum. The output control module 215 controls the first perceptual information output module 213 to output tactile information about the time element of the skilled worker's action as the first perceptual information, and the second perceptual information output module 214 to output visual information about the spatial element of the skilled worker's action as the second perceptual information. Visual information is effective in presenting spatial information, and can provide advance information about the action and allow the user to predict the target action (beat). On the other hand, humans have the characteristic of being less sensitive to tactile stimuli that match their own movements (considered a type of tactile suppression), but when there is a discrepancy between their own movements and tactile stimuli, the discrepancy is emphasized and perceived. Therefore, for example, when movement information with a strong temporal element, such as a movement with a constant rhythm, is presented as tactile information, the discrepancy can be clearly recognized. Furthermore, for actions that are difficult to time, by presenting both prediction information and timing information using spatial information, the user can repeatedly predict and compare the movements, thereby effectively mastering the action. By presenting the first and second perceptual information as a combination of visual and tactile information, it is possible to achieve low-stress and efficient motion instruction that takes advantage of the characteristics of the modalities. [Embodiment 2] In the first embodiment, the skill to be learned was playing the drums, but in the present embodiment, the skill to be learned is writing, as shown in Table 2 below. Furthermore, the sensory information output devices 104 and 105 in the first embodiment were attached to different body parts of the learner to present sensory information related to the movements of the different body parts of the learner. The sensory information output devices 104 and 105 in the present embodiment are integrated, for example, as a wristwatch-type wearable device and attached to one body part of the learner. Furthermore, the sensory information output devices 104 and 105 in the present embodiment present sensory information related to the movements of the same body part (here, the right hand) of the learner as at least two different types of sensory information. Other configurations, operations, and effects may be similar to those of the first and second embodiments, and therefore will not be described again.

[0095] [Table 2]

[0096] FIG. 11 is a diagram showing an example of how the sensory information output device is worn. The wristwatch-type wearable device is provided with, for example, a haptic module (an example of a vibration generating device) including a vibration actuator and a control circuit for controlling its operation as the first sensory information output device 104 that presents vibration information. This haptic module is configured to be able to output vibration information based on, for example, one or more combinations of vibration intensities, intervals, and durations.

[0097] The wristwatch-type wearable device also includes a display module including, for example, an organic liquid crystal display and a drive circuit for controlling its operation, as a second sensory information output device 105 that presents visual information. This display module is configured to be able to display still images or moving images of any color on the organic liquid crystal display.

[0098] 1. Acquisition of activity information A method for acquiring activity information 500 relating to the movements of a skilled worker when he or she is writing a model character or sentence, etc., in the skilled worker terminal 102 will be described. The sensor 306 that acquires the activity information 500 is, but is not limited to, a film-like pressure sensor made of a piezoelectric material (such as polyvinylidene fluoride or polylactic acid). The pressure sensor is configured to be able to detect pressure applied to the film surface and its position.

[0099] The pressure sensor 306 is connected to the skilled worker terminal 102. The activity information acquisition module 311 of the skilled worker terminal 102 can acquire, via the pressure sensor 306, activity information 500 relating to writing actions such as the trajectory and writing pressure of the writing instrument when the skilled worker writes characters or the like on the surface of the pressure sensor 306 with a writing instrument (such as a pencil, pen, or brush). The activity information acquisition module 311 stores the acquired activity information 500 in, for example, the auxiliary storage device 302.

[0100] 2.Generation of perceptual presentation information The sensory information generation module 212 of the server 101 generates first sensory presentation information 600 and second sensory presentation information 700 based on the activity information 500 . The sensory information generation module 212 first acquires the activity information 500 from the auxiliary storage device 202. Next, the sensory information generation module 212 acquires, from the surface pressure distribution on the film surface of the pressure-sensitive sensor recorded as the activity information 500, information on, for example, the trajectory of the stroke (trajectory of the pen position) and information on the writing pressure.

[0101] The information about the writing pressure includes elements related to the magnitude and the change thereof. Therefore, the sensory information generation module 212 generates first sensory presentation information 600 for presenting the information about the writing pressure as tactile information (vibration information) on the vibration generating device. For example, the sensory information generation module 212 When the magnitude of the pen pressure satisfies a first criterion, vibration information is output at a first perception value; When the magnitude of the pen pressure satisfies a second criterion, the vibration information is output at a second perception value. In this way, the first sensory presentation information 600 is generated. Specifically, the sensory information generation module 212 generates the first sensory presentation information 600 so that, for example, the greater the writing pressure, the greater the frequency and / or amplitude of the vibration.

[0102] Information about the trajectory of the stroke includes a spatial element. Therefore, the perceptual information generation module 212 generates second sensory presentation information 700 for presenting the information about the trajectory of the stroke as visual information on the organic liquid crystal display. For example, the perceptual information generation module 212 may generate second sensory presentation information 700 that represents the direction of the stroke. Alternatively, for example, the perceptual information generation module 212 may generate second sensory presentation information 700 that shows the entire trajectory of the stroke and the current pen position on the model, superimposed on each other. The perceptual information generation module 212 may process the second sensory presentation information 700 into an appropriate format so that visual information in a desired form is displayed on the organic liquid crystal display, which is the second sensory information output device 105.

[0103] 3. Output of sensory information The learner wears a wristwatch-type wearable device on his / her wrist as the sensory information output device 104, 105. Then, when the learner is ready to practice the writing action, he / she sends an output instruction to the server 101 via the learner terminal 103 as necessary, for example, to satisfy the output criteria.

[0104] Then, the output control module 215 of the server 101 cooperates with, for example, the output control module 415 of the learner terminal 103 to execute the following process. That is, the first perceptual information output module 213 outputs the first perceptual presentation information 600 to the learner terminal 103 , and the second perceptual information output module 214 outputs the second perceptual presentation information 700 to the learner terminal 103 . The output control module 415 of the learner terminal 103 stores the acquired first sensory presentation information 600 and second sensory presentation information 700 in the auxiliary storage device 402, for example.

[0105] In addition, the first sensory information output module 413 of the learner terminal 103 outputs the acquired first sensory presentation information 600 to the first sensory information output device 104 (haptics module), causing the first sensory information output device 104 to output first sensory information (vibration information). In addition, the second sensory information output module 414 of the learner terminal 103 outputs the acquired second sensory presentation information 700 to the second sensory information output device 105 (organic liquid crystal display), causing the second sensory information output device 105 to output second sensory information (visual information).

[0106] Here, the output control module 215 controls the driving of the first perceptual information output module 413 and the second perceptual information output module 414 so that the output of vibration information corresponding to the pen pressure output by the haptics module and the output of visual information corresponding to the pen movement presented by the organic liquid crystal display are synchronized. As a result, information corresponding to the stroke and information corresponding to the pen pressure are presented to the learner in synchronization.

[0107] The learner practices writing movements based on the sensory information presented on the first sensory information output device 104 and the second sensory information output device 105. For example, the learner adjusts the writing pressure to correspond to the magnitude of the vibration presented by the haptics module while moving the writing implement in the direction of the writing movements presented by the organic liquid crystal display. This reduces the cognitive load on the learner when receiving information about the writing movements and pressure in the writing movement, allowing the learner to efficiently practice and learn the writing movement.

[0108] 4. Composition and effects In the third embodiment, the output control module 215 causes the haptics module (first perceptual information output device 104) to output vibration information (first perceptual information) at a first perceptual value when the writing pressure of the skilled worker's writing action satisfies a first criterion, and causes the haptics module to output vibration information at a second perceptual value different from the first perceptual value when the writing pressure satisfies a second criterion. For example, the system is configured so that the greater the writing pressure of the skilled worker's writing action, the greater the vibration information imparted to the learner. This allows the learner to recognize the magnitude and changes in the writing pressure of the model with little effort, while paying attention to the direction of the writing action presented as the second perceptual information.

[0109] [Embodiment 3] In the first embodiment, the skill to be learned was playing the drums, but in this embodiment, the skill to be learned is intervention movements and range of motion training movements performed by a physical therapist or the like, as shown in Table 3 below. Furthermore, in the first embodiment, a line LED that can be attached to a drumstick is used as the second sensory information output device 105. In this embodiment, the second sensory information output device 105 is a bracelet-type LED device that can be attached to the left wrist of the learner (e.g., a physical therapist). Other configurations, actions, and effects not specifically mentioned below may be the same as those in the first embodiment, and repeated explanations will be omitted.

[0110] [Table 3]

[0111] For example, doctors and physical therapists may provide range-of-motion training to physically disabled individuals with the goal of improving joint movement and preventing contracture. When providing exercise intervention to physically disabled individuals, for example, it is necessary to master the use of the left and right arms to support the individual. Furthermore, in manual passive range-of-motion training, it is important to move the individual's joints in the anatomically correct direction. At this time, it is also important to properly understand the individual's range of motion.

[0112] The motion sensor 306 used in this embodiment is composed of one or more light stations (light sources) and a tracker that receives light from the light stations. The light stations (light sources) emit, for example, a directional laser (e.g., an infrared laser). The tracker includes one or more (typically multiple, e.g., three) light receiving units that receive the light from the light stations and a transmitter that transmits the received light information to the technician terminal 102. By analyzing the light receiving signals from the light receiving units, it is possible to monitor the position and posture of the tracker in three-dimensional space, as well as their changes over time. For example, by placing the light receiving units of the tracker around the joints of a patient receiving treatment by a physical therapist, it is possible to grasp the spatial movement around the joints of the patient.

[0113] FIG. 12 is a diagram showing an example of how the second sensory information output device 105 according to an embodiment is worn. According to the study by the present inventors, it has been confirmed that the bracelet-type LED device as shown in Fig. 12 is useful, for example, when a doctor, a physical therapist, or the like remotely learns intervention actions for a treatment recipient. In this embodiment, an example of using such a bracelet-type LED device as the second sensory information output device 105 is disclosed.

[0114] The second sensory information output device 105 is a somewhat wide, band-like device that is wrapped around and fixed to the learner's wrist. The second sensory information output device 105 has small LEDs arranged in a matrix on its outer surface, and is configured to display various characters, arbitrary symbols, patterns, etc., electronically on an XY plane (display area) in which the longitudinal direction of the matrix-arranged LEDs (e.g., the circumferential direction of the arm when worn) is the Y axis and the width direction (e.g., the direction in which the arm extends when worn) is the X axis. The second sensory information output device 105 is worn in a position determined, for example, so that the longitudinal center of the display area is located at the width center on the back of the learner's wrist.

[0115] When using the second sensory information output device 105 configured as described above, various characters, arbitrary symbols, patterns, etc. can be presented as visual information. The electronic display of characters, symbols, patterns, etc. using LEDs is useful in that the learner can intuitively grasp the content of the display. The electronic display of characters, symbols, patterns, etc. by the second sensory information output device 105 may be presented fixedly at a predetermined position in the display area (for example, an area on the back of the wrist), or may be presented by moving or flashing within the display area. For example, light, characters, symbols, etc. can be presented to flow or flash in the direction of movement.

[0116] Furthermore, the second sensory information output device 105 provides a three-dimensional display area when worn on the learner's wrist, allowing movement-related information to be presented in two or three dimensions. Furthermore, for example, depending on the angle and position of the skilled worker's arm obtained by the tracker, letters, symbols, etc. can be presented in an area of ​​the display area that is easy for the learner to see. This allows for an increased amount of information about the movement to be presented to the learner.

[0117] 1. Acquisition of activity information A method for acquiring, in the skilled worker terminal 102, activity information 500 relating to the movements of the skilled worker when the skilled worker is performing model range of motion training movements on the patient undergoing treatment will be described. The sensor 306 that acquires the activity information 500 is, but is not limited to, the above-mentioned motion sensor and pressure sensor, for example. The light receiving unit of the motion sensor tracker is attached, for example, to the joints and bones surrounding the joints of the patient who is the object of the technician's operation. The pressure sensor is attached, for example, to the fingertips of the technician's dominant hand or both hands. The pressure sensor is, for example, a film-like pressure sensor made of a piezoelectric material (for example, polyvinylidene fluoride or polylactic acid). The pressure sensor is configured to detect pressure applied to the film surface and its position. The light receiving unit of the motion sensor tracker may be attached, for example, to the technician (for example, the arm or fingers) in addition to the patient.

[0118] The sensors 306 are connected to the skilled worker terminal 102. The activity information acquisition module 311 of the skilled worker terminal 102 acquires activity information 500 related to movements, for example, when a skilled worker performs range of motion training on a patient undergoing treatment, via these sensors 306. Here, for example, the activity information 500 acquired includes skeletal movements around the joints of the patient undergoing treatment and pressure applied to the fingertips of the skilled worker's dominant hand. The activity information acquisition module 311 stores the acquired activity information 500 in, for example, the auxiliary storage device 302.

[0119] 2.Generation of perceptual presentation information The sensory information generation module 212 of the server 101 generates first sensory presentation information 600 and second sensory presentation information 700 based on the activity information 500 . The sensory information generation module 212 first acquires the activity information 500 from the auxiliary storage device 202. Next, the sensory information generation module 212 acquires information recorded as the activity information 500 regarding the movement around the joint of the patient and the magnitude of pressure generated on the skilled worker's fingers when a moment is applied around the joint.

[0120] <Information about the pressure exerted on the technician's fingers> The information about the pressure includes elements related to the magnitude and change of the pressure. Therefore, the sensory information generation module 212 generates first sensory presentation information 600 for presenting the information about the pressure as tactile information (vibration information) on the vibration generating device. For example, the sensory information generation module 212 When the magnitude of the pressure satisfies a first criterion, the vibration information is output at a first perception value; When the magnitude of the pressure satisfies a second criterion, the vibration information is output at a second perception value; In this way, the first sensory presentation information 600 is generated. Specifically, the sensory information generation module 212 generates the first sensory presentation information 600 so that, for example, the greater the pressure, the greater the frequency and / or amplitude of the vibration.

[0121] <Information about the movement of the patient's joints> Information about the movement of the skeleton around the joints of the treatment recipient includes a spatial element. Therefore, the sensory information generation module 212 generates second sensory presentation information 700 for presenting, for example, information about the direction of moving a distal part (peripheral part) relative to the joint as visual information. The sensory information generation module 212 generates the second sensory presentation information 700 for presenting, for example, the direction of moving a distal part relative to a target joint on the bracelet-type LED device based on the movement around the joints of the treatment recipient recorded as the activity information 500.

[0122] In addition, a reference posture of the second sensory information output device 105 relative to the posture of the person to be treated may be determined in advance so that the second sensory information output device 105 can appropriately present the direction in which to move the distal portion around the joint. Furthermore, for example, movement information corresponding to the activity information 500 may be associated in advance with notations such as characters, symbols, patterns, etc. that present such information. This allows the sensory information generation module 212 to easily generate the second sensory presentation information 700 for presenting characters, etc., that correspond to the activity information 500 based on a predetermined association rule.

[0123] The sensory information generation module 212 can generate second sensory presentation information 700 for presenting the direction in which the distal portion is to be moved by, for example, a symbol such as an arrow or a flowing direction of light (i.e., moving the lighting position of the LED along the direction in which the distal portion is to be moved). The sensory information generation module 212 can also generate second sensory presentation information 700 so that, for example, letters, symbols, patterns, etc. representing the direction in which the distal portion is to be moved are fixed and presented at a predetermined position in the display area (e.g., an area on the back of the wrist). The sensory information generation module 212 can generate second sensory presentation information 700 so that, for example, lights, letters, symbols, etc. representing the direction in which the arm or fingers are to be moved flow or flash.

[0124] Furthermore, the sensory information generation module 212 provides a display area that expands so that information related to movement is presented three-dimensionally, taking into account the three-dimensional shape of the display area of ​​the second sensory information output device 105. Therefore, the second sensory presentation information 700 can be generated so that it is presented two-dimensionally or three-dimensionally. Furthermore, the sensory information generation module 212 can generate second sensory presentation information 700 so as to present information indicating the direction in which to move the distal part in an area of ​​the display area that is easily visible to the learner, for example, depending on the angle or position of the technician's arm or the position of the person being treated, which are obtained by the tracker. 3. Output of sensory information The learner wears a vibration generating device as the first sensory information output device 104, for example, on the wrist of his right hand, and wears a bracelet-type LED device as the second sensory information output device 105, for example, on the wrist of his left hand. Then, when the learner is ready to perform an intervention action on the subject, he sends an output instruction to the server 101 via the learner terminal 103, for example, as necessary, to satisfy the output criteria.

[0125] Then, for example, the server 101 and the learner terminal 103 cooperate to cause the first sensory information output device 104 to output first sensory information (vibration information) corresponding to the magnitude of the moment applied around the joint. Also, the second sensory information output device 105 to output second sensory information (visual information) corresponding to the direction of the moment applied around the joint. This allows the learner, such as a physical therapist, to learn appropriate intervention movements and range of motion training movements for the subject.

[0126] In the second embodiment, the information about the movement (second sensory information) presented on the second sensory information output device 105 indicates, for example, the direction in which the distal part of the person to be treated moves relative to the joint. It can include information indicating at least one of the direction, speed, strength, and changes therein of the target movement. In this way, by expressing information about the aspect of the movement, such as the direction, speed, strength, and changes therein, as sensory information, the learner can intuitively grasp the aspect of the movement to be learned while performing the movement, which is advantageous.

[0127] 4. Variations In the third embodiment, the information presented by the second sensory information output device 105 is the direction of the moment applied around the joint of the treatment recipient by the physical therapist or the like. However, the information presented by the second sensory information output device 105 may be, for example, (1) the movement direction of the left hand (supporting hand) for supporting the treatment recipient in the intervention performed by the physical therapist or the like, or (2) the amount of force to be applied when supporting the treatment recipient, or both. This allows the physical therapist or the like to learn, for example, how to use their arms appropriately and how much force to apply when supporting the treatment recipient.

[0128] [Embodiment 4] In this embodiment, the second sensory information output device 105 that presents visual information is an AR glass. The other configurations, functions, and effects may be the same as those of any of the first to third embodiments, and therefore, repeated description will be omitted.

[0129] FIG. 13 is a conceptual diagram showing a state (field of view) in which perceptual information is presented by AR glasses. The sensory information generation module 212 generates, for example, second sensory presentation information 700 for presenting information about movement as visual information as AR display data in a format that enables AR display. The trigger for overlaying the AR display data on the real world may be location (position information) type, vision (image recognition) type, or a combination of these. Alternatively, the AR display data may be object-tracking AR model data that recognizes the shape of an object as a marker in real time and displays an augmented AR image without requiring a marker. In this case, for example, the AR display data may be configured to track a shape that is identical or similar to the shape of a part of the learner's body (e.g., a hand) so that the orientation and dimensions are the same. However, in this embodiment, for example, as shown in FIG. 13, a marker on a wristband worn on the learner's wrist serves as the trigger for AR display.

[0130] This allows the AR image to be displayed with high precision near the learner's wrist (marker).With this configuration, visual information about movement can be presented as a more informative two-dimensional or three-dimensional image.

[0131] [Embodiment 5] In the third embodiment, the content and format of the sensory information to be presented to the learner for the writing action were predetermined. In contrast, the present embodiment is configured to accept designation of the content and format of the sensory information to be presented to the learner. The other configurations, actions, and effects may be the same as those of any of the first to fourth embodiments, and will not be described again.

[0132] 14 and 15 are diagrams each showing an example of a modality reception screen. The output control module 215 of the server 101 outputs a modality reception screen 1400 to, for example, a display (an example of the output device 405) of the learner terminal 103 (which may be the skilled worker terminal 102; the same applies below). The modality reception screen 1400 is configured to allow the learner to select the content of the first sensory information to be presented to the learner for, for example, a writing action, which is the content of the skill to be learned. Furthermore, the modality reception screen 1400 is configured to allow the learner to select, for example, the modality (information format) of the first sensory information to be presented to the learner.

[0133] Furthermore, the output control module 215 of the server 101 outputs a modality reception screen 1500, for example, to a display (an example of the output device 405) of the learner terminal 103. The modality reception screen 1500 is configured to allow the learner to select the content of the second sensory information to be presented to the learner for, for example, a writing action, which is the content of the acquired skill. The modality reception screen 1500 is also configured to allow the learner to select the modality of the second sensory information to be presented to the learner. Here, for example, the modality reception screen 1500 is configured so that a modality selected as a modality of the first sensory information cannot be selected as a modality of the second sensory information.

[0134] The output control module 215 is configured to receive, for example, for each learner or in response to a request from a learner, designation of the content of information to be presented and its modality via the modality reception screens 1400 and 1500. This allows the server 101 to obtain information designating the content of information to be presented and its modality.

[0135] Then, the sensory information generation module 212 of the server 101 generates, for example, the first sensory presentation information 600 and the second sensory presentation information 700 based on such a designation and the activity information 500. Furthermore, the first sensory information output device 104 and the second sensory presentation information 700 respectively generate the following based on the first sensory presentation information 600 and the second sensory presentation information 700: Outputs perceptual information in the specified format.

[0136] When a learner is trying to learn a new skill, the actions that are difficult to master may vary from person to person. With the above configuration, for example, the first perceptual information and the second perceptual information can be displayed in a modality that is specified by the learner. This can further improve the efficiency of the learner's learning of a new skill.

[0137] It should be noted that the present technology is not limited to the above-described examples and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0138] Each module disclosed in the above embodiments may be configured by combining multiple sub-modules. Furthermore, some or all of the operations or functions performed by one module may be performed or realized by another module.

[0139] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0140] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. The above-described embodiments disclose at least the configurations described in the claims. [Explanation of symbols]

[0141] 1...skill acquisition support system, 101...server, 102...skilled worker terminal, 103...learner terminal, 104...first perceptual information output device, 105...second perceptual information output device, 211...activity information acquisition module, 212...perceptual information generation module, 213...first perceptual information output module, 214...second perceptual information output module, 215...output control module

Claims

1. an activity information acquisition unit that acquires activity information related to the activity of the first living organism; a sensory information output unit configured to output sensory information in a perceptible region of the second living body; a control unit that outputs sensory information to the sensory information output unit; Equipped with the sensory information output unit includes a first sensory information output unit and a second sensory information output unit, When the output standard is satisfied, the control unit causing the first sensory information output unit to output first sensory information corresponding to a first activity among the first biological activity information; causing the second sensory information output unit to output second sensory information corresponding to a second activity of the first living body activity information, the second sensory information being in an information format different from that of the first sensory information; Skill acquisition support system.

2. The control unit when the first biological activity information satisfies a first criterion, causing the first sensory information output unit to output the first sensory information with a first sensory value; when the first biological activity information satisfies a second criterion, causing the first sensory information output unit to output the first sensory information with a second sensory value different from the first sensory value; The skill acquisition support system according to claim 1 .

3. The control unit when the first biological activity information satisfies a first criterion, causing the second sensory information output unit to output the second sensory information with a first sensory value; when the first biological activity information satisfies a second criterion, causing the second sensory information output unit to output the second sensory information with a second sensory value different from the first sensory value; The skill acquisition support system according to claim 1 .

4. The control unit outputting the first sensory information as tactile information to the first sensory information output unit; outputting the second perceptual information to the second perceptual information output unit as visual information; The skill acquisition support system according to claim 1 .

5. the activity information of the first living body is information about a movement of the first living body, The control unit outputting, as the first perceptual information, a temporal element of the movement of the first living body as information related to a sense of touch to the first perceptual information output unit; causing the second sensory information output unit to output, as the second sensory information, a spatial element of the movement of the first living organism as visual information; The skill acquisition support system according to claim 4.

6. At least one of the first sensory information and the second sensory information includes information representing at least one of a direction, a speed, a strength, and a change therein of a motion of the first living organism. The skill acquisition support system according to claim 1 .

7. the activity information of the first living body is information about the movement of the same part of the first living body, the control unit controls the first sensory information output unit and the second sensory information output unit to output information about the movement of the one part of the first living organism as the first sensory information and the second sensory information, which are different from each other; The skill acquisition support system according to claim 1 .

8. The control unit outputting information about the strength of the movement of the same part of the first living body as information about the sense of touch to the first perception information output unit; outputting information about the direction of movement of the same part of the first living body as information about vision to the second perception information output unit; The skill acquisition support system according to claim 7.

9. the activity information of the first living body is information regarding the movements of two or more different parts of the first living body, The control unit causing the first sensory information output unit to output information regarding a movement of a first part of the first living body as the first sensory information; causing the second sensory information output unit to output information relating to the movement of a second part of the first living body as the second sensory information; The skill acquisition support system according to claim 1 .

10. The control unit receiving designation of an information format of the first sensory information to be output by the first sensory information output unit and an information format of the second sensory information to be output by the second sensory information output unit; outputting first sensory information and second sensory information according to a designated information format from the first sensory information output unit and the second sensory information output unit, respectively; The skill acquisition support system according to claim 1 .

11. an activity information acquisition unit that acquires activity information related to the activity of the first living organism; a sensory information output unit configured to output sensory information in a perceptible region of the second living body; a control unit that outputs sensory information to the sensory information output unit; Equipped with the sensory information output unit includes a first sensory information output unit and a second sensory information output unit, When the output standard is satisfied, the control unit causing the first sensory information output unit to output first sensory information corresponding to a first activity among the first biological activity information; causing the second sensory information output unit to output second sensory information corresponding to a second activity of the first living body activity information, the second sensory information being in an information format different from that of the first sensory information; Information processing device.

12. The computer acquiring first sensory information corresponding to a first activity from activity information related to a first biological activity acquired in advance; acquiring second sensory information corresponding to a second activity of the first biological activity information, the second sensory information having an information format different from that of the first sensory information; causing a first sensory information output unit to output the first sensory information and a second sensory information output unit to output the second sensory information; A control method for performing

13. On the computer, acquiring first sensory information corresponding to a first activity from activity information related to a first biological activity acquired in advance; acquiring second sensory information corresponding to a second activity of the first biological activity information, the second sensory information having an information format different from that of the first sensory information; causing a first sensory information output unit to output the first sensory information and a second sensory information output unit to output the second sensory information; A program to execute.

Images