Information processing system, information processing method, and information processing program

Abstract

To provide feedback for correcting a motion to a user who is performing a motion accompanying a skill to be learned on a timely basis.SOLUTION: An information processing system includes an acquisition unit that acquires first measurement information obtained by measuring a motion of a user, and a feedback unit that gives feedback to the user so as to reduce an error between the motion of the user indicated by the first measurement information and a target motion. The acquisition unit acquires second measurement information obtained by measuring a motion of the user to whom the feedback is given. In addition, the feedback unit adjusts a timing at which new feedback is given to the user on the basis of the motion of the user indicated by the second measurement information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] Patent document 1 describes a technology that measures a learner's movement pattern, calculates the muscle activity that needs to be corrected to achieve a target movement pattern based on the measured movement pattern, and provides the learner with electrical or vibration stimulation according to the calculated muscle activity. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 7-36363 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 has the problem that the timing of the stimulus application may be too early or too late depending on individual differences, physical condition, etc., making it difficult to achieve the target exercise pattern.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and an exemplary purpose thereof is to provide a technology that provides timely feedback to a user performing an action associated with a skill to be mastered, so as to correct the action. [Means for solving the problem]

[0006] An information processing system according to an exemplary aspect of the present disclosure comprises an acquisition means for acquiring first measurement information that measures a user's movement, and a feedback means for providing feedback to the user so as to reduce an error between the user's movement indicated by the first measurement information and a target movement, wherein the acquisition means acquires second measurement information that measures the user's movement to which the feedback has been given, and the feedback means adjusts the timing of providing new feedback to the user based on the user's movement indicated by the second measurement information.

[0007] An information processing method according to an exemplary aspect of the present disclosure includes an acquisition process in which at least one processor acquires first measurement information measuring a user's movement, and a feedback process in which the at least one processor provides feedback to the user so as to reduce an error between the user's movement indicated by the first measurement information and a target movement, wherein in the acquisition process, the at least one processor acquires second measurement information measuring the user's movement to which the feedback has been given, and in the feedback process, the at least one processor adjusts the timing of providing new feedback to the user based on the user's movement indicated by the second measurement information.

[0008] An information processing program according to an exemplary aspect of the present disclosure is a program that causes a computer to function as an information processing system, and causes the computer to function as an acquisition means that acquires first measurement information that measures a user's movements, and a feedback means that provides feedback to the user so as to reduce an error between the user's movements indicated by the first measurement information and a target movement, wherein the acquisition means acquires second measurement information that measures the user's movements to which the feedback has been given, and the feedback means adjusts the timing of providing new feedback to the user based on the user's movements indicated by the second measurement information. [Effects of the Invention]

[0009] According to an exemplary aspect of the present disclosure, an exemplary effect can be achieved in which a technology can be provided that provides timely feedback to a user performing an action associated with a skill to be mastered, so as to correct the action. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram illustrating a configuration of an information processing system according to the present disclosure. [Figure 2] FIG. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. [Figure 3] 1 is a diagram schematically illustrating an overview of an information processing system according to the present disclosure. [Figure 4] 1 is a block diagram illustrating a configuration of an information processing system according to the present disclosure. [Figure 5] FIG. 1 is a flow diagram illustrating a flow of an information processing system according to the present disclosure. [Figure 6] FIG. 2 is a block diagram showing the hardware configuration of a computer that functions as each device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technologies (part or all of the products or methods) employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] First Exemplary Embodiment A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technique employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technique employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technique shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0013] (Configuration of Information Processing System 1) The configuration of the information processing system 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the information processing system 1. As shown in FIG. 1, the information processing system 1 includes an acquisition unit 11 and a feedback unit 12. The acquisition unit 11 is an example of a configuration that realizes an acquisition means. The feedback unit 12 is an example of a configuration that realizes a feedback means. For example, if the information processing system 1 includes at least one processor, the acquisition unit 11 and the feedback unit 12 are realized by the at least one processor executing a program. Note that the information processing system 1 may be configured by one computer including at least one processor, or may be configured by multiple computers each including at least one processor.

[0014] The acquisition unit 11 acquires first measurement information obtained by measuring a user's motion. The feedback unit 12 provides feedback to the user so as to reduce an error between the user's motion indicated by the first measurement information and a target motion.

[0015] For example, the movement may be a movement associated with a skill. Furthermore, the acquisition unit 11 may measure the user's movement associated with a skill the user is currently learning. However, the acquisition unit 11 is not limited to this, and may also measure the user's movement when the user is actually performing the skill. The skill may be, for example, a skill that requires repeated training to master. Furthermore, the skill may be a skill for which it is difficult to recreate an actual environment for training. Furthermore, the skill may impose a heavy burden on the user due to repeated training. Furthermore, the skill may be a skill for which it is difficult for the user to evaluate the movement. Examples of application of skills include, but are not limited to, playing a musical instrument, sports, dancing, astronaut skills, medical professional techniques, artisan skills, diving, makeup, craftsmanship, precision work, dental techniques, and archery. The user's movement is not necessarily limited to a movement associated with an activity referred to as a skill.

[0016] As an example, the acquisition unit 11 acquires the first measurement information using, for example, a sensor. Specific examples of the sensor include, but are not limited to, an acceleration sensor, a gyro sensor, a positioning sensor, a distance measurement sensor, a camera, and a combination of some or all of these. Furthermore, the sensor may be attached to the user's body or may be installed at a location remote from the user.

[0017] For example, if the target motion is a motion associated with a skill, it may be a motion associated with the skill performed by a skilled user of the skill. The target motion may be represented in the same format as the first measurement information. For example, if the first measurement information includes a time series of sensor information obtained from one or more sensors, the target information may be represented as a time series of the sensor information. In this case, the error between the motion indicated by the first measurement information and the target motion can be calculated by comparing information in the same format. Furthermore, the target motion may be represented in a format different from that of the first measurement information. For example, the first measurement information may include a time series of the sensor information described above, and the target motion may be represented as a time series of a 3D model or point cloud data. In this case, the error between the motion indicated by the first measurement information and the target motion can be calculated by comparing the target motion with a motion in the same format as the target motion calculated based on the first measurement information.

[0018] Furthermore, the error between the user's movement indicated by the first measurement information and the target movement may include a temporal difference or a spatial difference between the movement indicated by the first measurement information and the target movement. A specific example of the temporal difference can be calculated by matching each time point in the movement indicated by the first measurement information with each time point in the target movement and comparing the corresponding time points in time. A specific example of the spatial difference can be calculated by matching each body part in the movement indicated by the first measurement information with each body part in the target movement and comparing the corresponding body parts at the corresponding time points in space.

[0019] The feedback unit 12 provides the user with feedback to reduce the above-mentioned error, for example, by using a feedback providing device. Specific examples of the feedback providing device include a stimulating device, a corrective device, a display, etc.

[0020] For example, the feedback unit 12 may provide feedback to the user using a stimulation device that applies electrical or vibration stimulation to the user's body. Such a stimulation device is attached to the user's body. Alternatively, the feedback unit 12 may provide feedback to the user using a correction device that corrects movement by applying an external force to the user's body. An example of a correction device is an assistive device that assists human joint movement and has a correction function attached to it. Such a correction device is attached to the user's body.

[0021] Furthermore, for example, the feedback unit 12 may provide feedback to the user using a display that presents visual information to the user. For example, the visual information as feedback may be, but is not limited to, a guide object that indicates the target timing for the next action, the target position of the body part that should be moved next, etc. As an example, such a display may be attached to the user's body as a wearable device, or may be located at a position away from the user so as to be visible to the user.

[0022] To reduce the error, the feedback unit 12 may provide feedback to the user if the error is equal to or greater than a threshold, and may not provide feedback to the user if the error is equal to or less than the threshold. Furthermore, the feedback unit 12 may provide greater feedback to the user as the error increases. The magnitude of the feedback may be the strength of the feedback, the length of the feedback, or a combination of these.

[0023] The acquisition unit 11 also acquires second measurement information that measures the user's behavior for which feedback has been given. The feedback unit 12 also adjusts the timing for giving new feedback to the user based on the user's behavior indicated by the second measurement information.

[0024] For example, the acquisition unit 11 may acquire the second measurement information using a sensor. An example of a sensor used to acquire the second measurement information is the same as the example of the sensor used to acquire the first measurement information, and therefore detailed description thereof will not be repeated.

[0025] Furthermore, the timing at which the feedback unit 12 provides feedback to the user is variable, and "adjusting the timing" means "determining a new timing based on the previous timing." For example, the feedback unit 12 may adjust the timing at which the above-mentioned stimulation device provides a stimulus to the user, the timing at which the above-mentioned correction device provides an external force to the user, or the timing at which visual information is presented to the user by the above-mentioned display.

[0026] For example, the feedback unit 12 may adjust the timing of providing new feedback to the user based on the error between the movement indicated by the second measurement information and the target movement. An example of the error between the movement indicated by the second measurement information and the target movement has been described as an example of the error between the movement indicated by the first measurement information and the target movement, and therefore detailed description thereof will not be repeated.

[0027] For example, the feedback unit 12 may adjust the timing according to the time difference between the movement indicated by the second measurement information and the target movement. Specifically, the feedback unit 12 may determine the timing to be earlier than the previous timing as the user's movement indicated by the second measurement information is later in time with respect to the target movement. Alternatively, the feedback unit 12 may determine the timing to be later than the previous timing as the movement indicated by the second measurement information is earlier in time with respect to the target movement.

[0028] Furthermore, for example, the feedback unit 12 may adjust the timing according to the spatial difference between the movement indicated by the second measurement information and the target movement. Specifically, the feedback unit 12 may determine a timing that is earlier than the previous timing when the user's movement indicated by the second measurement information is smaller than the target movement. The feedback unit 12 may determine a timing that is later than the previous timing when the movement indicated by the second measurement information is larger than the target movement. Note that the comparison of the magnitude of the movement can be made, for example, by comparing the length of the distance a certain body part is moved.

[0029] (Effects of Information Processing System 1) As described above, the information processing system 1 is configured to include the above-described acquisition unit 11 and feedback unit 12. Therefore, for example, a user who has received feedback to correct a movement to be mastered can receive new feedback at an appropriate timing in accordance with the next movement performed in response to the feedback. As a result, the information processing system 1 has the effect of being able to provide timely feedback to a user performing a movement associated with a skill to be mastered to correct the movement.

[0030] Furthermore, in the information processing system 1, the feedback unit 12 may provide feedback to the user using a stimulation device that provides electrical or vibration stimulation to the user's body, a correction device that corrects movements by applying external force to the user's body, or a display that presents visual information to the user. Therefore, the information processing system 1 has the effect of allowing the user to receive electrical or vibration stimulation, movement correction by external force, or visual information at an appropriate time as feedback for correcting the movements to be mastered.

[0031] (Flow of information processing method S1) The flow of information processing method S1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of information processing method S1. For example, when information processing system 1 includes at least one processor, information processing method S1 is realized by at least one processor executing an information processing program that causes a computer to function as acquisition unit 11 and feedback unit 12. As shown in Fig. 2, information processing method S1 includes first measurement information acquisition processing S11, feedback processing S12, second measurement information acquisition processing S13, and adjustment processing S14.

[0032] The first measurement information acquisition process S11 is an example of a process included in the acquisition process described in the claims. In the first measurement information acquisition process S11, at least one processor (e.g., the acquisition unit 11) acquires first measurement information that measures the user's movement.

[0033] The feedback process S12 is an example of a process included in the feedback process described in the claims. In the feedback process S12, at least one processor (e.g., the feedback unit 12) provides feedback to the user so as to reduce an error between the user's motion indicated by the first measurement information and a target motion.

[0034] The second measurement information acquisition process S13 is an example of a process included in the acquisition process described in the claims. In the second measurement information acquisition process S13, at least one processor (e.g., the acquisition unit 11) acquires second measurement information that measures the user's behavior to which feedback has been given.

[0035] The adjustment process S14 is an example of a process included in the feedback process described in the claims. In the adjustment process S14, at least one processor (e.g., the feedback unit 12) adjusts the timing of providing new feedback to the user based on the user's behavior indicated by the second measurement information.

[0036] (Effect of information processing method S1) As described above, the information processing method S1 includes the first measurement information acquisition process S11, the feedback process S12, the second measurement information acquisition process S13, and the adjustment process S14. Therefore, the information processing method S1 can achieve the same effects as the information processing system 1.

[0037] Second Exemplary Embodiment A second exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technology employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0038] (Outline of Information Processing System 1A) FIG. 3 is a diagram schematically illustrating an overview of an information processing system 1A. This exemplary embodiment is intended for a case where a user performs a periodic motion. For example, the information processing system 1A can be used for a user who is learning a skill involving a periodic motion. The periodic motion is composed of repeatable unit motions. For example, a specific example of a periodic motion is running. In this case, the unit motion may be a motion of alternately swinging up the left and right limbs once each. Another specific example of a periodic motion is playing a musical instrument or dancing, which is composed of repeated parts. In this case, the unit motion may be the repeated part. However, the periodic motion and the unit motion are not limited to the examples described above.

[0039] As shown in FIG. 3, in the information processing system 1A, a camera 40 captures images of a user running to improve their running technique. The server 10 acquires the images (an example of first measurement information) captured by the camera 40 via a first terminal 20 connected via a network NW. For example, the camera 40 may be installed in a location where the user can be photographed in a place (e.g., a running ground) where the user is to improve their running technique, and the first terminal 20 may be connected to the camera. The first terminal 20 incorporating the camera 40 may be carried by a collaborator (e.g., a coach, a manager, etc.). In the information processing system 1A, other sensors capable of acquiring the first measurement information and the second measurement information may be used instead of or in addition to the camera 40. However, the following description will focus on an example in which the camera 40 is used.

[0040] The server 10 compares the movement shown in the captured image with the target movement, and if the target movement is not achieved, provides feedback from the feedback giving devices 50a to 50d to the user via the second terminal 30 connected via the network NW. While FIG. 3 shows four feedback giving devices 50a to 50d, the number of feedback giving devices 50a to 50d is not limited to this. Hereinafter, when there is no need to distinguish between the feedback giving devices 50a to 50d, each will be simply referred to as a feedback giving device 50. When multiple feedback giving devices 50 are used, at least two of the feedback giving devices 50 may be the same or different types. For example, the feedback giving device 50a may be a stimulation device, and the feedback giving device 50b may be a correction device. Details of the stimulation device and the correction device will be described later.

[0041] Furthermore, the feedback providing device 50 may be attached to a body part (e.g., a joint) of the user and connected to the second terminal 30. The feedback providing device 50 may be built into the second terminal 30. The second terminal 30 may be carried by the user while running, or may be attached to the user as a wearable terminal. The feedback providing device 50 may be, for example, a display, and in that case, may be attached to the user as wearable glasses or may be installed in a location visible to the user. The second terminal 30 does not necessarily have to be carried by or attached to the user, but may be installed in a location away from the user, for example, a location for mastering running techniques.

[0042] Furthermore, the server 10 receives the feedback and acquires, via the first terminal 20, a captured image (an example of second measurement information) of the user's running movements taken by the camera 40. The server 10 also adjusts the timing of providing new feedback to the user based on the movements shown in the captured image. Note that while FIG. 3 illustrates an example in which the skill the user is to learn is running, this exemplary embodiment is not limited to running and can also be applied to other skills involving periodic movements.

[0043] (Configuration of information processing system 1A) FIG. 4 is a block diagram showing the configuration of an information processing system 1A. As shown in FIG. 4, the information processing system 1A includes a server 10, a first terminal 20, a second terminal 30, a camera 40, and a feedback providing device 50. The server 10, the first terminal 20, and the second terminal 30 are communicatively connected via a network NW. The network NW includes, as an example, a 5G core network NW-1 and 5G base stations NW-2 and NW-3. In other words, in the example of FIG. 4, the network NW is configured to include a 5G (fifth generation mobile communication system) network. Note that the configuration of the network NW is not limited thereto, and may include a mobile communication network such as 4G (fourth generation mobile communication system), local 5G, or local 4G instead of or in addition to the 5G network. Furthermore, the network NW is not limited to including a mobile communication system, and may not necessarily include a mobile communication system. However, if the network NW includes a 5G network, the 5G network's high speed, large capacity, ultra-low latency, and multiple simultaneous connections have the advantage of more effectively providing timely feedback to users using the information processing system 1A.

[0044] (Server 10 configuration) The server 10 includes an analysis unit 13, a database 14, and an identification unit 15 in addition to the acquisition unit 11 and feedback unit 12 included in the information processing system 1. The identification unit 15 is an example of a configuration that realizes an identification means. When the information processing system 1A includes at least one processor, the identification unit 15 is realized by the at least one processor executing a program.

[0045] The database 14 stores information indicating target movements associated with skills. For example, in the case of running, the database 14 stores target movements associated with running. The target movements associated with running may indicate, for example, ideal timing for swinging up each of the left and right limbs, ideal swing-up positions for each of the left and right limbs, ideal swing-up angles for each of the left and right elbow joints and knee joints, etc., but are not limited to these.

[0046] Furthermore, the database 14 stores user information corresponding to each user who uses the information processing system 1A. The user information includes, for example, a user ID for identifying the user, an initial timing of feedback corresponding to the user, and the like. The initial timing is applied, for example, as the timing at which feedback is given for the first time to a user who performs an action associated with a skill. The user information may also include information indicating an adjustment timing adjusted according to the user. The user information may also include information indicating the attributes of the user (for example, age, gender, etc.).

[0047] The analysis unit 13 compares the user's motion indicated by the first measurement information with the target motion, thereby analyzing the error therebetween. For example, the analysis unit 13 may analyze the temporal difference and / or spatial difference between the user's motion indicated by the first measurement information and the target motion as the error. Specific examples of the temporal difference and spatial difference are the same as those in the first exemplary embodiment. The analysis unit 13 may also refer to user information to perform the analysis.

[0048] Furthermore, the analysis unit 13 identifies the timing of a reaction movement in response to the feedback in the user's movement indicated by the second measurement information. For example, the analysis unit 13 may identify, in the user's movement indicated by the second measurement information, the timing at which a specific movement is performed after the feedback is given as the timing of the reaction movement.

[0049] The identification unit 15 identifies a user. For example, the identification unit 15 may identify a user ID that identifies a user who is trying to acquire a skill. As an example, the identification unit 15 may identify a user based on an image captured by the camera 40, or may identify a user based on authentication information acquired from the user (for example, an ID and password, biometric authentication information, etc.).

[0050] The feedback unit 12 adjusts the timing of providing new feedback to the user according to the identified user. For example, the feedback unit 12 may adjust the previous timing applied to the user based on the magnitude of deviation of the timing of the reaction movement in the movement indicated by the second measurement information from the timing of the target movement. Furthermore, the feedback unit 12 may apply a user-specific amount as the amount of timing adjustment to adjust the timing according to the magnitude of the deviation of the reaction movement timing. For example, even if the timing of users' reaction movements deviates from the target movement by the same amount, the appropriate amount of timing adjustment may vary from person to person. Therefore, for example, for the same degree of deviation in the timing of the reaction movements, the amount of timing adjustment applied to one user may be different from the amount of timing adjustment applied to another user. Such a user-specific adjustment amount may be determined, for example, according to the user's attributes.

[0051] Furthermore, the feedback unit 12 adjusts the intensity of the new feedback based on the user's movement indicated by the second measurement information. For example, the feedback unit 12 may increase the intensity of the feedback as the deviation between the timing of the reaction movement and the timing of the target movement increases. The intensity of the feedback may be, for example, the magnitude of a stimulus applied to the user by a stimulus device, which is an example of the feedback providing device 50. The intensity of the feedback may be, for example, the magnitude of an external force applied to the user by a correction device, which is an example of the feedback providing device 50. The intensity of the feedback may be, for example, the brightness of visual information presented to the user by a display, which is an example of the feedback providing device 50, the size of a display area for the visual information, or a combination thereof.

[0052] Furthermore, the feedback unit 12 adjusts the timing of new feedback to be given to the user for the next unit action based on the second measurement information measured for the unit actions that constitute the periodic action. As a result, feedback is given to the user for each unit action. Furthermore, the timing of giving feedback to the user for each unit action is adjusted. For example, the feedback unit 12 determines a timing adjusted by advancing or delaying the timing at which feedback was given for the previous unit action. Hereinafter, the adjusted timing will also be referred to as the adjusted timing. Furthermore, the feedback unit 12 gives new feedback to the user for performing the next unit action at the adjusted timing.

[0053] (Flow of information processing method S1A) The information processing system 1A configured as described above executes an information processing method S1A. Fig. 5 is a flow diagram illustrating the flow of the information processing system 1A. As shown in Fig. 5, the information processing method S1A includes steps S101 to S110. The following description focuses on an example in which running, which is an example of a periodic movement, is applied as a skill that the user should acquire. A unit movement in running is assumed to be a movement in which the left and right limbs are alternately swung up once each.

[0054] Step S101 is an example of the identification process. In step S101, the identification unit 15 identifies a user. For example, the identification unit 15 may acquire a captured image taken by the camera 40 from the first terminal 20, and identify a user included as a subject in the captured image based on the acquired captured image. Furthermore, for example, the identification unit 15 may acquire authentication information of the user from the second terminal 30, and identify the user based on the acquired authentication information.

[0055] Step S102 is an example of a first measurement information acquisition process. In step S102, the acquisition unit 11 acquires, from the first terminal 20, a captured image (an example of first measurement information) of a user who is acquiring a skill, captured by the camera 40. For example, the captured image may be a captured image of one unit movement performed by the user. In the example of running, the captured image of the unit movement may be a time series of still images including, as a subject, the user performing one movement of swinging up his or her arms and legs.

[0056] In step S103, the analysis unit 13 compares and analyzes the user's motion shown in the captured image with a target motion. In the example of running, the analysis unit 13 may calculate the angle of the right elbow joint when the right hand is raised from the captured image as a measurement value indicating the user's motion. Furthermore, a target value for the angle of the right elbow joint may be recorded in the database 14 as information indicating the target running motion. In this case, the analysis unit 13 may calculate the difference between the measurement value of the angle of the right elbow joint obtained from the captured image and the target value as the comparison result. Note that the analysis unit 13 may calculate multiple types of measurement values ​​indicating the user's motion (for example, the angles of the left and right elbow joints, the positions at which the left and right hands are raised, the angles of the left and right knee joints, the positions at which the left and right knees are raised, etc.) from the captured image.

[0057] In step S104, the feedback unit 12 determines whether the error between the user's motion shown in the captured image and the target motion is outside the allowable range based on the comparison result. In the example of running, if the difference between the measured value of the angle of the right elbow joint and the target value is equal to or greater than a threshold, it may be determined that the error is outside the allowable range. Note that specific examples of conditions for determining that the error is outside the allowable range are not limited to these. Furthermore, if multiple types of measured values ​​are calculated, it may be determined whether each of them is outside the allowable range.

[0058] If the determination in step S104 is No (that is, if the error is within the allowable range), step S110, which will be described later, is executed. If the determination in step S104 is Yes, the next step S105 is executed.

[0059] Step S105 is an example of feedback processing. In step S105, the feedback unit 12 controls the feedback-providing device 50 via the second terminal 30 to provide feedback to the user at a timing appropriate to the user so that the error described above is reduced. The timing appropriate to the user may be an initial timing included in the user information stored in the database 14. Furthermore, the feedback unit 12 may apply an intensity appropriate to the error described above as the intensity of the feedback to be provided to the user at the initial timing. For example, a larger intensity may be set as the error increases.

[0060] Furthermore, when step S105 is executed for the second or subsequent time as a result of a determination of No in step S110 described later, the timing according to the user is the adjustment timing determined the previous time step S109 described later was executed. Furthermore, the feedback unit 12 may apply, as the strength of the feedback to be given to the user at the adjustment timing, an intensity according to the error described above, adjusted in accordance with the deviation in the timing of the reaction action identified the previous time step S107 described later was executed.

[0061] The initial timing may be determined, for example, by being earlier than a predetermined time based on a predicted timing at which the action indicated by the first measurement information is next predicted to be performed. Such predicted timing may be analyzed based on a history of the first measurement information acquired up to that point. The initial timing may also be set by prior calibration.

[0062] In the running example, suppose that the measured angle of the right elbow joint in the immediately preceding swing-up motion is determined to be greater than the target position by a threshold value or more and outside the allowable range. In this case, a stimulation device, which is an example of a feedback-imparting device 50, attached to the user's right elbow may be used to impart vibration to the user's right elbow. Furthermore, if step S105 is performed for the first time, the timing at which the vibration is imparted to the user's right elbow may be an initial timing that is advanced by a predetermined length from the predicted timing at which the right elbow will next be swung up. Furthermore, if step S105 is performed for the second time, the timing at which the vibration is imparted to the user's right elbow may be an adjusted timing that is advanced from the initial timing.

[0063] Step S106 is an example of a second measurement information acquisition process. In step S106, the acquisition unit 11 acquires a captured image (an example of second measurement information) of the user to whom feedback has been given, captured by the camera 40, from the first terminal 20. For example, the captured image may be an image of one unit movement performed by the user.

[0064] In step S107, the analysis unit 13 identifies the timing of the reaction movement in the user's movement indicated by the second measurement information.

[0065] In the example of running, the analysis unit 13 may identify a measurement value of the timing of swinging up the right hand from a captured image of a unit movement of the user to which feedback has been given. The measurement value of the swing-up timing may be expressed as a time standardized using the unit period from the start to the end of the unit movement indicated by the second measurement information (for example, a time in which the start time of the unit movement is 0 and the end time is 1). Note that the analysis unit 13 may calculate multiple types of measurement values ​​(for example, the timing of swinging up each of the left and right hands, the timing of swinging up each of the left and right feet, etc.) as the timing of the user's reaction movement from the captured image.

[0066] In step S108, the analysis unit 13 determines whether the timing deviation of the reaction movement is outside the allowable range. The timing deviation of the reaction movement can be calculated as the difference between the measured value and the target value of the timing of the reaction movement. For example, the database 14 may record a target timing value as the target movement. The target timing value may be expressed as a time standardized with the unit period being the period from the start to the end of the target unit movement. In the example of running, the database 14 may record a target value for the timing of raising the right hand. In this case, the analysis unit 13 may determine that the timing deviation is outside the allowable range if the difference between the measured value and the target value of the timing of raising the right hand is equal to or greater than a threshold. Note that if multiple types of measured values ​​are calculated as the timing of the reaction movement, it may be determined whether each of them is outside the allowable range.

[0067] If the determination in step S108 is No (that is, if the deviation is within the allowable range), step S110, which will be described later, is executed. If the determination in step S108 is Yes, the next step S109 is executed.

[0068] Step S109 is an example of an adjustment process. In step S109, the feedback unit 12 determines an adjustment timing for providing new feedback depending on the magnitude of the deviation in the timing of the reaction behavior. For example, the feedback unit 12 may determine an adjustment timing that is advanced or delayed by an amount corresponding to the deviation from the timing of the previously provided feedback.

[0069] In the running example, suppose that the measured value of the timing of the right hand swing in a unit movement performed by a user receiving feedback is delayed from the target value by more than a threshold value. In this case, it is determined that the timing deviation of the reaction movement is outside the acceptable range. Therefore, the adjustment timing for applying vibration to the user's right elbow may be determined to be earlier than the previous timing by an adjustment amount corresponding to the deviation.

[0070] Step S110 is executed when step S104 or step S108 judges No, or following step S109. In step S110, the server 10 judges whether to terminate the information processing method S1A. For example, the server 10 may judge to terminate the information processing method S1A when the elapsed time from the start of the information processing method S1A exceeds a predetermined time, when the number of repetitions of the unit action exceeds a predetermined number, or when an end operation is received from the user. If the judgment in step S110 is Yes, the information processing method S1A terminates.

[0071] If the determination in step S110 is No, the process from step S103 is repeated. In this case, in step S103, the captured image acquired as the second measurement information in the previous step S106 is treated as the first measurement information and processed. Furthermore, if the error between the unit movement indicated by the captured image and the target movement is outside the allowable range, feedback is provided to the user in step S105. At this time, the adjustment timing determined in the previous step S109 is applied as the timing for providing feedback in step S105. Furthermore, the strength of the feedback provided in step S105 may be adjusted according to the timing difference of the reaction movement identified in the previous step S107. In this way, feedback is provided for each unit movement to the user who continues the cyclical movement, and the timing and strength of the feedback are repeatedly adjusted for each unit movement.

[0072] In the above-described information processing method S1A, a specific example has been described in which the body part to be measured in the captured image (an example of the first measurement information and the second measurement information) and the body part to which feedback is given are the same. However, these are not limited to being the same and may be different. For example, measurement values ​​for multiple types of body parts may be calculated from the captured image, and a single feedback-giving device 50 may be attached to the user's body (for example, on the wrist). In this case, feedback for correcting any of the positions of the left and right hand tips, the angles of the left and right elbows, the positions of the left and right thighs, and the angles of the left and right knees may be given to the user by the single feedback-giving device 50.

[0073] (Effects of Information Processing System 1A) As described above, the information processing system 1A is configured to further include the identification unit 15 that identifies a user, and the feedback unit 12 adjusts the timing described above in accordance with the identified user. Therefore, in addition to the effects of the information processing system 1, the information processing system 1A can provide feedback to a user performing an action associated with a skill to be acquired, at an appropriate timing depending on the user, to correct the action.

[0074] Furthermore, the information processing system 1A is configured such that the feedback unit 12 adjusts the strength of the new feedback based on the user's movement indicated by the second measurement information. Therefore, in addition to the effects of the information processing system 1, the information processing system 1A can provide an effect of providing feedback with appropriate strength to a user performing a movement associated with a skill to be mastered, for correcting the movement.

[0075] Furthermore, the information processing system 1A is configured such that, when a user performs a periodic movement, the timing of new feedback to be given to the user for the next unit movement is adjusted based on second measurement information measured for the unit movements that constitute the periodic movement. Therefore, the information processing system 1A can provide timely feedback to a user performing a periodic movement associated with a skill to be acquired, to correct the next unit movement. As a result, in addition to the effects of the information processing system 1, the information processing system 1A can also provide an effect of supporting the acquisition of a skill involving a periodic movement.

[0076] [Modification] The information processing system 1A has been described as being intended for cases where a user performs periodic movements. However, the information processing system 1A can be modified to be intended for cases where a user performs non-periodic movements. For example, the information processing system 1A can be used for a user who is learning a skill involving non-periodic movements. Specific examples of non-periodic movements include, but are not limited to, playing a musical instrument and dancing.

[0077] In this modification, the feedback section 12 adjusts the timing of new feedback to be given to the user in the next non-periodic motion, based on the second measurement information measured in the non-periodic motion.

[0078] For example, the acquisition unit 11 measures first measurement information at predetermined timings during the first non-periodic movement performed by the user. Furthermore, the feedback unit 12 determines and records the timing of feedback to correct the user's movement during the next non-periodic movement so as to reduce the error between the user's movement indicated by the first measurement information and the target movement. For example, when the user practices a dance for the first time, the position of a predetermined body part may be measured for each measure as the first measurement information. Furthermore, the timing of feedback to be given to the user during the next practice may be determined and recorded so as to reduce the error between the position of the predetermined body part and the target position.

[0079] Note that the "first time" refers to the time when the first measurement information is recorded, and is not limited to the first time when the user actually performs the non-periodic action. Also, the "second time," "third time," etc. hereinafter refer to the times counted from the time when the first measurement information is recorded, and are not limited to the second or third time when the user actually performs the non-periodic action.

[0080] Furthermore, when the user performs the non-periodic movement for the second time, the feedback unit 12 provides the feedback at each of the recorded feedback timings. For example, when the user practices a dance for the second time, the feedback may be provided to the user at the feedback timings recorded for each measure.

[0081] Furthermore, the acquisition unit 11 measures the second measurement information each time feedback is given during the second non-periodic movement performed by the user. Furthermore, the feedback unit 12 records adjustment timing obtained by adjusting the feedback timing for correcting the user's movement at a predetermined timing during the next non-periodic movement, based on the user's movement indicated by the second measurement information. For example, when the user practices a dance for the second time, the adjustment timing of the feedback to be given to the user for each measure during the next practice may be determined so as to reduce a timing discrepancy in the user's reaction movement to the feedback given for each measure.

[0082] Furthermore, when the user performs the non-periodic movement for the third time, the feedback unit 12 provides the feedback at each of the adjustment timings of the recorded feedback. For example, when the user practices a dance for the third time, the feedback may be provided to the user at the adjustment timings recorded for each measure.

[0083] In this modified information processing system 1A, when the user performs a non-periodic movement, the feedback unit 12 adjusts the timing of new feedback to be given to the user for the next non-periodic movement based on the second measurement information measured during the non-periodic movement. Therefore, the information processing system 1A can provide timely feedback to the user performing a non-periodic movement associated with a skill to be acquired, to correct the next non-periodic movement. As a result, the information processing system 1A can achieve the effect of supporting the acquisition of skills involving non-periodic movements, in addition to the effect achieved by the information processing system 1.

[0084] [Software implementation example] Some or all of the functions of the devices constituting the information processing systems 1 and 1A may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0085] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 6. Figure 6 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0086] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to operate as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0087] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0088] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0089] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0090] Furthermore, the functions of each of the above devices may be realized by a single processor provided in a single computer, by multiple processors provided in a single computer working in cooperation, or by multiple processors provided in each of multiple computers working in cooperation. Furthermore, the programs for causing each of the above devices to realize the functions may be stored in a single memory provided in a single computer, or may be distributed and stored in multiple memories provided in a single computer, or may be distributed and stored in multiple memories provided in each of multiple computers.

[0091] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0092] (Appendix A1) an acquisition means for acquiring first measurement information that measures a user's motion; a feedback means for providing feedback to the user so as to reduce an error between the user's motion indicated by the first measurement information and a target motion; the acquiring means acquires second measurement information that measures the user's action to which the feedback has been given; the feedback means adjusts the timing of providing new feedback to the user based on the user's behavior indicated by the second measurement information. Information processing system.

[0093] (Appendix A2) further comprising an identification means for identifying the user; the feedback means adjusts the timing in accordance with the identified user. 1. An information processing system as described in Appendix A1.

[0094] (Appendix A3) The feedback means adjusts the intensity of the new feedback based on the user's behavior indicated by the second measurement information. An information processing system as described in Appendix A1 or A2.

[0095] (Appendix A4) the feedback means provides the feedback to the user using a stimulation device that provides electrical or vibration stimulation to the body of the user; 10. The information processing system according to any one of appendices A1 to A3.

[0096] (Appendix A5) the feedback means provides the feedback to the user using a correction device that corrects the movement by applying an external force to the body of the user. 10. An information processing system according to any one of appendices A1 to A4.

[0097] (Appendix A6) the feedback means provides the feedback to the user using a display that presents visual information to the user. 10. The information processing system according to any one of appendices A1 to A5.

[0098] (Appendix A7) When the user performs a periodic action, the feedback means adjusts the timing of the new feedback to be given to the user for the next unit operation based on the second measurement information measured for the unit operations that constitute the periodic operation. An information processing system as described in any one of Appendices A1 to A6.

[0099] (Appendix A8) When the user performs a non-periodic action, the feedback means adjusts timing of the new feedback to be given to the user during the next non-periodic operation based on the second measurement information measured during the non-periodic operation. An information processing system as described in any one of Appendices A1 to A7.

[0100] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0101] (Appendix B1) an acquisition process in which at least one processor acquires first measurement information that measures a user's behavior; a feedback process performed by the at least one processor to provide feedback to the user so that an error between the user's movement indicated by the first measurement information and a target movement is reduced; In the obtaining process, the at least one processor obtains second measurement information that measures the user's behavior for which the feedback is given; and and in the feedback processing, the at least one processor adjusts timing for providing new feedback to the user based on the user's behavior indicated by the second measurement information. Information processing methods.

[0102] (Appendix B2) The at least one processor further includes an identification process for identifying the user; In the feedback process, the at least one processor adjusts the timing according to the identified user. 1. The information processing method described in Appendix B1.

[0103] (Appendix B3) and in the feedback processing, the at least one processor adjusts the intensity of the new feedback based on the user's behavior indicated by the second measurement information. An information processing method as described in Appendix B1 or B2.

[0104] (Appendix B4) In the feedback processing, the at least one processor provides the feedback to the user using a stimulation device that provides electrical or vibration stimulation to the body of the user. 1. An information processing method according to any one of appendices B1 to B3.

[0105] (Appendix B5) In the feedback processing, the at least one processor provides the feedback to the user using a correction device that corrects the user's movement by applying an external force to the user's body. 1. An information processing method according to any one of appendices B1 to B4.

[0106] (Appendix B6) In the feedback process, the at least one processor provides the feedback to the user using a display that presents visual information to the user. 1. An information processing method according to any one of Appendices B1 to B5.

[0107] (Appendix B7) When the user performs a periodic action, and in the feedback processing, the at least one processor adjusts timing of the new feedback to be given to the user for the next unit operation based on the second measurement information measured for the unit operations that constitute the periodic operation. An information processing method as described in any one of Appendices B1 to B6.

[0108] (Appendix B8) When the user performs a non-periodic action, and in the feedback processing, the at least one processor adjusts timing of the new feedback to be provided to the user in a next non-periodic operation based on the second measurement information measured in the non-periodic operation. An information processing method as set forth in any one of Appendices B1 to B7.

[0109] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0110] (Appendix C1) A program that causes a computer to function as an information processing system, The computer an acquisition means for acquiring first measurement information that measures a user's motion; a feedback means for providing feedback to the user so as to reduce an error between the user's motion indicated by the first measurement information and a target motion; the acquiring means acquires second measurement information that measures the user's action to which the feedback has been given; the feedback means adjusts the timing of providing new feedback to the user based on the user's behavior indicated by the second measurement information. Information processing program.

[0111] (Appendix C2) The computer further functioning as an identification means for identifying the user; the feedback means adjusts the timing in accordance with the identified user. An information processing program as described in Appendix C1.

[0112] (Appendix C3) The feedback means adjusts the intensity of the new feedback based on the user's behavior indicated by the second measurement information. An information processing program as described in Appendix C1 or C2.

[0113] (Appendix C4) the feedback means provides the feedback to the user using a stimulation device that provides electrical or vibration stimulation to the body of the user; An information processing program according to any one of appendices C1 to C3.

[0114] (Appendix C5) the feedback means provides the feedback to the user using a correction device that corrects the movement by applying an external force to the body of the user. An information processing program according to any one of appendices C1 to C4.

[0115] (Appendix C6) the feedback means provides the feedback to the user using a display that presents visual information to the user. An information processing program according to any one of appendices C1 to C5.

[0116] (Appendix C7) When the user performs a periodic action, the feedback means adjusts the timing of the new feedback to be given to the user for the next unit operation based on the second measurement information measured for the unit operations that constitute the periodic operation. An information processing program as described in any one of appendices C1 to C6.

[0117] (Appendix C8) When the user performs a non-periodic action, the feedback means adjusts timing of the new feedback to be given to the user during the next non-periodic operation based on the second measurement information measured during the non-periodic operation. An information processing program as described in any one of appendices C1 to C7.

[0118] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0119] (Appendix D1) at least one processor, an acquisition process of acquiring first measurement information that measures a user's behavior; a feedback process for providing feedback to the user so as to reduce an error between the user's motion indicated by the first measurement information and a target motion; In the obtaining process, the at least one processor obtains second measurement information that measures the user's behavior for which the feedback is given; and and in the feedback processing, the at least one processor adjusts timing for providing new feedback to the user based on the user's behavior indicated by the second measurement information. Information processing system.

[0120] The information processing system may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0121] (Appendix D2) the at least one processor: further performing an identification process for identifying the user; In the feedback process, the at least one processor adjusts the timing according to the identified user. 10. The information processing system of claim 1.

[0122] (Appendix D3) and in the feedback processing, the at least one processor adjusts the intensity of the new feedback based on the user's behavior indicated by the second measurement information. An information processing system as described in Appendix D1 or D2.

[0123] (Appendix D4) In the feedback processing, the at least one processor provides the feedback to the user using a stimulation device that provides electrical or vibration stimulation to the body of the user. 10. The information processing system according to any one of appendices D1 to D3.

[0124] (Appendix D5) In the feedback processing, the at least one processor provides the feedback to the user using a correction device that corrects the user's movement by applying an external force to the user's body. 10. The information processing system according to any one of appendices D1 to D4.

[0125] (Appendix D6) In the feedback process, the at least one processor provides the feedback to the user using a display that presents visual information to the user. 10. The information processing system according to any one of appendices D1 to D5.

[0126] (Appendix D7) When the user performs a periodic action, and in the feedback processing, the at least one processor adjusts timing of the new feedback to be given to the user for the next unit operation based on the second measurement information measured for the unit operations that constitute the periodic operation. An information processing system according to any one of appendices D1 to D6.

[0127] (Appendix D8) When the user performs a non-periodic action, and in the feedback processing, the at least one processor adjusts timing of the new feedback to be provided to the user in a next non-periodic operation based on the second measurement information measured in the non-periodic operation. An information processing system according to any one of appendices D1 to D7.

[0128] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0129] (Appendix E1) A program that causes a computer to function as an information processing system, The computer, an acquisition process of acquiring first measurement information that measures a user's behavior; a feedback process for providing feedback to the user so as to reduce an error between the user's motion indicated by the first measurement information and a target motion; In the acquisition process, the computer acquires second measurement information that measures the user's behavior to which the feedback is given, In the feedback processing, the computer adjusts timing for providing new feedback to the user based on the user's behavior indicated by the second measurement information. A non-transitory recording medium on which an information processing program is recorded. [Explanation of symbols]

[0130] 1. 1A Information Processing System 10 Servers 11 Acquisition Department 12 Feedback section 13 Analysis Department 14 Database 15 Specific section 20 Terminal 1 30 Terminal 2 40 Sensor (camera) 50 Feedback Device C1 processor C2 Memory

Claims

1. an acquisition means for acquiring first measurement information that measures a user's motion; a feedback means for providing feedback to the user so as to reduce an error between the user's motion indicated by the first measurement information and a target motion; the acquiring means acquires second measurement information that measures the user's action to which the feedback has been given; the feedback means adjusts the timing of providing new feedback to the user based on the user's behavior indicated by the second measurement information. Information processing system.

2. further comprising an identification means for identifying the user; the feedback means adjusts the timing in accordance with the identified user. The information processing system according to claim 1 .

3. the feedback means adjusts the intensity of the new feedback based on the user's movement indicated by the second measurement information.

3. The information processing system according to claim 1 or 2.

4. the feedback means provides the feedback to the user using a stimulation device that provides electrical or vibration stimulation to the body of the user; 3. The information processing system according to claim 1 or 2.

5. the feedback means provides the feedback to the user using a correction device that corrects the movement by applying an external force to the body of the user.

3. The information processing system according to claim 1 or 2.

6. the feedback means provides the feedback to the user using a display that presents visual information to the user.

3. The information processing system according to claim 1 or 2.

7. When the user performs a periodic action, the feedback means adjusts timing of the new feedback to be given to the user for the next unit operation based on the second measurement information measured for the unit operations that constitute the periodic operation.

3. The information processing system according to claim 1 or 2.

8. When the user performs a non-periodic action, the feedback means adjusts timing of the new feedback to be given to the user in the next non-periodic operation based on the second measurement information measured in the non-periodic operation.

3. The information processing system according to claim 1 or 2.

9. an acquisition process in which at least one processor acquires first measurement information that measures a user's behavior; a feedback process performed by the at least one processor to provide feedback to the user so that an error between the user's movement indicated by the first measurement information and a target movement is reduced; In the obtaining process, the at least one processor obtains second measurement information that measures the user's behavior for which the feedback is given; and and in the feedback processing, the at least one processor adjusts timing for providing new feedback to the user based on the user's behavior indicated by the second measurement information. Information processing methods.

10. A program that causes a computer to function as an information processing system, The computer an acquisition means for acquiring first measurement information that measures a user's motion; a feedback unit that provides feedback to the user so as to reduce an error between the user's motion indicated by the first measurement information and a target motion; the acquiring means acquires second measurement information that measures the user's action to which the feedback has been given; the feedback means adjusts the timing of providing new feedback to the user based on the user's behavior indicated by the second measurement information. Information processing program.

Images