Information processing device, information processing system, information processing method, and information processing program

Abstract

To provide an information processing apparatus, an information processing system, an information processing method, and an information processing program for enhancing the practice efficiency of a student.SOLUTION: An information processing apparatus 10 includes an interface 13 that obtains an image of a first user, information about a posture or motion of the first user, and information about a posture or motion of a second user, and a control unit 11 that creates difference information between the posture or motion of the first user and the posture or motion of the second user. The control unit 11 displays the image of the first user or the second user on a display device, and notifies the second user about the difference information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] 2. Description of the Related Art A system is known that displays an image of a teacher and an image of a student to provide education (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-110782 A Summary of the Invention [Problem to be solved by the invention]

[0004] When a student practices using an image of a teacher as a model, the student can practice efficiently by recognizing the difference between the posture or movement of the teacher and the student's own posture or movement. There is a demand for improving the efficiency of practice.

[0005] The purpose of the present disclosure, made in consideration of such circumstances, is to improve students' practice efficiency. [Means for solving the problem]

[0006] An information processing device according to an embodiment of the present disclosure includes an interface and a control unit. The interface acquires an image of a first user, information related to a posture or movement of the first user, and information related to a posture or movement of a second user. The control unit generates difference information related to a difference between the posture or movement of the first user and the posture or movement of the second user. The control unit causes a display device to display an image of the first user or the second user, and notifies the second user of the difference information.

[0007] An information processing system according to an embodiment of the present disclosure includes the information processing devices as a first information processing device and a second information processing device. The first information processing device includes a first interface and a first control unit. The first interface acquires information related to a posture or movement of the first user and information related to a posture or movement of a second user. The first control unit generates difference information related to a difference between the posture or movement of the first user and the posture or movement of the second user, and outputs the difference information to the second information processing device. The second information processing device includes a second interface that acquires an image of the first user and the difference information, displays the image of the first user, and notifies the second user of the difference information.

[0008] An information processing method according to one embodiment of the present disclosure includes acquiring an image of a first user, information relating to a posture or movement of the first user, and information relating to a posture or movement of a second user, generating difference information relating to a difference between the posture or movement of the first user and the posture or movement of the second user, displaying an image of the first user or the second user on a display device, and notifying the second user of the difference information.

[0009] An information processing program according to one embodiment of the present disclosure causes an information processing device to perform operations including acquiring an image of a first user, information regarding the posture or movement of the first user, and information regarding the posture or movement of a second user, generating difference information regarding a difference between the posture or movement of the first user and the posture or movement of the second user, displaying an image of the first user or the second user on a display device, and notifying the second user of the difference information. Effect of the Invention

[0010] According to an information processing device, an information processing system, an information processing method, and an information processing program according to an embodiment of the present disclosure, the practice efficiency of students can be improved. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of multiple windows included in a depth image. [Figure 3A] FIG. 11 is a diagram showing an example of display of first user information. [Figure 3B] FIG. 13 is a diagram showing an example of display of second user information. [Figure 4] 1 is a flowchart illustrating an example of a procedure of an information processing method according to an embodiment. [Diagram 5] FIG. 1 is a block diagram showing an example of the configuration of an information processing system in which a plurality of second information processing devices are connected to one first information processing device. [Figure 6] 11 is a block diagram showing a configuration example of an information processing system further including a third information processing device. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (Example of configuration of information processing system 1) As shown in FIG. 1, an information processing system 1 according to an embodiment includes a first information processing device 10 and a second information processing device 20. When there is no need to distinguish between the first information processing device 10 and the second information processing device 20, they are also simply referred to as information processing devices. The information processing system 1 is capable of providing a service in which a student can learn by imitating a model movement shown by an instructor while watching the model movement. The subject of learning may include, for example, dance, calligraphy, cooking, and the like. The information processing system 1 according to this embodiment is used to provide dance lessons. The first information processing device 10 is used by an instructor who shows a model of dance. The second information processing device 20 is used by a student who learns dance while watching a model of dance. Among users of the information processing system 1, an instructor who uses the first information processing device 10 is also referred to as a first user. A student who uses the second information processing device 20 is also referred to as a second user.

[0013] The number of the first information processing device 10 and the second information processing device 20 included in the information processing system 1 is not limited to one, and may be two or more. The first information processing device 10 or the second information processing device 20 may be configured to include a PC such as a notebook PC (Personal Computer), a desktop PC, or a tablet PC. The first information processing device 10 or the second information processing device 20 may be configured to include a mobile terminal such as a smartphone or tablet carried by a user. The first information processing device 10 or the second information processing device 20 is not limited to these examples, and may be configured to include various devices.

[0014] The first information processing device 10 and the second information processing device 20 may be connected to each other via a network so as to be able to communicate with each other in a wired or wireless manner. The first information processing device 10 and the second information processing device 20 may be connected to each other so as to be able to communicate with each other by wired or wireless communication not via a network.

[0015] The first information processing device 10 includes a control unit 11, a storage unit 12, and an interface 13. The second information processing device 20 includes a control unit 21, a storage unit 22, and an interface 23. The control unit 11 is also referred to as a first control unit. The control unit 21 is also referred to as a second control unit. The interface 13 is also referred to as a first interface. The interface 23 is also referred to as a second interface.

[0016] The control unit 11 controls at least one component of the first information processing device 10. The control unit 21 controls at least one component of the second information processing device 20. The control units 11 and 21 may be configured identically or similarly. The control unit 11 or 21 may be configured to include one or more processors. In this embodiment, the "processor" is a general-purpose processor, a dedicated processor specialized for a specific process, etc., but is not limited to these. The control unit 11 or 21 may be configured to include one or more dedicated circuits. The dedicated circuit may include, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 11 or 21 may be configured to include a dedicated circuit instead of a processor, or may be configured to include a dedicated circuit together with a processor.

[0017] The storage unit 12 stores any information used in the operation of the first information processing device 10. The storage unit 22 stores any information used in the operation of the second information processing device 20. The storage units 12 and 22 may be configured identically or similarly. The storage units 12 and 22 may store a system program, an application program, or the like. The storage units 12 and 22 may be configured to include, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like, but may be configured to include various other memories without being limited thereto. The storage units 12 and 22 may function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage units 12 and 22 may be configured to include an electromagnetic storage medium such as a magnetic disk. The storage units 12 and 22 may be configured to include a non-transitory computer-readable medium. The storage unit 12 may be included in the control unit 11. The storage unit 22 may be included in the control unit 21.

[0018] In the first information processing device 10, the interface 13 outputs information or data from the control unit 11 and inputs information or data to the control unit 11. In the second information processing device 20, the interface 23 outputs information or data from the control unit 21 and inputs information or data to the control unit 21. The interface 13 may include a communication module configured to be able to communicate with the interface 23 of the second information processing device 20 via a network or without a network. The interface 23 may include a communication module configured to be able to communicate with the interface 13 of the first information processing device 10 via a network or without a network. The communication module may be compatible with a mobile communication standard such as 4G (4th Generation) or 5G (5th Generation). The communication module may be compatible with a communication standard such as LAN (Local Area Network). The communication module may be compatible with a wired or wireless communication standard. The communication module is not limited to these and may be compatible with various communication standards. The interface 13 or 23 may be configured to be connectable to an external communication module.

[0019] The network may include the Internet, at least one Wide Area Network (WAN), at least one Metropolitan Area Network (MAN), or any combination thereof. The network may include at least one wireless network, at least one optical network, or any combination thereof. The wireless network may include, for example, an ad-hoc network, a cellular network, a wireless Local Area Network (LAN), a satellite communication network, or a terrestrial microwave network.

[0020] The interface 13 or 23 may be configured to comply with standards such as USB (Universal Serial Bus) or Bluetooth (registered trademark).

[0021] The interface 13 or 23 may be configured to include an input device that accepts input of information, data, etc. from a user. The input device may be configured to include, for example, a touch panel or a touch sensor, or a pointing device such as a mouse. The input device may be configured to include physical keys. The input device may be configured to include a voice input device such as a microphone. The interface 13 or 23 may be configured to be connectable to an external input device.

[0022] The interface 13 or 23 may be configured to include an output device that outputs information or data to a user. The output device may include, for example, a display device that outputs visual information such as an image, character, or figure. The display device may include, for example, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display or an inorganic EL display, or a PDP (Plasma Display Panel). The display device is not limited to these displays and may include displays of various other types. The display device may include a light-emitting device such as an LED (Light Emitting Diode) or an LD (Laser Diode). The display device may include various other devices. The output device may include, for example, an audio output device such as a speaker that outputs auditory information such as sound. The output device may include, for example, a vibration device such as a vibrator that outputs tactile information such as vibration. The output device is not limited to these examples and may include various other devices. The interface 13 or 23 may be configured to be connectable to an external output device. The output device may be configured to be wearable by the first user or the second user, such as a wristband, etc. The output device may be configured as a tool used by the first user or the second user when practicing an action, such as a brush used in calligraphy, or a knife or cutting board used in cooking, etc.

[0023] The first information processing device 10 further includes an imaging unit 14, although this is not essential. The second information processing device 20 further includes an imaging unit 24, although this is not essential. The imaging unit 14 or 24 may include an imaging device such as a camera that captures an RGB image. The imaging unit 14 or 24 may include a distance measuring device such as a depth sensor or a stereo camera that captures a depth image. The first information processing device 10 or the second information processing device 20 may acquire image or distance measuring data (depth data) from an external imaging device or distance measuring device via the interface 13 or 23. The function of the imaging device or distance measuring device of the imaging unit 14 or 24 may be included in the function of the interface 13 or 23.

[0024] (Example of operation of information processing system 1) In the information processing system 1 according to this embodiment, the first information processing device 10 captures an example of a dance performed by the first user using the image capturing unit 14. The image capturing unit 14 may capture the posture of the first user as a still image. The image capturing unit 14 may capture the movements of the first user as a video. The first information processing device 10 outputs the image captured by the image capturing unit 14 to the second information processing device 20. The second information processing device 20 displays the image acquired from the first information processing device 10 on a display device of the interface 23 so that the second user can view and learn from the image acquired from the first information processing device 10.

[0025] <Display of an image captured by the first information processing device 10 on the second information processing device 20> The information processing system 1 may capture an RGB image and a depth image of the first user with the first information processing device 10, and display a stereoscopic image of the first user based on the RGB image and the depth image with the second information processing device 20. A specific operation for displaying a stereoscopic image will be described below.

[0026] The photographing unit 14 of the first information processing device 10 photographs an RGB image of at least a part of the body of the first user and measures the distance to at least a part of the body of the first user. As shown in FIG. 2, the photographing unit 14 measures the distance to each part included in at least a part of the body of the first user located within the measurement range MR, and generates a depth image of the first user within the measurement range MR. The depth image includes a plurality of windows W1,...,Wn set within the measurement range MR. The plurality of windows W1,...,Wn are arranged in a depth direction Z, which is a direction away from the photographing unit 14. That is, the plurality of windows W1,...,Wn are located at different depths.

[0027] The distance to each part of the first user's body is associated with the depth at which each window is located. Each window included in the depth image represents data of a part of the first user's body located at a depth corresponding to each window. The depth image represents three-dimensional data of at least a part of the first user's body as a collection of data of the parts located at a depth corresponding to each window. The number of windows (n) is an arbitrary integer equal to or greater than 2, and may be set to a value such as 30.

[0028] The control unit 11 of the first information processing device 10 identifies a window Wj corresponding to the position of at least a part of the first user's body from among a plurality of windows W1,...,Wn included in the depth image. The control unit 11 transmits a window identifier for identifying the window Wj to the second information processing device 20. Upon receiving the window identifier from the first information processing device 10, the control unit 21 of the second information processing device 20 identifies a window Wj corresponding to the received window identifier from among the plurality of windows W1,...,Wn.

[0029] The control unit 11 of the first information processing device 10 converts the measured distance for each pixel of the depth image into a relative distance within the window Wj and transmits the obtained converted value as pixel data to the second information processing device 20. The control unit 21 of the second information processing device 20 receives pixel data for each pixel of the depth image from the first information processing device 10 and restores the depth image by converting the relative distance within the window Wj indicated by the received pixel data into the measured distance. The control unit 21 combines the restored depth image with the pixel data of the RGB image and displays it on the display device of the interface 23.

[0030] The depth image may be configured as a collection of windows as described above, but is not limited to this, and may be represented as a distance map in which the distance to each part of the first user's body is assigned to each pixel. The depth image may be generated in various ways.

[0031] The RGB image of the first user and the depth image of the first user are included in information about the first user. The information about the first user is also called first user information. The first information processing device 10 outputs the first user information to the second information processing device 20 via the interface 13.

[0032] The control unit 21 of the second information processing device 20 causes a display device of the interface 23 to display an image of a model dance performed by the first user based on the first user information acquired from the first information processing device 10. The display device of the interface 23 may display the image of the model dance performed by the first user as a stereoscopic image obtained by combining an RGB image and a depth image, or may display only the RGB image. The control unit 21 may generate a stereoscopic image based on each pixel of the RGB image and the depth information corresponding to each pixel. The control unit 21 may generate a polygon of the first user based on the depth image, and generate the stereoscopic image by rendering the RGB image into the polygon.

[0033] The second information processing device 20 photographs the second user practicing a dance by looking at an image of a dance example using the photographing unit 24. The photographing unit 24 may photograph the posture of the second user as a still image. The photographing unit 24 may photograph the movement of the second user as a video. The photographing unit 24 also measures the distance to at least a part of the body of the second user and generates a depth image of the second user. The RGB image photographed of the second user and the depth image of the second user are included in the information about the second user. The information about the second user is also referred to as second user information. The second information processing device 20 outputs the second user information to the first information processing device 10 by the interface 23. The first information processing device 10 may display the image of the second user by a display device of the interface 13.

[0034] <Extraction of Difference Between First User Information and Second User Information> The information processing system 1 extracts a difference between the first user information and the second user information and notifies the first user or the second user. The extraction of the difference between the first user information and the second user information and the notification of the difference may be executed by the control unit 11 of the first information processing device 10, or may be executed by the control unit 21 of the second information processing device 20. An example of an operation in which the control unit 11 of the first information processing device 10 extracts the difference between the first user information and the second user information and notifies the difference will be described below.

[0035] The control unit 11 of the first information processing device 10 may analyze the image of the first user captured by the image capturing unit 14 and extract skeletal information of the first user. As shown in FIG. 3A, the control unit 11 may detect, for example, the position of a joint U11 corresponding to the shoulder joint of the right arm of the first user U1, the position of a joint U12 corresponding to the elbow joint, or the position of a joint U13 corresponding to the hand. The control unit 11 may detect a bone U14 connecting the joints U11 and U12, or a bone U15 connecting the joints U12 and U13. The first information processing device 10 may output the skeletal information of the first user to the second information processing device 20 via the interface 13. The skeletal information of the first user may be included in the first user information.

[0036] The control unit 11 may analyze the image of the second user acquired from the second information processing device 20 and extract skeletal information of the second user. As shown in FIG. 3B, the control unit 11 may detect, for example, the position of a joint U21 corresponding to the shoulder joint of the right arm of the second user U2, the position of a joint U22 corresponding to the elbow joint, or the position of a joint U23 corresponding to the hand. The control unit 11 may detect a bone U24 connecting the joints U21 and U22, or a bone U25 connecting the joints U22 and U23. The control unit 21 of the second information processing device 20 may extract the skeletal information of the second user and output it to the first information processing device 10.

[0037] The control unit 11 may set the skeletal information to be extracted based on the type of movement of the first user or the second user. When the first user or the second user performs a dance movement, the control unit 11 may extract skeletal information of the entire body of the first user or the second user. When the first user or the second user performs a calligraphy or cooking movement, the control unit 11 may extract skeletal information of the upper body of the first user or the second user. The control unit 11 may receive an input from the input device of the first information processing device 10, in which the first user designates a body part from which skeletal information is to be extracted.

[0038] The control unit 11 compares the skeletal information of the first user with the skeletal information of the second user, and detects a difference in the posture or movement of the second user with respect to the dance sample performed by the first user. The control unit 11 may detect, for example, a difference between bone U14 or U15 of the first user U1 in FIG. 3A and bone U24 or U25 of the second user U2 in FIG. 3B. As a value representing the difference, the control unit 11 may calculate an angle difference representing the direction of the bone, or may calculate a coordinate difference representing the position of the bone, or a distance by which the position of the bone is shifted. The control unit 11 may represent the position of the bone in three-dimensional coordinates taking into account the depth direction, or may represent the position of the bone in two-dimensional coordinates ignoring the depth direction.

[0039] The control unit 11 may detect a difference between the position of the joint U11, U12, or U13 of the first user U1 in Fig. 3A and the position of the joint U21, U22, or U23 of the second user U2 in Fig. 3B. The control unit 11 may calculate a difference in coordinates representing the positions of the joints or a distance by which the positions of the joints are shifted as a value representing the difference. The control unit 11 may represent the positions of the joints in three-dimensional coordinates taking into account the depth direction, or may represent the positions of the joints in two-dimensional coordinates ignoring the depth direction.

[0040] The control unit 11 may detect the difference in consideration of the difference between the physique of the first user and the physique of the second user. The control unit 11 may reduce the influence of the difference in physique by, for example, comparing the directions of at least some bones between the first user and the second user. The control unit 11 may reduce the influence of the difference in physique by, for example, comparing the balance of at least some parts of the corresponding bodies of the first user and the second user.

[0041] The control unit 11 may calculate the degree of deviation, which represents the magnitude of deviation between the image of the first user and the image of the second user as a numerical value, as a difference between the posture or movement of the first user and the posture or movement of the second user. The greater the difference, the greater the value of the degree of deviation is calculated. The control unit 11 may calculate the degree of deviation for at least a part of the body of the second user. The control unit 11 may detect the difference by calculating the degree of deviation of each of the images of the first user and the second user. For example, the control unit 11 may analyze the image of the tip of the left hand of the first user U1 in FIG. 3A as the attention point U16 of the posture of the first user. The control unit 11 may also analyze the image of the tip of the left hand of the second user U2 in FIG. 3B as the attention point U26 of the posture of the second user. The control unit 11 may calculate the degree of deviation of the images as the difference in posture between the first user and the second user based on the analysis result of the images corresponding to the attention points U16 and U26.

[0042] When the images of the first user and the second user are videos, the control unit 11 may extract skeletal information of the first user and the second user in at least some frames of the video to detect the difference. The control unit 11 may analyze attention points of the postures of the first user and the second user in at least some frames of the video to detect the difference.

[0043] The control unit 11 may detect the difference by synchronizing and comparing frames of a video of the first user and a video of the second user. For example, the control unit 11 may detect the difference between the posture of the first user and the posture of the second user captured in frames corresponding to the same time as the difference in the movement of the second user relative to the movement of the first user. Also, for example, the control unit 11 may compare the posture of the first user captured in a frame corresponding to a certain time with the posture of the second user captured in a frame corresponding to a different time. The control unit 11 may detect that the timing of the movement of the second user is shifted from the movement of the first user when the posture of the first user captured in a frame corresponding to a certain time matches the posture of the second user captured in a frame corresponding to a different time.

[0044] The control unit 11 may detect the difference between the movement speed of the first user and the movement speed of the second user by comparing each of a plurality of frames of the video of the first user with each of a plurality of frames of the video of the second user.

[0045] The control unit 11 may receive an input from the first user specifying which body parts of the first user and the second user are to be compared. The body parts to be compared are also referred to as comparison target parts. The control unit 11 may display an image of the first user or the second user on a display device of the first information processing device 10, and receive an input from the first user specifying the comparison target parts via an input device of the first information processing device 10. The control unit 11 may display skeletal information of the first user or the second user on a display device of the first information processing device 10, and receive an input from the first user specifying a bone or joint as a comparison target part via an input device of the first information processing device 10.

[0046] The control unit 11 may receive an input from the first user specifying the balance of posture or movement to be compared between the first user and the second user.

[0047] The control unit 11 may receive an input from the first user specifying which of a plurality of frames included in the videos of the first user and the second user is to be compared. The frame to be compared is also referred to as a comparison frame. The control unit 11 may display the video of the first user or the second user frame by frame on a display device of the first information processing device 10, and receive an input from the first user specifying the comparison frame via an input device of the first information processing device 10.

[0048] The control unit 11 may receive an input from the first user specifying which periods of the videos of the first user and the second user are to be compared. The period to be compared is also referred to as a comparison period. The control unit 11 may display the video of the first user or the second user on a display device of the first information processing device 10, and receive an input from the first user specifying the start and end of the comparison period via an input device of the first information processing device 10. The comparison period may include only one frame of the video. In other words, the comparison period may include a comparison frame.

[0049] The control unit 11 may automatically determine the comparison target part or the start and end of the comparison target period based on the image of the first user or the second user.

[0050] As described above, the control unit 11 may detect a difference in the posture or movement of the second user relative to the posture or movement of the first user based on the images of the first user and the second user. The difference may be calculated for at least a part of the body of the second user. The difference may be calculated as a positional deviation amount of a bone or joint. The difference may be calculated as a directional deviation amount of a bone. The difference may be calculated as a time difference in the timing of a movement. The difference may be calculated as a difference in the speed of a movement.

[0051] <Notification of difference information> The control unit 11 of the first information processing device 10 detects a difference between the posture or movement of the second user and the posture or movement of the first user, and outputs information about the detected difference to the second information processing device 20. Information about the difference is also referred to as difference information.

[0052] The control unit 21 of the second information processing device 20 notifies the second user of the difference information while displaying the image of the second user on the display device of the interface 23. For example, the control unit 21 may generate an image in which information indicating that there is a large difference from the posture or movement of the first user is superimposed on the image of the second user, and display the image on the display device. The control unit 21 may superimpose a marker image that emphasizes the part with a large difference on the image of the second user.

[0053] The control unit 21 may notify the second user that the difference between the posture or movement of the second user and the posture or movement of the first user has increased, via an output device of the interface 23, while displaying the image of the second user on the display device. That is, the control unit 21 may synchronize the display of the image of the second user on the display device with the notification of the difference information to the second user. As the notification by the output device, the control unit 21 may output, for example, a sound or a vibration to be given to the second user, at the timing when the difference has increased. The control unit 21 may notify the second user of the difference information, while displaying the image of the first user on the display device.

[0054] The control unit 21 may determine that the difference has become large when the value representing the difference is equal to or larger than a comparison threshold. The comparison threshold may be determined as appropriate. The comparison threshold may be set based on a designation by the first user. The comparison threshold may be set based on a posture or movement of the second user that is indicated by the first user as causing a large difference.

[0055] By checking the notified difference information, the second user can recognize that the difference between the posture or movement of the first user, who is the dance model, and the second user's own posture or movement has increased. By recognizing the posture or movement that has increased in difference between the dance model and the second user's own posture or movement, the second user can practice to bring his or her own posture or movement closer to the dance model. In other words, according to the information processing system 1 according to this embodiment, the practice efficiency of the student, who is the second user, can be improved.

[0056] The difference information may be notified to the first user. Specifically, the control unit 11 of the first information processing device 10 may notify the first user of the difference information while displaying an image of the first user on the display device. The control unit 11 may notify the first user of the difference information while displaying an image of the second user on the display device.

[0057] <Example of information processing procedure> As described above, in the information processing system 1 according to this embodiment, the first information processing device 10 and the second information processing device 20 notify the second user, who practices by looking at the example of the first user, of difference information on the posture or movement of the second user. The control unit 11 of the first information processing device 10 and the control unit 21 of the second information processing device 20 may execute an information processing method including the procedure of the flowchart exemplified in FIG. 4, for example. The information processing method may be realized as an information processing program executed by the control units 11 and 21. The information processing program may be stored in a non-transitory computer-readable medium.

[0058] The control unit 11 of the first information processing device 10 starts acquiring first user information (step S11). The control unit 21 of the second information processing device 20 starts acquiring second user information (step S21). The control unit 11 and the control unit 21 may synchronize the start of information acquisition. Specifically, when starting acquisition of the first user information, the control unit 11 may notify the control unit 21 of the start of information acquisition. The control unit 21 may start acquiring the second user information in response to the notification from the control unit 11.

[0059] The control unit 11 analyzes the acquired first user information and stores it in the storage unit 12 (step S12). The control unit 21 analyzes the acquired second user information and stores it in the storage unit 22 (step S22).

[0060] The control unit 11 ends the acquisition of the first user information (step S13). The control unit 21 ends the acquisition of the second user information (step S23). The control unit 11 and the control unit 21 may synchronize the end of the information acquisition. Specifically, when the control unit 11 ends the acquisition of the first user information, the control unit 11 may notify the control unit 21 of the end of the information acquisition. The control unit 21 may end the acquisition of the second user information in response to the notification from the control unit 11.

[0061] The control unit 21 outputs the second user information to the first information processing device 10 (step S24). The control unit 11 acquires the second user information from the second information processing device 20 (step S14). The control unit 11 generates difference information based on a comparison between the first user information and the second user information, and outputs the generated difference information to the second information processing device 20 (step S15). The control unit 21 acquires the difference information from the first information processing device 10 (step S25).

[0062] Control unit 21 notifies the second user of the difference information while displaying the image of the second user (step S26). After executing the procedure of step S26, control unit 21 ends the execution of the procedure of the flowchart in Fig. 4. After executing the procedure of step S15, control unit 11 ends the execution of the procedure of the flowchart in Fig. 4. After executing the procedure of step S15, control unit 11 may notify the first user of the difference information while displaying the image of the first user.

[0063] <Summary> As described above, according to the information processing system 1, the first information processing device 10, and the second information processing device 20 of the present embodiment, the second user can easily recognize the difference between the model posture or movement shown by the first user and the second user's own posture or movement. By recognizing the difference from the model, the second user can practice to reduce the difference. As a result, the practice efficiency of the second user can be improved.

[0064] (Other embodiments) An example of the configuration of an information processing system 1 according to another embodiment will be described below.

[0065] <Configuration with multiple second information processing devices 20> As shown in FIG. 5, the information processing system 1 may include one first information processing device 10 and a plurality of second information processing devices 20. Each of the second information processing devices 20 is used by a different second user, and each of the second information processing devices 20 acquires different second user information. The first information processing device 10 is communicatively connected to each of the second information processing devices 20. The first information processing device 10 and each of the second information processing devices 20 may be communicatively connected to each other via a network in a wired or wireless manner. The first information processing device 10 and each of the second information processing devices 20 may be communicatively connected to each other by wired or wireless communication not via a network.

[0066] The control unit 11 of the first information processing device 10 acquires the first user information from the photographing unit 14, and acquires, from each second information processing device 20, different second user information of a second user who uses each second information processing device 20. The control unit 11 generates difference information between the first user information and each piece of second user information. That is, the control unit 11 generates difference information for each different second user.

[0067] The control unit 11 may classify the second users into a plurality of groups based on the difference information generated for each second user. For example, the control unit 11 may classify the second users in which the difference between the posture or movement of the second user and the specific posture or movement of the first user is the same or similar as a group in which the second users make mistakes in the specific posture or movement. The control unit 11 may classify the second users in which the difference between the posture or movement of the first user and the specific posture or movement is greater than a predetermined value as a group in which the second users make mistakes in the specific posture or movement. The predetermined value may be set as a threshold value of the difference between the position or orientation of the joint or bone included in the skeletal information. The predetermined value may be set as a threshold value of the degree of deviation between the image of the first user and the image of the second user.

[0068] The control unit 11 may classify second users whose posture or movement difference at a specific timing or period is the same or similar as a group of users who make incorrect posture or movement at a specific timing or period. The control unit 11 may classify second users whose posture or movement difference at a specific timing or period is greater than a predetermined value as a group of users who make incorrect posture or movement at a specific timing or period.

[0069] The specific timing or the start and end of the specific period may be specified by the first user. The control unit 11 may set the specific timing or the start and end of the specific period based on, for example, information input by the first user through an input device of the interface 13. The control unit 11 may set, for example, the timing when the first user makes a signaling action as the start and end of the specific timing or the specific period. The signaling action may include, for example, a specific gesture of the first user, or a specific facial expression of the first user, such as closing one eye.

[0070] The specific timing or the start and end of a specific period may be set to coincide with, for example, a dividing point in a piece of music used in a dance movement or the start and end of a piece of music. The specific timing or the start and end of a specific period may be set to coincide with, for example, an important point in a character written in calligraphy or the start and end of each stroke. The specific timing or the start and end of a specific period may be set to coincide with, for example, an important point in each task or a dividing point in a task included in a cooking process.

[0071] The first information processing device 10 may communicate with a plurality of second information processing devices 20 used by a plurality of second users classified in the same group. The second users classified in the same group have the same or similar posture or movement differences. In other words, the second users classified in the same group make the same or similar mistakes. Therefore, the first user using the first information processing device 10 can communicate with the second information processing device 20 used by the second users classified in the same group to provide focused instruction on the incorrect posture or movement. As a result, the practice efficiency of the second users can be improved.

[0072] A plurality of second information processing devices 20 used by a plurality of second users classified into the same group may communicate with each other. The second users classified into the same group may point out mistakes in posture or movement to each other. As a result, the practice efficiency of the second users may be improved.

[0073] The control unit 11 may classify the second users into a group that makes mistakes in a specific posture or movement and a group that does not make mistakes in a specific posture or movement. The second information processing device 20 used by the second users classified into the group that makes mistakes in a specific posture or movement may communicate with the second information processing device 20 used by the second users classified into the group that does not make mistakes in a specific posture or movement. In this case, the second users that do not make mistakes in a specific posture or movement can instruct the second users that make mistakes in a specific posture or movement. By allowing the second users to instruct each other, the burden on the first user can be reduced. In addition, the second users can practice without waiting for the first users. As a result, the practice efficiency of the second users can be improved.

[0074] <Configuration example including third information processing device 30> As shown in FIG. 6, the information processing system 1 may further include a third information processing device 30. The third information processing device 30 may function as a host server. In this case, the first information processing device 10 may function as a client server (terminal device) used by the first user. The second information processing device 20 may function as a client server (terminal device) used by the second user. As a host server, the third information processing device 30 may acquire first user information and second user information from the client server to generate difference information, and output the difference information to each of the client servers of the first user and the second user.

[0075] The information processing system 1 may cause one of the first information processing device 10 or the second information processing device 20 to function as a host server, and the other to function as a client server.

[0076] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each means or step can be rearranged so as not to be logically inconsistent, and multiple means or steps can be combined or divided into one.

[0077] Some of the embodiments of the present disclosure will be described below. However, it should be noted that the embodiments of the present disclosure are not limited to these. [Appendix 1] an interface for acquiring an image of a first user, information relating to a posture or movement of the first user, and information relating to a posture or movement of a second user; a control unit that generates difference information regarding a difference between a posture or movement of the first user and a posture or movement of the second user; Equipped with The control unit is an information processing device that causes a display device to display an image of the first user or the second user, and notifies the second user of the difference information. [Appendix 2] The interface acquires skeletal information including bones or joints of the first user and the second user as information regarding the posture or movement of the first user and the second user, The information processing device according to [Supplementary Note 1], wherein the control unit generates the difference information based on a comparison between skeletal information of the first user and skeletal information of the second user. [Appendix 3] The interface acquires information for designating a portion of the skeletal information to be compared; The information processing device according to [Appendix 2], wherein the control unit generates the difference information based on a portion of the skeletal information that is to be compared. [Appendix 4] The information processing device described in [Appendix 3], wherein the interface obtains, as information specifying a part of the skeletal information to be compared, information specifying a bone or joint contained in the skeletal information, or information specifying at least a part of the first user's body in an image of the first user. [Appendix 5] The interface includes: acquiring a video including a plurality of frames as an image of the first user; acquiring, as the skeletal information, information that associates movements of bones or joints included in the skeletal information with the moving image; acquiring information for designating a frame of the video as information for designating a portion of the skeletal information to be compared; The information processing device according to [Appendix 3] or [Appendix 4], wherein the control unit generates the difference information based on a comparison of skeletal information associated with a specified frame. [Appendix 6] The information processing device according to any one of [Appendix 2] to [Appendix 5], wherein the control unit generates the difference information based on a comparison of orientations of bones or joints included in the skeletal information. [Appendix 7] The control unit notifies the second user of the difference information by displaying an image on the display device in which the difference information is superimposed on the image of the second user. [Appendix 8] The information processing device according to [Appendix 7], wherein the control unit synchronizes the display of the image of the second user on the display device with the notification of the difference information to the second user. [Appendix 9] The interface acquires information regarding a posture or a movement of each of the second users; The information processing device described in any one of [Appendix 1] to [Appendix 8], wherein the control unit generates the difference information for each of the multiple second users and classifies the multiple second users into groups based on the difference information for each of the multiple second users. [Appendix 10] The information processing device described in [Appendix 9], wherein the control unit classifies, among the multiple second users, users for whom a value representing a difference in posture or movement of a comparison target part is equal to or greater than a comparison threshold into the same group. [Appendix 11] The information processing device described in [Appendix 9] or [Appendix 10], wherein the control unit classifies, among the multiple second users, users whose values ​​representing the difference in posture or movement within a comparison period are equal to or greater than a comparison threshold into the same group. [Appendix 12] The information processing device according to [Appendix 11], wherein the interface acquires information specifying the comparison period. [Appendix 13] The information processing device according to [Appendix 11], wherein the control unit sets the start or end of the comparison period based on an image of the first user. [Appendix 14] The information processing device described in any one of [Appendix 10] to [Appendix 13], wherein the control unit causes images of users classified into the same group among the multiple second users to be displayed on each user's display device. [Appendix 15] The information processing device described in any one of [Appendix 10] to [Appendix 14], wherein the control unit causes images of users classified into the same group among the multiple second users to be displayed on the display device of the first user. [Appendix 16] The information processing device according to any one of [Supplementary Note 1] to [Supplementary Note 15] is provided as a first information processing device and a second information processing device, The first information processing device is a first interface that acquires information regarding a posture or movement of the first user and information regarding a posture or movement of a second user; a first control unit that generates difference information regarding a difference between a posture or a movement of the first user and a posture or a movement of the second user and outputs the difference information to the second information processing device; Equipped with the second information processing device includes a second interface that acquires an image of the first user and the difference information, displays the image of the first user, and notifies the second user of the difference information; Information processing system. [Appendix 17] acquiring an image of a first user, information relating to a posture or movement of the first user, and information relating to a posture or movement of a second user; generating difference information relating to a difference between a posture or movement of the first user and a posture or movement of the second user; causing a display device to display an image of the first user or the second user; notifying the second user of the difference information; An information processing method comprising: [Appendix 18] In the information processing device, acquiring an image of a first user, information relating to a posture or movement of the first user, and information relating to a posture or movement of a second user; generating difference information relating to a difference between a posture or movement of the first user and a posture or movement of the second user; causing a display device to display an image of the first user or the second user; notifying the second user of the difference information; An information processing program that causes an operation including the steps of: [Explanation of symbols]

[0078] 1. Information Processing Systems 10 first information processing device (11: control unit, 12: memory unit, 13: interface, 14: imaging unit) 20 second information processing device (21: control unit, 22: storage unit, 23: interface, 24: imaging unit) 30 third information processing device (31: control unit, 32: storage unit, 33: interface) 40 Network U1, U2 1st user, 2nd user U11, U12, U13, U21, U22, 23 joints U14, U15, U24, U25 Bourne U16 and U26 highlights MR Measurement Range W1, W2, Wn Windows

Claims

1. an interface that acquires an image of a first user and information regarding a posture or movement of the first user, and an image of each of a plurality of second users and information regarding a posture or movement of each of the plurality of second users; a control unit that generates difference information regarding a difference between a posture or movement of the first user and each of the postures or movements of the plurality of second users at a specific timing or a specific period, and classifies the plurality of second users into groups based on the difference information of each of the plurality of second users at the specific timing or the specific period; Equipped with The control unit displays an image of the first user or each of the plurality of second users on a display device, and notifies each of the plurality of second users of the difference information.

2. The interface accepts input of the specific timing or a start and end of the specific period from the first user; The information processing device according to claim 1 , wherein the control unit sets the specific timing or a start and end of the specific period in response to an input from the first user.

3. The interface acquires skeletal information including bones or joints of each of the first user and the multiple second users as information regarding a posture or movement of each of the first user and the multiple second users; The information processing device according to claim 1 , wherein the control unit generates the difference information based on a comparison between skeletal information of the first user and skeletal information of each of the plurality of second users.

4. The interface acquires information for designating a portion of the skeletal information to be compared; The information processing device according to claim 3 , wherein the control unit generates the difference information based on a portion of the skeletal information to be compared.

5. 5. The information processing device according to claim 4, wherein the interface acquires, as information specifying a part of the skeletal information to be compared, information specifying a bone or joint contained in the skeletal information, or information specifying at least a part of the body of the first user in an image of the first user.

6. The interface includes: acquiring a video including a plurality of frames as an image of the first user; acquiring, as the skeletal information, information that associates movements of bones or joints included in the skeletal information with the moving image; acquiring information for designating a frame of the video as information for designating a portion of the skeletal information to be compared; The information processing device according to claim 4 , wherein the control unit generates the difference information based on a comparison of skeletal information associated with a specified frame.

7. The information processing device according to claim 3 , wherein the control unit generates the difference information based on a comparison of orientations of bones or joints included in the skeletal information.

8. The information processing device according to claim 1 , wherein the control unit notifies each of the second users of the difference information by causing the display device to display an image in which the difference information is superimposed on each image of the second users.

9. The information processing device according to claim 8 , wherein the control unit synchronizes display of the images of the second users on the display device with notification of the difference information to the second users.

10. The information processing device according to claim 1 , wherein the control unit classifies, among the plurality of second users, users for whom a value representing a difference in posture or movement of a comparison target body part is equal to or greater than a comparison threshold into the same group.

11. The information processing device according to claim 1 , wherein the control unit classifies, among the plurality of second users, users for whom a value representing a difference in posture or movement within a comparison period is equal to or greater than a comparison threshold, into the same group.

12. The information processing apparatus according to claim 11 , wherein the interface acquires information that designates the comparison period.

13. The information processing device according to claim 11 , wherein the control unit sets a start or an end of the comparison period based on an image of the first user.

14. The information processing device according to claim 10 , wherein the control unit causes images of users classified into the same group among the plurality of second users to be displayed on display devices of the respective users.

15. The information processing device according to claim 10 , wherein the control unit causes an image of a user who is classified into the same group among the plurality of second users to be displayed on a display device of the first user.

16. The information processing device according to any one of claims 1 to 15 is provided as a first information processing device and a plurality of second information processing devices, The first information processing device is a first interface that acquires information regarding a posture or movement of the first user, an image of each of the second users, and information regarding the posture or movement of the second users; a first control unit that generates difference information regarding a difference between a posture or a movement of the first user and each of the postures or movements of the second users at a specific timing or during a specific period, and outputs the difference information to each of the second information processing devices; Equipped with Each of the second information processing devices includes a second interface that acquires an image of the first user and the difference information, displays the image of the first user, and notifies each of the second users of the difference information. Information processing system.

17. acquiring an image of a first user and information relating to a posture or movement of the first user, and an image of each of a plurality of second users and information relating to a posture or movement of each of the plurality of second users; generating difference information regarding a difference between a posture or movement of the first user and a posture or movement of each of the plurality of second users at a specific timing or a specific period of time, and classifying the plurality of second users into groups based on the difference information of each of the plurality of second users at the specific timing or the specific period of time; causing a display device to display an image of the first user or each of the plurality of second users; notifying each of the plurality of second users of the difference information; An information processing method comprising:

18. In the information processing device, acquiring an image of a first user and information relating to a posture or movement of the first user, and an image of each of a plurality of second users and information relating to a posture or movement of each of the plurality of second users; generating difference information regarding a difference between a posture or movement of the first user and a posture or movement of each of the plurality of second users at a specific timing or a specific period of time, and classifying the plurality of second users into groups based on the difference information of each of the plurality of second users at the specific timing or the specific period of time; causing a display device to display an image of the first user or each of the plurality of second users; notifying each of the plurality of second users of the difference information; An information processing program that causes an operation including the steps of:

Images