Information processing apparatus, information processing method, and program

Abstract

The present invention relates to an information processing device, information processing method, and program, which make it possible to provide more effective learning content when studying body movements. An adjustment section adjusts a second virtual object, which reflects the body movements of a second person, to be superimposed on a first virtual object, which reflects the body movements of a first person, on the basis of feature point information on the first person contained in the first virtual object to generate a post-adjustment second virtual object. The present technology may be applied to, for example, an information processing device such as a smartphone.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program, and more particularly to an information processing apparatus, an information processing method, and a program that can provide more effective learning content for learning body movements.

Background Art

[0002] Conventionally, there is a technique in which a user can easily learn the movements of an instructor by arranging and displaying a video of the instructor performing exercises such as aerobics, yoga, or dance and a video of the user performing exercises side by side.

[0003] In recent years, athletes have been performing exercises while wearing a device capable of receiving various types of information from the outside via a network. For example, Patent Document 1 discloses a technique in which, at the location where one athlete is performing exercises, virtual objects of other athletes who have performed exercises at that location in the past are superimposed and displayed on a display unit that shows the surroundings.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, if the video of the instructor performing exercises and the video of the user performing exercises can be superimposed and displayed, the user should be able to learn the movements of the instructor more accurately.

[0006] The present disclosure has been made in view of such a situation, and aims to provide more effective learning content for learning body movements.

Means for Solving the Problem

[0007] The information processing apparatus of the present disclosure includes an adjustment unit that generates an adjusted second virtual object by adjusting a second virtual object that reflects the body movement of a second person and is superimposed on the first virtual object based on the feature point information of the first person included in the first virtual object that reflects the body movement of the first person.

[0008] The information processing method of the present disclosure is an information processing method in which an information processing apparatus generates an adjusted second virtual object by adjusting a second virtual object that reflects the body movement of a second person and is superimposed on the first virtual object based on the feature point information of the first person included in the first virtual object that reflects the body movement of the first person.

[0009] The program of the present disclosure is a program for causing a computer to execute a process of generating an adjusted second virtual object by adjusting a second virtual object that reflects the body movement of a second person and is superimposed on the first virtual object based on the feature point information of the first person included in the first virtual object that reflects the body movement of the first person.

[0010] In the present disclosure, an adjusted second virtual object is generated by adjusting a second virtual object that reflects the body movement of a second person and is superimposed on the first virtual object based on the feature point information of the first person included in the first virtual object that reflects the body movement of the first person.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order.

[0013] 1. Outline and Use Case of the Technology According to the Present Disclosure 2. Configuration and Operation of the Information Processing System 3. Application Examples of 5G Network Slicing 4. Modification Examples 5. Configuration Example of a Computer

[0014] <1. Outline and Use Case of the Technology According to the Present Disclosure> (Outline of the Information Processing System) FIG. 1 is a diagram showing an example of the outline of an information processing system to which the technology according to the present disclosure is applied.

[0015] In the information processing system of FIG. 1, a reference digital twin, which is a virtual object reflecting the body movements of an instructor TE, who is a reference person such as in aerobics, yoga, or dance in studio SU, is superimposed on a user digital twin, which is a virtual object reflecting the body movements of a user ST who is at home HO, and is displayed on a device DE within home HO.

[0016] Generally, a digital twin refers to something in which information representing an object or environment in the real space and their states is constructed and expressed in real time on a virtual space, or the technology therefor. The digital twin in the present embodiment shall refer to a virtual object in which the skeleton, body shape, and movements of a person in the real space are reflected in real time on the virtual space. Specifically, the digital twin is computer graphics (3DCG) in which three-dimensional information of a person displayed on the virtual space is modeled in three dimensions. The digital twin is generated based on sensor data obtained by sensing the instructor TE or the user ST by one or more sensors installed in the studio SU or within the home HO. The digital twin may be drawn while maintaining the skeleton, body shape, and scale of the corresponding person as they are, or may be drawn with the skeleton, body shape, and scale adjusted for the purpose of protecting the privacy of the person.

[0017] Hereinafter, the reference digital twin of the instructor TE will be appropriately referred to as the teacher digital twin, and the user digital twin of the user ST will be referred to as the student digital twin.

[0018] The user ST can learn the movements of the instructor TE more accurately by moving their own body while watching the movements of the teacher digital twin superimposed on their own student digital twin.

[0019] Also, the instructor TE can give instructions to the user ST regarding the movements of the user ST by watching the movements of the student digital twin superimposed on their own teacher digital twin in the studio SU.

[0020] The studio SU and the home HO may directly exchange (transmit and receive) information through wired communication or wireless communication, or may do so via the MEC (Mobile Edge Computing) server 10 or the cloud server 20. When the information transmission and reception is performed by wireless communication, communication methods such as LTE (Long Term Evolution), Wi-Fi (registered trademark), 4G, 5G, etc. can be applied to part or all of the wireless communication.

[0021] (Example of superimposed video) With reference to FIGS. 2 and 3, an example of a superimposed video in which a teacher digital twin is superimposed on a student digital twin and is displayed on the device DE in the home HO will be described.

[0022] In the state of screen #1 in FIG. 2, a grid-like teacher digital twin 30TE, which is also standing upright, is superimposed on the standing student digital twin 30ST. In the figure, a button 41, which is a GUI (Graphical User Interface) for starting a lesson by the instructor TE in the studio SU, is displayed in the upper right of screen #1.

[0023] As shown in the state of screen #2, when it is determined that the hand of the corresponding student digital twin 30ST overlaps the area of the button 41 by the user ST raising one hand, the lesson by the instructor TE is started. Here, determination processing is performed based on the positional relationship between the coordinates of the button 41 in the virtual space and the coordinates of the hand of the student digital twin 30ST. Therefore, as in screen #2, even if the hand of the student digital twin 30ST overlaps the area of the button 41 in a front view, if the hand of the student digital twin 30ST is displaced from the area of the button 41 in the depth direction, the lesson is not started.

[0024] In the state of the screen #3, the instructor TE bends one knee from the upright state and stands on one leg, and the teacher digital twin 30TE corresponding to the bent knee and standing on one leg is displayed. Further, on the teacher digital twin 30TE and the student digital twin 30ST, attention points (fitting points) indicating the body parts to be moved in that movement are superimposed and displayed. Specifically, the fitting points indicating the positions of the waist, knee, and heel in the state of standing on one leg of the teacher digital twin 30TE and the fitting points indicating the positions of the waist, knee, and heel in the upright state (before standing on one leg) of the student digital twin 30ST are displayed.

[0025] By displaying the fitting points of the teacher digital twin 30TE and the student digital twin 30ST in this way, the movement of the user ST can be guided. In addition to the fitting points, lines and figures for assisting and guiding the movement of the user ST may be superimposed and displayed on the teacher digital twin 30TE and the student digital twin 30ST.

[0026] In the state of the screen #4, the user ST stands on one leg in accordance with the movement of the teacher digital twin 30TE, and the fitting points of the student digital twin 30ST are made to coincide with the fitting points of the teacher digital twin 30TE. At this time, an effect video 43 for prompting to maintain the posture is displayed around (the background of) the student digital twin 30ST. Further, an indicator 45 representing the time for which the user ST (student digital twin 30ST) maintains the posture is displayed in the upper left of the screen #4.

[0027] In screens #3 and #4, a superimposed video (student digital twin 30ST and teacher digital twin 30TE) in a front view is displayed, but a superimposed video from different viewpoints (angles) can also be displayed. Thereby, the user ST can more precisely confirm the deviation from the movement of the instructor TE.

[0028] Furthermore, for parts (regions) where there are differences in movement between the student digital twin 30ST and the teacher digital twin 30TE, an effect video that highlights that part may be superimposed. Conversely, for parts (regions) where the movements of the student digital twin 30ST and the teacher digital twin 30TE match, an effect video that highlights that part may also be superimposed.

[0029] The movements shown in screens #3 and #4 are repeated, and when the lesson ends, a pop-up 47 showing the result of the lesson is displayed as shown in the state of screen #5. The pop-up 47 shows the matching rate of the movements as the evaluation result of the student digital twin 30ST's movements with respect to the teacher digital twin 30TE. As the evaluation result of the movements, not limited to the matching rate, the degree of achievement according to the level of the movements, etc. may be scored and shown.

[0030] In this way, the user ST can learn the movements of the instructor TE while watching the superimposed video and also grasp the achievement of their own movements.

[0031] (Applicable use cases) Here, with reference to FIG. 4, applicable use cases for the digital twin as described above will be explained. Five use cases UC1 to UC5 are shown in FIG. 4.

[0032] In use case UC1, as the teacher digital twin, a digital twin in which the teacher's body movements are reflected in real time is applied. Also, as the student digital twin, a digital twin in which the student's body movements are reflected in real time is applied.

[0033] Use case UC1 can be applied to, for example, a case where an instructor who becomes a teacher conducts classes such as aerobics, yoga, and dance in real time for a student who is a user at home, for example, from a studio (real-time studio class). In this use case, the teacher can conduct real-time classes not only from the studio but also from home or any other arbitrary space, and the same shall apply to the subsequent use cases. Use case UC1 can be realized with a system configuration consisting of devices on the teacher side and the student side, and the MEC server 10.

[0034] In use case UC2, as the teacher digital twin, a digital twin in which the teacher's body movements are reflected in real time, or a digital twin in which the teacher's body movements reflected in pre-shot video content (recorded content) are applied. Also, as the student digital twin, a digital twin in which the student's body movements are reflected in real time is applied.

[0035] Use case UC2 can be applied to real-time classes such as aerobics, yoga, and dance, similar to use case UC1. However, in the real-time class of use case UC2, the teacher can proceed while switching between the case where the teacher performs in real time and the case where the teacher shows video content (a digital twin based on the video content is presented). Also, use case UC2 can be applied to, for example, a soccer class where a professional soccer player lectures junior players on shooting (kicking) and dribbling techniques. Use case UC2 can be realized with a system configuration consisting of devices on the teacher side and the student side, the MEC server 10, and a cloud server 20 capable of processing video content.

[0036] In Use Case UC3, as the teacher digital twin, a digital twin that reflects the teacher's body movements in real time is applied. Also, as the student digital twin, a digital twin that reflects the body movements of the students shown in pre-recorded video content is applied.

[0037] Use Case UC3 can also be applied to real-time classes such as aerobics, yoga, and dance, similar to Use Case UC1. However, in the real-time class of Use Case UC3, the teacher can add instruction information such as instructions and advice for the student video content in real time by checking the movements of the student digital twin based on the student video content. Use Case UC3 can also be applied to, for example, a soccer class where a professional soccer player lectures junior players on shooting and dribbling. Use Case UC3 can be realized with a system configuration consisting of the devices on the teacher side and the student side, MEC Server 10, and Cloud Server 20 that can process video content.

[0038] In Use Case UC4, as the teacher digital twin, a digital twin that reflects the teacher's body movements in real time, or a digital twin that reflects the teacher's body movements shown in pre-recorded video content is applied. Also, as the student digital twin, a digital twin that reflects the body movements of the students shown in pre-recorded video content is applied.

[0039] Use case UC4 can also be applied to real-time classes such as aerobics, yoga, and dance, similar to use case UC1. However, in the real-time classes of use case UC3, it is possible to switch between the case where the teacher adds instruction information such as instructions and advice on the student's video content in real time and the case of showing the video content while progressing. Use case UC4 can also be applied to, for example, a soccer training session where a professional soccer player lectures junior players on shooting and dribbling skills. Use case UC4 can be realized with a system configuration consisting of devices on the teacher's side and the student's side, the MEC server 10, and the cloud server 20 capable of processing video content.

[0040] In use case UC5, as the teacher digital twin and the student digital twin, digital twins that reflect the physical movements of the students shown in pre-shot video content are applied to both.

[0041] Use case UC5 can be applied to, for example, golf self-conditioning (checking movements such as swings performed by oneself). Specifically, the student can check their own movements by overlaying a digital twin based on the current video content on top of a digital twin based on past video content of themselves (using the digital twin based on the past video content as the teacher digital twin). Use case UC5 can also be applied to, for example, self-conditioning of shooting and dribbling skills by professional soccer players. Use case UC5 can be realized with a system configuration consisting of the student's device, the MEC server 10, and the cloud server 20 capable of processing video content.

[0042] <2. Configuration and Operation of the Information Processing System> In the following, the specific configuration and operation of the information processing system to which the technology according to the present disclosure is applied will be described.

[0043] (Configuration Example of the Information Processing System) FIG. 5 is a block diagram showing a configuration example of an information processing system to which the technology according to the present disclosure is applied.

[0044] The information processing system in FIG. 5 is composed of an instructor-side device 100 and a student-side device 200. In the example of FIG. 5, it is assumed that the instructor-side device 100 and the student-side device 200 communicate directly with each other, but they can also communicate via the MEC server 10 or the cloud server 20.

[0045] The instructor-side device 100 is installed in a space such as a studio or a home where an instructor is present.

[0046] On the other hand, the student-side device 200 is installed in a space such as a studio or a home where a student (user) is present.

[0047] When the instructor-side device 100 and the student-side device 200 are installed in a large space such as a studio, they are configured as relatively large-scale devices (or systems), such as a booth-type housing that surrounds a person or a device with a full-body mirror-type display surface that shows the entire body of a person. On the other hand, when the instructor-side device 100 and the student-side device 200 are installed in a narrow space such as a home, they are configured as small-scale devices (or systems), such as a smartphone equipped with various sensors or a display connectable to the smartphone. Note that the instructor-side device 100 and the student-side device 200 may be configured as devices (or systems) of the same scale.

[0048] The instructor-side device 100 includes a display unit 110, an operation unit 120, a storage unit 130, a communication unit 140, a sensor unit 150, and a control unit 160.

[0049] The display unit 110 is composed of a liquid crystal display, an organic EL (Electro-Luminescence) display, etc., and displays a digital twin and various information based on the control of the control unit 160.

[0050] The operation unit 120 is composed of a touch panel integrated with the display constituting the display unit 110, physical buttons provided on the housing of the device 100, a microphone, and the like. The operation unit 120 receives operations by the teacher and supplies operation information corresponding to the operations to the control unit 160.

[0051] The storage unit 130 stores programs necessary for operating the device 100, various data set in advance by the teacher and to be used in the lesson, and the like.

[0052] The communication unit 140 is composed of a network interface and the like, and performs communication with the device 200 on the student side based on the control of the control unit 160.

[0053] The sensor unit 150 is composed of one or more sensors, and supplies various sensor data obtained by sensing the body movements of the teacher to the control unit 160.

[0054] For example, the sensor unit 150 is composed of one or more ToF (Time of Flight) sensors and an RGB sensor. The control unit 160 generates a teacher digital twin based on the ToF data obtained by the ToF sensor and the RGB data (video data) obtained by the RGB sensor. When the sensor unit 150 is composed of a plurality of ToF sensors and an RGB sensor, the control unit 160 can also generate a teacher digital twin based on the volumetric capture data generated by volumetric capture using the acquired sensor data. The sensor unit 150 may be composed of various sensors capable of acquiring sensor data other than ToF data and RGB data.

[0055] The control unit 160 executes various processes based on the programs stored in the storage unit 130, the operation information from the operation unit 120, and the information acquired via the communication unit 140.

[0056] The control unit 160 includes a digital twin generation unit 161 and an instruction information generation unit 162. Each functional unit included in the control unit 160 is realized by executing a program stored in the storage unit 130.

[0057] On the other hand, the student-side device 200 includes a display unit 210, an operation unit 220, a storage unit 230, a communication unit 240, a sensor unit 250, and a control unit 260.

[0058] The display unit 210 is composed of a liquid crystal display, an organic EL display, etc., and displays a digital twin and various information based on the control of the control unit 260.

[0059] The operation unit 220 is composed of a touch panel integrated with the display constituting the display unit 210, physical buttons provided on the housing of the device 200, a microphone, etc. The operation unit 220 accepts operations by students and supplies operation information corresponding to the operations to the control unit 260.

[0060] The storage unit 230 stores programs necessary for operating the device 200, various data prepared in advance by students, etc.

[0061] The communication unit 240 is composed of a network interface, etc., and performs communication with the teacher-side device 100 based on the control of the control unit 260.

[0062] The sensor unit 250 is composed of a plurality of sensors, and supplies various sensor data obtained by sensing the physical movements of students to the control unit 260.

[0063] Specifically, the sensor unit 250 is composed of one or more ToF sensors and RGB sensors. The sensor unit 250 of the student-side device 200 may be configured in the same way as the sensors of the teacher-side device 100, or may be calibrated with a different number and type of sensors from the sensors of the teacher-side device 100.

[0064] The control unit 260 executes various processes based on the programs stored in the storage unit 230, the operation information from the operation unit 220, and the information acquired via the communication unit 240.

[0065] The control unit 260 includes a digital twin generation unit 261, a digital twin adjustment unit 262, an overlay video generation unit 263, an evaluation unit 264, an effect generation unit 265, and a display control unit 266. Each functional unit included in the control unit 260 is realized by executing a program stored in the storage unit 230.

[0066] As shown in FIG. 5, each functional unit included in the control unit 160 of the teacher-side device 100 and each functional unit included in the control unit 260 of the student-side device 200 execute each process by transmitting and receiving information to and from each other as indicated by the arrows in the figure. In FIG. 5, the information corresponding to the dashed arrows is actually transmitted and received via the communication unit 140 of the teacher-side device 100 and the communication unit 240 of the student-side device 200.

[0067] Hereinafter, details of each functional unit included in the teacher-side device 100 (control unit 160) and each functional unit included in the student-side device 200 (control unit 260) will be described.

[0068] (Details of the digital twin generation unit) FIG. 6 is a diagram for explaining details of the digital twin generation unit 161 of the teacher-side device 100 and the digital twin generation unit 261 of the student-side device 200.

[0069] Note that the digital twin generation unit 161 of the teacher-side device 100 and the digital twin generation unit 261 of the student-side device 200 are similarly configured, and thus will be described as the digital twin generation unit N61 as shown in FIG. 6. Also, the sensor unit 150 of the teacher-side device 100 and the sensor unit 250 of the student-side device 200 will be similarly described as the sensor unit N50.

[0070] The digital twin generation unit N61 generates a virtual object that performs the same body movements as a person, i.e., a digital twin that reflects the body movements of that person, based on the body movements of the person. The digital twin generation unit N61 is composed of a feature point extraction unit N71, a background processing unit N72, and a 3D model generation unit N73.

[0071] Based on the sensor data from the sensor unit N50, the feature point extraction unit N71 extracts, as the feature point information of the person (teacher or student), the skeleton information representing the skeleton and joint points of the person, the 3D contour information representing the three-dimensional contour of the person, and the acceleration information representing the movement of the person's body. The feature point information is data on the time axis that changes continuously with time.

[0072] The skeleton information is extracted, for example, by performing skeleton estimation using machine learning or the like. The skeleton estimation may be performed using only one of the ToF data and the RGB data, or may be performed using both the ToF data and the RGB data.

[0073] The 3D contour information is extracted, for example, based on a depth image composed of ToF data.

[0074] The acceleration information is calculated, for example, based on the displacement of the skeleton and joint points represented by the skeleton information. When the person wears an acceleration sensor as one of the sensor units N50 on each part of the body, the acceleration information may be acquired based on the sensor data from the acceleration sensor. The acceleration information also includes left-right information indicating which part of the body on the left or right (such as hands, arms, legs, etc.) is being moved.

[0075] These feature point information are supplied to the background processing unit N72 together with the RGB data (video data).

[0076] Based on the feature point information from the feature point extraction unit N71 and the video data, the background processing unit N72 removes the background of the person in the video data. The video data with the background removed is supplied to the 3D model generation unit N73 together with the feature point information.

[0077] Based on the video data with the background removed and the feature point information from the background processing unit N72, the 3D model generation unit N73 generates a digital twin of a person.

[0078] First, based on the three-dimensional contour information, the 3D model generation unit N73 creates a three-dimensional model (3D model) by modeling the target person. Next, the 3D model generation unit N73 associates the skeleton and joint points represented by the skeleton information with the created 3D model. Thereby, the body movement of the person can be reflected in the 3D model. Then, the 3D model generation unit N73 synthesizes skin data corresponding to human skin for the 3D model.

[0079] As the skin data, skin data with different visual textures is prepared for each purpose of the body movement of the person. The purposes of the body movement include, for example, aerobics, yoga, dance, golf, soccer, etc., and are pre-selected by a teacher or a student. Note that the purposes of the body movement are not limited to the above-mentioned sports, and may include artistic creation activities such as playing musical instruments such as the guitar and the piano, and operating a potter's wheel in pottery.

[0080] Then, the 3D model generation unit N73 generates a digital twin of the corresponding type according to the purpose by synthesizing the skin data corresponding to the selected purpose of the body movement for the 3D model. For example, when soccer is selected as the purpose of the body movement, soccer skin data is synthesized for the 3D model, and a digital twin for soccer is generated. At this time, meta information indicating the purpose of the body movement (for example, soccer) may be associated with the sensor data and stored for the generated digital twin.

[0081] As described above, the digital twin generation unit N61 extracts feature point information based on the sensor data, and generates a digital twin as a 3D model based on the extracted feature point information. The generated digital twin is added with the feature point information extracted based on the sensor data and output to the subsequent stage.

[0082] (Details of the Instruction Information Generation Unit) FIG. 7 is a diagram for explaining the details of the instruction information generation unit 162 of the teacher's device 100.

[0083] Based on the operation information corresponding to the operation of the operation unit 120 by the teacher, the instruction information generation unit 162 generates instruction information representing instructions for the students and supplies it to the display control unit 266 of the student's device 200.

[0084] The operation information here includes, for example, setting information for setting a GUI such as the button 41 shown on the screens #1 and #2 in FIG. 2, and setting information for setting a fitting point shown on the screen #3 in FIG. 3. That is, the teacher can set the GUI and fitting points displayed on the display unit 210 of the student's device 200 by operating the operation unit 120.

[0085] In this case, based on the operation information (setting information), the instruction information generation unit 162 generates display information for displaying a GUI and fitting points as shown in FIG. 2 as instruction information. Such display information may be generated, for example, based on the display data stored in the storage unit 130, or may be generated based on the display data acquired via the communication unit 140.

[0086] In addition, the instruction information generation unit 162 may generate instruction information based on the evaluation value from the evaluation unit 264 of the student-side device 200. The evaluation value represents, for example, the evaluation result of the lesson (such as the movement matching rate) shown in the pop-up 47 on the screen #5 of FIG. 3, and a comment corresponding to the evaluation value is automatically generated as the instruction information. These comments may be prepared in advance for each evaluation value, and the comment corresponding to the evaluation value may be selected. The comment generated as the instruction information may be integrated with the comment input by the teacher as the operation information corresponding to the operation of the operation unit 120. Note that the teacher may not be able to input an appropriate comment based only on the evaluation result such as the movement matching rate. Therefore, the instruction information generation unit 162 may generate instruction information based on the superimposed video, effect video, single student digital twin, or RGB data (video data) of the student from the student-side device 200, or may accept the input of the comment by the teacher.

[0087] These instruction information are displayed on the display unit 210 based on the control of the display control unit 266 in the student-side device 200.

[0088] (Details of the Digital Twin Adjustment Unit) FIG. 8 is a diagram for explaining the details of the digital twin adjustment unit 262 of the student-side device 200.

[0089] The digital twin adjustment unit 262 generates an adjusted teacher digital twin (adjusted reference digital twin) by adjusting the teacher digital twin from the digital twin generation unit 161 to be superimposed with the student digital twin from the digital twin generation unit 261. The generated adjusted teacher digital twin is supplied to the superimposed video generation unit 263 and the evaluation unit 264.

[0090] Here, the teacher digital twin is adjusted to match the student digital twin based on the student digital twin so that the student, who is the user, can easily imitate by comparing his / her own movement with the movement of the teacher who is the instructor.

[0091] Specifically, the digital twin adjustment unit 262 changes the teacher's feature point information included in the teacher digital twin so as to approach the student's feature point information, based on the student's feature point information included in the student digital twin.

[0092] For example, by changing the skeletal information of the teacher digital twin to match the skeletal information of the student digital twin, the size (scale) of the teacher digital twin is adjusted. By changing the left-right information of the teacher digital twin to match the left-right information of the student digital twin, the dominant arm and leg of the teacher digital twin are adjusted. By changing the three-dimensional contour information of the teacher digital twin to match the three-dimensional contour information of the student digital twin, the body shape of the teacher digital twin is adjusted.

[0093] Then, the digital twin adjustment unit 262 creates a 3D model based on the changed teacher's feature point information, and generates an adjusted teacher digital twin including the adjusted feature point information as an adjusted 3D model. The digital twin adjustment unit 262 can generate an adjusted teacher digital twin in the same manner as the digital twin generation unit N61 in FIG. 6.

[0094] (Details of the superimposed video generation unit) FIG. 9 is a diagram for explaining the details of the superimposed video generation unit 263 of the student-side device 200.

[0095] The superimposed video generation unit 263 generates a superimposed video by superimposing the student digital twin from the digital twin generation unit 261 and the adjusted teacher digital twin from the digital twin adjustment unit 262, and supplies it to the effect generation unit 265 and the display control unit 266.

[0096] Specifically, the superimposed video generation unit 263 generates a superimposed video by mapping the adjusted teacher digital twin and the student digital twin to a predetermined reference position in the virtual space and synchronizing them at a predetermined reference time.

[0097] The superimposed image is displayed on the display unit 210 based on the control of the display control unit 266.

[0098] (Details of the evaluation unit) FIG. 10 is a diagram for explaining the details of the evaluation unit 264 of the student-side device 200.

[0099] The evaluation unit 264 calculates an evaluation value of the student digital twin (i.e., the physical movement of the student) by comparing the student digital twin from the digital twin generation unit 261 and the adjusted teacher digital twin from the digital twin adjustment unit 262.

[0100] For example, the evaluation unit 264 obtains, as the evaluation value, the difference (posture deviation) of the contour information between the student digital twin and the adjusted teacher digital twin. Also, the evaluation unit 264 obtains, as the evaluation value, the difference (movement deviation) of the acceleration information between the student digital twin and the adjusted teacher digital twin. Furthermore, the evaluation unit 264 obtains, as the evaluation value, the difference (posture deviation) of the fitting points between the student digital twin and the adjusted teacher digital twin.

[0101] Among the evaluation values calculated in this way, the 3D model information representing (visualizing) the difference in a 3D model is supplied to the effect generation unit 265. Also, among the calculated evaluation values, the meta information (such as the amount of deviation and the deviated part) obtained by digitizing or textifying the difference is supplied to the display control unit 266 and the instruction information generation unit 162 (teacher-side device 100).

[0102] (Details of the effect generation unit) FIG. 11 is a diagram for explaining the details of the effect generation unit 265 of the student-side device 200.

[0103] Based on the evaluation value (3D model information) from the evaluation unit 264, the effect generation unit 265 generates an effect video for the superimposed video from the superimposed video generation unit 263. The effect video is, for example, a video for highlighting the areas (parts) where the 3D models of the student digital twin and the adjusted teacher digital twin are misaligned with a predetermined color or texture, a predetermined figure or pattern to be synthesized on the background of the student digital twin when misaligned, and further, lines or afterimages representing the movement trajectories of the student digital twin and the adjusted teacher digital twin.

[0104] The effect generation unit 265 maps the effect video to a predetermined reference position in the virtual space and synchronizes it at a predetermined reference time, then superimposes the effect video on the superimposed video and supplies it to the display control unit 266.

[0105] Similar to the skin data, for each purpose of the human body movement, different types of effect videos with different visual textures are prepared for the effect video. That is, the effect generation unit 265 generates the type of effect video corresponding to the selected purpose of the body movement. For example, when soccer is selected as the purpose of the body movement, the type of effect video corresponding to soccer is generated, and when aerobics is selected as the purpose of the body movement, the type of effect video corresponding to aerobics is generated.

[0106] As described above, the display control unit 266 may display only the superimposed video from the superimposed video generation unit 263 on the display unit 210, or may display the superimposed video with the effect video from the effect generation unit 265 superimposed on the display unit 210.

[0107] When the effect video is displayed on the display unit 210, the display control unit 266 can also switch the effect video displayed on the display unit 210 to another texture effect video, etc., according to, for example, the operation of the user (student). In this case, for one purpose of the body movement, a plurality of types of effect videos with different textures are prepared.

[0108] (Operation of the Information Processing System) Next, the operations of the teacher-side device 100 and the student-side device 200 that constitute the above-described information processing system will be described.

[0109] FIG. 12 is a flowchart for explaining the operation of the teacher-side device 100 when a teacher is performing, for example, in a real-time class. The process of FIG. 12 is executed, for example, when a student instructs the start of a lesson.

[0110] In step S11, the digital twin generation unit 161 generates a teacher digital twin based on the sensor data sensed by the sensor unit 150 for the teacher.

[0111] In step S12, the control unit 160 controls the communication unit 140 to transmit the teacher digital twin generated by the digital twin generation unit 161 to the student-side device 200.

[0112] FIG. 13 is a flowchart for explaining the operation of the student-side device 200 when a teacher is performing, for example, in a real-time class. The process of FIG. 13 is executed in conjunction with the process of FIG. 12.

[0113] In step S21, the digital twin generation unit 261 generates a student digital twin based on the sensor data sensed by the sensor unit 250 for the student.

[0114] In step S22, the digital twin adjustment unit 262 generates an adjusted teacher digital twin by adjusting the teacher digital twin received from the student-side device 200 based on the student digital twin generated by the digital twin generation unit 261.

[0115] In step S23, the superimposed video generation unit 263 generates a superimposed video by superimposing the student digital twin and the adjusted teacher digital twin.

[0116] In step S24, the evaluation unit 264 calculates the evaluation value of the student digital twin by evaluating the student digital twin using the adjusted teacher digital twin.

[0117] In step S25, the effect generation unit 265 generates an effect video for the superimposed image based on the 3D model information among the evaluation values calculated by the evaluation unit 264.

[0118] In step S26, the display control unit 266 causes the display unit 210 to display the superimposed video generated by the superimposed video generation unit 263 and the effect video generated by the effect generation unit 265.

[0119] By the way, the meta information among the evaluation values calculated by the evaluation unit 264 is also transmitted to the teacher's device 100.

[0120] FIG. 14 is a flowchart for explaining the operation of the teacher's device 100 based on the evaluation value from the student's device 200. The process of FIG. 14 is executed in parallel with the process of FIG. 13.

[0121] In step S31, the instruction information generation unit 162 generates instruction information based on the evaluation value (meta information) from the student's device 200. Specifically, the instruction information generation unit 162 generates, as the instruction information, display information representing the amount of deviation and the deviated part of the student's movement with respect to the teacher's movement. This display information may include comments automatically generated according to the evaluation value (meta information) and comments input by the teacher.

[0122] In step S32, the control unit 160 controls the communication unit 140 to transmit the instruction information generated by the instruction information generation unit 162 to the student's device 200.

[0123] On the student's device 200, the display control unit 266 causes the display unit 210 to display the instruction information from the teacher's device 100 together with the superimposed video and the effect video.

[0124] According to the above process, since the teacher digital twin is adjusted according to the student digital twin, the student can easily imitate by comparing his own movements with those of the teacher while watching the superimposed video.

[0125] In addition, since the effect video based on the difference from the teacher's movement is superimposed and displayed on the superimposed video, the student can easily grasp the deviation between his own movement and the teacher's movement.

[0126] Furthermore, together with the effect video, the instruction information indicating the amount and part of the deviation of the student's own movement and the corresponding comments is displayed, so that the student can understand how specifically his own movement deviates and how to move.

[0127] In the above way, it becomes possible to provide more effective learning content for students to learn body movements.

[0128] Note that in the above, only the evaluation value calculated by the evaluation unit 264 is transmitted from the device 200 on the student side to the device 100 on the teacher side. However, this is not the only case. The superimposed video generated by the superimposed video generation unit 263 or the effect video generated by the effect generation unit 265 may be transmitted from the device 200 on the student side to the device 100 on the teacher side. In this case, on the device 100 on the teacher side, the superimposed video and the effect video are displayed on the display unit 110 under the control of the control unit 160.

[0129] Thereby, the teacher can also easily grasp the deviation between his own movement and the student's movement, and can present more appropriate instructions and advice to the student as instruction information (comments). Note that the comments for the student may be presented not only as character information but also as voice information.

[0130] <Application Example of 3.5G Network Slicing> As described above, in the information processing system to which the technology according to the present disclosure is applied, 5G can be applied as a communication method between devices.

[0131] 5G has three characteristics: "high speed and large capacity", "low latency", and "many simultaneous connections". These functions can be realized by a technology called network slicing that virtually divides (slices) the network. In 5G, according to the type and use of data, it can be transmitted through a high-speed and large-capacity network slice (hereinafter simply referred to as a slice) or through a low-latency network slice.

[0132] (3-1. Application Example 1 of 5G Network Slicing) Hereinafter, an example of applying 5G network slicing to the information processing system to which the technology according to the present disclosure is applied will be described.

[0133] (3-1-1. Device-Device Configuration 1) FIG. 15 is a diagram showing an example of applying 5G network slicing to the information processing system described above. In the figure, thick arrows indicate 5G-compatible transmission paths.

[0134] In the example of FIG. 15, among the teacher digital twins generated by the digital twin generation unit 161 on the teacher side, the feature point information is transmitted to the student side (digital twin adjustment unit 262) via a low-latency slice, and the 3D model is transmitted via a large-capacity slice.

[0135] Also, from the student side, the evaluation value calculated by the evaluation unit 264, the superimposed video generated by the superimposed video generation unit 263, and the effect video generated by the effect generation unit 265 are transmitted to the teacher side (instruction information generation unit 162) via a large-capacity slice via a low-latency slice.

[0136] In this case, the instruction information generation unit 162 may generate instruction information for the student based on the superimposed video from the superimposed video generation unit 263 and the effect video from the effect generation unit 265. Further, the superimposed video and the effect video supplied to the instruction information generation unit 162 may be displayed on the display unit 110 based on the control of the control unit 160.

[0137] In this way, since the feature point information and the evaluation value that require real-time performance are transmitted through low-latency slices, it is possible to ensure the followability of the digital twin to the teacher's body movement and the rapidity of the feedback regarding the student's body movement.

[0138] By the way, each functional unit of the control unit 160 described above and each functional unit of the control unit 260 do not necessarily have to be realized on the teacher-side device 100 and the student-side device 200, respectively.

[0139] (3-1-2. Device - Device Configuration 2) As shown in FIG. 16, the digital twin adjustment unit 262 may be realized on the teacher-side device 100.

[0140] In the example of FIG. 16, from the teacher side, among the adjusted teacher digital twins generated by the digital twin adjustment unit 262, the adjusted feature point information is transmitted to the student side (the superimposed video generation unit 263 and the evaluation unit 264) through a low-latency slice, and the adjusted 3D model is transmitted through a large-capacity slice.

[0141] Further, from the student side, the evaluation value calculated by the evaluation unit 264 is transmitted to the teacher side (the instruction information generation unit 162) through a low-latency slice, and the superimposed video generated by the superimposed video generation unit 263 and the effect video generated by the effect generation unit 265 are transmitted through a large-capacity slice. Note that the feature point information of the student digital twin generated by the digital twin generation unit 261 may be transmitted to the teacher side (the digital twin adjustment unit 262) through a low-latency slice.

[0142] In the above, an example in which the digital twin adjustment unit 262 is realized either on the teacher's device 100 or on the student's device 200 has been described, but it may be realized on both devices 100 and 200. Further, the functions of the teacher's device 100 and the functions of the student's device 200 may be switched at a predetermined timing.

[0143] (3-1-3. Configuration 1 of Device-MEC-Device) As shown in FIG. 17, the digital twin generation unit 161 and the instruction information generation unit 162 may be realized on the MEC server 10TE close to the teacher's device 100, and the digital twin generation units 261 to the effect generation unit 265 may be realized on the MEC server 10ST close to the student's device 200.

[0144] In this case, the teacher's device 100 transmits the sensing data acquired by the sensor unit 150 to the MEC server 10TE (digital twin generation unit 161). Similarly, the student's device 200 transmits the sensing data acquired by the sensor unit 250 to the MEC server 10ST (MEC server 10ST).

[0145] Note that the MEC server 10TE (digital twin generation unit 161) may generate a teacher digital twin by extracting feature points from the recorded content stored in the cloud server 20. Thereby, the use case UC2 and the use case UC4 in FIG. 4 are realized.

[0146] In the example of FIG. 17, among the teacher digital twins generated by the digital twin generation unit 161 from the teacher-side MEC server 10TE, the feature point information is transmitted to the student-side MEC server 10ST (digital twin adjustment unit 262) via the low-latency slice, and the 3D model is transmitted via the large-capacity slice.

[0147] Also, from the MEC server 10ST on the student side, the superimposed video generated by the superimposed video generation unit 263 and the effect video generated by the effect generation unit 265 are transmitted to the MEC server 10TE (instruction information generation unit 162) on the teacher side via a large-capacity slice, with the evaluation value calculated by the evaluation unit 264 transmitted via a low-latency slice.

[0148] (3-1-4. Device-MEC-Device Configuration 2) As shown in FIG. 18, the digital twin generation unit 161 and the instruction information generation unit 162 may be realized on the MEC server 10TE close to the teacher-side device 100, and the digital twin generation unit 261, the digital twin adjustment unit 262, and the evaluation unit 264 may be realized on the MEC server 10ST close to the student-side device 200.

[0149] In the example of FIG. 18, among the teacher digital twins generated by the digital twin generation unit 161 from the MEC server 10TE on the teacher side, the feature point information is transmitted to the MEC server 10ST (digital twin adjustment unit 262) on the student side via a low-latency slice, and the 3D model is transmitted via a large-capacity slice.

[0150] From the MEC server 10ST on the student side, among the student digital twins generated by the digital twin generation unit 261, the feature point information is transmitted to the student-side device 200 (superimposed video generation unit 263) via a low-latency slice, and the 3D model is transmitted via a large-capacity slice. Similarly, among the adjusted teacher digital twins generated by the digital twin adjustment unit 262, the adjusted feature point information is transmitted to the student-side device 200 (superimposed video generation unit 263) via a low-latency slice, and the adjusted 3D model is transmitted via a large-capacity slice.

[0151] Also, the evaluation value calculated by the evaluation unit 264 is transmitted from the MEC server 10ST on the student side to the MEC server 10TE (instruction information generation unit 162) on the teacher side via a low-latency slice. Further, the superimposed video generated by the superimposed video generation unit 263 and the effect video generated by the effect generation unit 265 are transmitted from the device 200 on the student side to the MEC server 10TE (instruction information generation unit 162) on the teacher side via a large-capacity slice.

[0152] (3-1-5. Device-MEC-Device Configuration 3) As shown in FIG. 19, the digital twin generation unit 161 may be implemented on the MEC server 10TE close to the teacher-side device 100, and the digital twin generation unit 261 and the digital twin adjustment unit 262 may be implemented on the MEC server 10ST close to the student-side device 200.

[0153] In the example of FIG. 19, among the teacher digital twins generated by the digital twin generation unit 161 from the teacher-side MEC server 10TE, the feature point information is transmitted to the student-side MEC server 10ST (digital twin adjustment unit 262) via a low-latency slice, and the 3D model is transmitted via a large-capacity slice.

[0154] From the student-side MEC server 10ST, among the student digital twins generated by the digital twin generation unit 261, the feature point information is transmitted to the student-side device 200 (superimposed video generation unit 263 and evaluation unit 264) via a low-latency slice, and the 3D model is transmitted via a large-capacity slice. Similarly, among the adjusted teacher digital twins generated by the digital twin adjustment unit 262, the adjusted feature point information is transmitted to the student-side device 200 (superimposed video generation unit 263 and evaluation unit 264) via a low-latency slice, and the adjusted 3D model is transmitted via a large-capacity slice.

[0155] Further, from the student-side device 200, the superimposed video generated by the superimposed video generation unit 263 and the effect video generated by the effect generation unit 265 are transmitted via a large-capacity slice to the teacher-side device 100 (instruction information generation unit 162), and the evaluation value calculated by the evaluation unit 264 is transmitted via a low-latency slice.

[0156] (3-1-6. Device-MEC-Device Configuration 4) As shown in FIG. 20, the digital twin generation unit 161 may be implemented on the MEC server 10TE close to the teacher-side device 100, and the digital twin generation units 261 to 264 may be implemented on the MEC server 10ST close to the student-side device 200.

[0157] In the example of FIG. 20, among the teacher digital twins generated by the digital twin generation unit 161 from the teacher-side MEC server 10TE, the feature point information is transmitted via a low-latency slice, and the 3D model is transmitted via a large-capacity slice to the student-side MEC server 10ST (digital twin adjustment unit 262).

[0158] Further, from the student-side MEC server 10ST, the evaluation value calculated by the evaluation unit 264 is transmitted via a low-latency slice to the teacher-side device 100 (instruction information generation unit 162). Further, from the student-side device 200, the superimposed video generated by the superimposed video generation unit 263 and the effect video generated by the effect generation unit 265 are transmitted via a large-capacity slice to the teacher-side device 100 (instruction information generation unit 162).

[0159] Note that the superimposed video generated by the superimposed video generation unit 263 may be transmitted via a large-capacity slice from the student-side MEC server 10ST to the student-side device 200 (effect generation unit 265). Further, the evaluation value (3D model information) calculated by the evaluation unit 264 may be transmitted via a large-capacity slice from the student-side MEC server 10ST to the student-side device 200 (effect generation unit 265).

[0160] (3-1-7. Configuration of Device-MEC-Device 5) As shown in FIG. 21, only the digital twin adjustment unit 262 may be implemented on the MEC server 10ST close to the student-side device 200.

[0161] In the example of FIG. 21, among the teacher digital twins generated by the digital twin generation unit 161 from the teacher-side device 100, the feature point information is transmitted to the student-side MEC server 10ST (digital twin adjustment unit 262) via a low-latency slice, and the 3D model is transmitted via a large-capacity slice.

[0162] From the student-side MEC server 10ST, among the adjusted teacher digital twins generated by the digital twin adjustment unit 262, the adjusted feature point information is transmitted to the student-side device 200 (superimposed video generation unit 263 and evaluation unit 264) via a low-latency slice, and the adjusted 3D model is transmitted via a large-capacity slice.

[0163] Also, from the student-side device 200, the evaluation value calculated by the evaluation unit 264 is transmitted to the teacher-side device 100 (instruction information generation unit 162) via a low-latency slice, and the superimposed video generated by the superimposed video generation unit 263 and the effect video generated by the effect generation unit 265 are transmitted via a large-capacity slice.

[0164] Note that the feature point information among the student digital twins generated by the digital twin generation unit 261 may be transmitted to the student-side MEC server 10ST (digital twin adjustment unit 262) via a low-latency slice.

[0165] In the example of FIG. 21, although the digital twin adjustment unit 262 is implemented on the MEC server 10ST close to the student-side device 200, it may also be implemented on the MEC server 10TE close to the teacher-side device 100.

[0166] (3-2. Other Configuration Examples of Information Processing System) In the above, mainly the configuration of the information processing system that realizes the real-time class has been described. On the other hand, if a pre-generated teacher digital twin can be played back, a user (student) at home can receive a non-real-time lesson at a desired timing instead of a real-time class.

[0167] FIG. 22 is a block diagram showing another configuration example of the information processing system to which the technology according to the present disclosure is applied.

[0168] The information processing system of FIG. 22 is composed of a cloud server 20 and a student-side device 200. The student-side device 200 in FIG. 22 is configured in the same manner as the above-described student-side device 200, but in FIG. 22, only the main functional parts are shown.

[0169] The cloud server 20 includes a storage device 310 and an instruction information generation unit 320.

[0170] The storage device 310 stores a pre-generated teacher digital twin and supplies the teacher digital twin to the student-side device 200 in response to a request from the student-side device 200.

[0171] The instruction information generation unit 320 has basically the same function as the above-described instruction information generation unit 162, but is different from the instruction information generation unit 162 in that it automatically generates instruction information based on AI (artificial intelligence).

[0172] FIG. 23 is a diagram for explaining the details of the storage device 310.

[0173] As shown in FIG. 23, the storage device 310 includes a communication unit 311, a storage unit 312, and a control unit 313.

[0174] The communication unit 311 is composed of a network interface or the like and communicates with the student-side device 200 based on the control of the control unit 313.

[0175] The storage unit 312 stores programs necessary for operating the storage device 310, various types of prepared data, and the like.

[0176] Specifically, the storage unit 312 stores the real-time performance of a person and the teacher digital twin generated based on the recorded content. In response to a request from the student-side device 200, the stored teacher digital twin is read out.

[0177] Also, the storage unit 312 stores sensor data and feature point information acquired in advance, and the teacher digital twin may be generated based on the sensor data and the feature point information. Furthermore, predetermined recorded content may be stored in the storage unit 312, and the teacher digital twin may be generated based on the recorded content.

[0178] The control unit 313 executes various processes based on the programs stored in the storage unit 312. For example, the control unit 313 supplies the teacher digital twin stored in the storage unit 312 to the student-side device 200 in response to a request from the student-side device 200, or generates a teacher digital twin based on the sensor data and the feature point information stored in the storage unit 312.

[0179] Even in the above configuration, since the teacher digital twin is adjusted according to the student digital twin, the student can easily imitate by comparing their own movements with the teacher's movements while watching the superimposed video.

[0180] Also, since an effect video based on the difference from the teacher's movement is superimposed and displayed on the superimposed video, the student can easily grasp the deviation between their own movement and the teacher's movement.

[0181] Furthermore, along with the effect video, instruction information indicating the amount of deviation of one's own movement, the deviated part, and comments corresponding thereto is displayed, so that the student can understand how specifically their own movement is deviated and how they should move.

[0182] In the above manner, it becomes possible to provide more effective learning content for students to learn body movements.

[0183] In the storage device 310, the teacher digital twin, sensor data, and feature point information stored in the storage device 310 may be managed in association with the person who performed the body movement reflected therein. Also, in the storage device 310, for example, feature point information is extracted from a game video of a professional soccer player, and the teacher digital twin generated based on skeleton estimation using machine learning or the like may be managed in association with that professional soccer player.

[0184] For example, a person ID for identifying the person, time information indicating the date and time when the digital twin was generated, genre information indicating the purpose and type of the body movement, etc. are associated with the digital twin in which a person's body movement is reflected.

[0185] As a result, the user who is a student can select a desired person or digital twin of body movement and receive a non-real-time lesson.

[0186] Furthermore, the digital twin associated with the person ID may be the subject of an e-commerce transaction in a marketplace (electronic market). In this case, in the storage device 310, metadata of copyright information including the person ID, selling price, selling period, etc. of the digital twin is made into a database and managed centrally.

[0187] This makes it possible to manage the copyright of the digital twin provider, such as protecting the provider's own actions, such as those of an instructor who provides a digital twin, as a work, or having the provider enter into a license agreement with a predetermined company or organization, etc.

[0188] (3 - 3.5G Network Slicing Application Example 2) 5G network slicing can also be applied to the information processing system of Fig. 22.

[0189] (3 - 3 - 1. Device - MEC - Cloud Configuration 1) Fig. 24 is a diagram showing an example in which 5G network slicing is applied to the information processing system of Fig. 22. In the figure, the thick - line arrows indicate 5G - compatible transmission paths.

[0190] In the example of Fig. 24, the digital twin generation unit 261 to the effect generation unit 265 are realized on the MEC server 10ST close to the student - side device 200.

[0191] In this case, the student - side device 200 transmits the sensing data acquired by the sensor unit 250 to the MEC server 10ST (digital twin generation unit 261).

[0192] In the example of Fig. 24, from the cloud server 20, among the teacher digital twins stored in the storage device 310, the feature point information is transmitted to the student - side MEC server 10ST (digital twin adjustment unit 262) via the low - latency slice, and the 3D model is transmitted via the large - capacity slice.

[0193] Also, from the student - side MEC server 10ST, the evaluation value calculated by the evaluation unit 264 is transmitted to the cloud server 20 (instruction information generation unit 320) via the low - latency slice, and the superimposed video generated by the superimposed video generation unit 263 and the effect video generated by the effect generation unit 265 are transmitted via the large - capacity slice.

[0194] (3-3-2. Configuration of Device-MEC-Cloud 2) As shown in FIG. 25, the digital twin generation unit 261, the digital twin adjustment unit 262, and the evaluation unit 264 may be implemented on the MEC server 10ST close to the student-side device 200.

[0195] In the example of FIG. 25, from the cloud server 20, among the teacher digital twins stored in the storage device 310, the feature point information is transmitted to the student-side MEC server 10ST (digital twin adjustment unit 262) via the low-latency slice, and the 3D model is transmitted via the large-capacity slice.

[0196] From the student-side MEC server 10ST, among the student digital twins generated by the digital twin generation unit 261, the feature point information is transmitted to the student-side device 200 (superimposed video generation unit 263) via the low-latency slice, and the 3D model is transmitted via the large-capacity slice. Similarly, among the adjusted teacher digital twins generated by the digital twin adjustment unit 262, the adjusted feature point information is transmitted to the student-side device 200 (superimposed video generation unit 263) via the low-latency slice, and the adjusted 3D model is transmitted via the large-capacity slice.

[0197] Also, from the student-side MEC server 10ST, the evaluation value calculated by the evaluation unit 264 is transmitted to the cloud server 20 (instruction information generation unit 320) via the low-latency slice. Further, from the student-side device 200, the superimposed video generated by the superimposed video generation unit 263 and the effect video generated by the effect generation unit 265 are transmitted to the cloud server 20 (instruction information generation unit 320) via the large-capacity slice.

[0198] (3-3-3. Configuration of Device-MEC-Cloud 3) As shown in FIG. 26, the digital twin generation unit 261 and the digital twin adjustment unit 262 may be implemented on the MEC server 10ST close to the student-side device 200.

[0199] In the example of FIG. 26, among the teacher digital twins stored in the storage device 310, from the cloud server 20, the feature point information is transmitted to the student-side MEC server 10ST (digital twin adjustment unit 262) via a low-latency slice, and the 3D model is transmitted via a large-capacity slice.

[0200] From the student-side MEC server 10ST, among the student digital twins generated by the digital twin generation unit 261, the feature point information is transmitted to the student-side device 200 (superimposed video generation unit 263 and evaluation unit 264) via a low-latency slice, and the 3D model is transmitted via a large-capacity slice. Similarly, among the adjusted teacher digital twins generated by the digital twin adjustment unit 262, the adjusted feature point information is transmitted to the student-side device 200 (superimposed video generation unit 263 and evaluation unit 264) via a low-latency slice, and the adjusted 3D model is transmitted via a large-capacity slice.

[0201] Also, from the student-side device 200, the evaluation value calculated by the evaluation unit 264 is transmitted to the cloud server 20 (instruction information generation unit 320) via a low-latency slice, and the superimposed video generated by the superimposed video generation unit 263 and the effect video generated by the effect generation unit 265 are transmitted via a large-capacity slice.

[0202] (3-3-4. Device-MEC-Cloud Configuration 4) As shown in FIG. 27, the digital twin generation unit 261 to the evaluation unit 264 may be implemented on the MEC server 10ST close to the student-side device 200.

[0203] In the example of FIG. 27, among the teacher digital twins stored in the storage device 310, from the cloud server 20, the feature point information is transmitted to the student-side MEC server 10ST (digital twin adjustment unit 262) via a low-latency slice, and the 3D model is transmitted via a large-capacity slice.

[0204] Also, the evaluation value calculated by the evaluation unit 264 is transmitted from the student-side MEC server 10ST to the cloud server 20 (instruction information generation unit 320) via a low-latency slice. Further, the superimposed video generated by the superimposed video generation unit 263 and the effect video generated by the effect generation unit 265 are transmitted from the student-side device 200 to the cloud server 20 (instruction information generation unit 320) via a large-capacity slice.

[0205] Note that the superimposed video generated by the superimposed video generation unit 263 may be transmitted from the student-side MEC server 10ST to the student-side device 200 (effect generation unit 265) via a large-capacity slice. Further, the evaluation value (3D model information) calculated by the evaluation unit 264 may be transmitted from the student-side MEC server 10ST to the student-side device 200 (effect generation unit 265) via a large-capacity slice.

[0206] (3-3-5. Device-MEC-Cloud Configuration 5) As shown in FIG. 28, only the digital twin adjustment unit 262 may be realized on the MEC server 10ST close to the student-side device 200.

[0207] In the example of FIG. 28, among the teacher digital twins stored in the storage device 310 from the cloud server 20, the feature point information is transmitted to the student-side MEC server 10ST (digital twin adjustment unit 262) via a low-latency slice, and the 3D model is transmitted via a large-capacity slice.

[0208] From the student-side MEC server 10ST, among the adjusted teacher digital twins generated by the digital twin adjustment unit 262, the adjusted feature point information is transmitted to the student-side device 200 (superimposed video generation unit 263 and evaluation unit 264) via a low-latency slice, and the adjusted 3D model is transmitted via a large-capacity slice.

[0209] Also, from the student-side device 200, the superimposed video generated by the superimposed video generation unit 263 and the effect video generated by the effect generation unit 265 are transmitted to the cloud server 20 (instruction information generation unit 320) via a large-capacity slice, with the evaluation value calculated by the evaluation unit 264 transmitted via a low-latency slice.

[0210] Note that the feature point information of the student digital twin generated by the digital twin generation unit 261 may be transmitted to the teacher side (digital twin adjustment unit 262) via a low-latency slice.

[0211] As described above, 5G network slicing can also be applied to the information processing system of FIG. 22.

[0212] <4. Modification Example> In the following, modification examples of the above-described embodiments will be described.

[0213] (Display Example of Digital Twin) In the above, as the digital twin, a 3D model synthesized with skin data is displayed on the student-side device 200 or the like. In addition, as shown in FIG. 29, a skeleton image based on skeleton information may be superimposed and displayed on the 3D model as the digital twin.

[0214] In the example of FIG. 29, a skeleton image 430 representing the skeleton and joint points of the student is superimposed and displayed on the standing student digital twin 30ST (3D model). In the example of FIG. 29, the teacher digital twin 30TE shown in FIG. 2 is not superimposed, but the teacher digital twin 30TE may be further superimposed and displayed on the skeleton image 430.

[0215] (Presentation Example of Character Information) In the above, it is assumed that information such as digital twins, instruction information, and evaluation values is transmitted and received between the teacher's device 100 and the student's device 200. In addition to this, for example, status information representing the progress of the lesson received by the student and the state of the student performing physical movements during the lesson may be transmitted and received between the teacher's device 100 and the student's device 200.

[0216] Figures 30 to 32 are diagrams showing examples of presenting character information representing the above-mentioned status information on the student's device 200.

[0217] (Presentation Example 1) In the example of Figure 30, by performing an operation for the student to start the lesson, status information indicating that the lesson is to be started is transmitted from the student's device 200 to the teacher's device 100.

[0218] For example, in the state of screen #11 in Figure 30, together with the student digital twin 30ST standing upright, character information 441 representing the name of the lesson to be started is displayed. Also, in the upper right of screen #11, similar to Figure 2, a GUI button 41 for starting the lesson is displayed.

[0219] As shown in the state of screen #12, when the student raises one hand and it is determined that the hand of the corresponding student digital twin 30ST overlaps the area of button 41, the lesson by the instructor TE is started. At this time, the character information 441 changes to a particle-like particle video 442.

[0220] After that, status information indicating that the lesson is to be started is transmitted to the teacher's device 100, and as shown in the state of screen #13, the particles constituting the particle video 442 move so as to be sucked upward on screen #13.

[0221] In this way, the character information 441 changes to the particle video 442 and moves above the screen #13, so that the student who is the user can sensually understand that the status information indicating the start of the lesson has been transmitted to the teacher's device 100.

[0222] (Presentation Example 2) In the example of FIG. 31, status information indicating that it is dangerous for the student receiving the lesson to get too close to the display (display unit 210) is transmitted from the student's device 200 to the teacher's device 100.

[0223] For example, in the state of screen #21 in FIG. 31, character information 451 representing the distance between the student and the display is displayed together with the student digital twin 30ST. In screen #21, the character information 451 indicates that the distance between the student and the display is 146 cm.

[0224] When the distance between the student and the display falls below a predetermined threshold value (for example, 145 cm), as shown in the state of screen #22, the character information 451 changes to a particulate particle video 452.

[0225] Thereafter, status information indicating that the student has gotten too close to the display is transmitted to the teacher's device 100, and as shown in the state of screen #23, the particles constituting the particle video 452 move so as to be sucked upward on screen #23.

[0226] In this way, the character information 451 changes to the particle video 452 and moves above the screen #23, so that the student who is the user can sensually understand that the status information indicating that he / she has gotten too close to the display has been transmitted to the teacher's device 100.

[0227] (Presentation Example 3) In the example of FIG. 32, status information indicating the physical load of the student taking the lesson is transmitted from the student-side device 200 to the teacher-side device 100. The status information indicating the physical load of the student is generated based on, for example, vital signs obtained by a vital sensor provided as the sensor unit 250.

[0228] For example, in the state of screen #31 in FIG. 32, character information 461 representing the physical load state of the student is displayed together with the student digital twin 30ST. On screen #31, the character information 461 indicates that the physical load state of the student is "HARD".

[0229] When the vital signs of the student exceed a predetermined limit value, as shown in the state of screen #32, the character information 461 changes to a particulate particle image 462.

[0230] Thereafter, status information indicating that the physical load of the student has exceeded the limit is transmitted to the teacher-side device 100, and as shown in the state of screen #33, the particles constituting the particle image 462 move so as to be sucked upward on screen #33.

[0231] In this way, by the character information 461 changing to the particle image 462 and moving upward on screen #33, the student, who is the user, can sensually understand that status information indicating that his / her physical load has exceeded the limit has been transmitted to the teacher-side device 100.

[0232] In the above-described example, it is assumed that the progress status of the lesson and the character information representing the state of the student change to a particle image, but a part of the skeleton image 430 superimposed and displayed on the student digital twin 30ST may be changed to a particle image.

[0233] For example, when the skeletal image 430 corresponding to the part of the student digital twin that moves differently from the teacher digital twin changes to a particle video, the student can recognize that they have made a wrong movement.

[0234] In addition, when the character information changes to a particle video, for example, the display color may change, such as when the black character information changes to a red particle video.

[0235] (Application Example) The above-described presentation example can also be applied to a configuration in which, for example, a line supervisor in a factory individually monitors the states of on-site workers on the line. In this case, the line supervisor can grasp the work situation, physical load, mental stress, etc. of the on-site workers all at once, and when there is a possibility that the state of the on-site workers may interfere with the work, the line supervisor can immediately notify the factory management supervisor to that effect.

[0236] (Application of Face Recognition) In the above-described embodiment, face recognition may be performed, for example, when starting a lesson or starting work in a factory. Thereby, it is possible to avoid a teacher giving a lesson to the wrong student, or for a line supervisor in a factory to easily grasp the attendance status of on-site workers.

[0237] (Adjustment of Digital Twin) In the above-described embodiment, mainly, based on the student digital twin that reflects the body movement of the student who is the user (the first person), the teacher digital twin that reflects the body movement of the teacher who is the reference person (the second person) is adjusted so as to match the student digital twin. Conversely, based on the teacher digital twin, the student digital twin may be adjusted so as to match the teacher digital twin, or the reference person (the reference person) may be switched between the student and the teacher.

[0238] <5. Example of Computer Configuration> The above-described series of processes can be executed either by hardware or by software. When the series of processes is executed by software, the program constituting the software is installed in a computer. Here, the computer includes a computer incorporated in dedicated hardware, and a general-purpose personal computer, for example, which can execute various functions by installing various programs.

[0239] FIG. 33 is a block diagram showing a configuration example of the hardware of a computer that executes the above-described series of processes by a program.

[0240] In a computer, a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

[0241] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0242] The input unit 1006 includes a keyboard, a mouse, a microphone, etc. The output unit 1007 includes a display, a speaker, etc. The storage unit 1008 includes a hard disk, a non-volatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0243] In the computer configured as described above, the CPU 1001 loads and executes a program stored in the storage unit 1008 via the input / output interface 1005 and the bus 1004 into the RAM 1003, whereby the above-described series of processes is performed.

[0244] The program executed by the computer (CPU 1001) can be recorded and provided, for example, on a removable medium 1011 such as a package medium. Further, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0245] In the computer, the program can be installed in the storage unit 1008 via the input / output interface 1005 by attaching the removable medium 1011 to the drive 1010. Further, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Additionally, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

[0246] Note that the program executed by the computer may be a program in which processing is performed in time series according to the order described in this specification, or may be a program in which processing is performed in parallel or at a necessary timing such as when a call is made.

[0247] In this specification, the step of describing the program recorded on the recording medium includes not only the processing performed in time series according to the described order, but also the processing that is not necessarily processed in time series and is executed in parallel or individually.

[0248] Also, in this specification, a system means a collection of a plurality of components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a single device in which a plurality of modules are housed in one housing are both systems.

[0249] Embodiments of the technology according to the present disclosure are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the technology according to the present disclosure.

[0250] In addition, the effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0251] Furthermore, the technology according to the present disclosure can have the following configurations. (1) An adjustment unit that generates an adjusted second virtual object by adjusting a second virtual object that reflects the body movement of a second person and that is superimposed on the first virtual object based on the feature point information of the first person included in the first virtual object that reflects the body movement of the first person. An information processing apparatus comprising the same. (2) The adjustment unit changes the feature point information of the second person included in the second virtual object based on the feature point information of the first person included in the first virtual object. The information processing apparatus according to (1). (3) The feature point information includes at least any one of skeleton information, left-right information, and three-dimensional contour information. The information processing apparatus according to (2). (4) The adjustment unit adjusts at least the scale of the second virtual object based on the feature point information. The information processing apparatus according to any one of (1) to (3). (5) It further comprises a generation unit that generates a virtual object that reflects the body movement of the person based on the feature point information of the person. The information processing apparatus according to any one of (1) to (4). (6) The generation unit extracts the feature point information of the person based on the sensor data that senses the person. The information processing method according to (5). (7) The sensor data includes at least any one of ToF data, RGB data, and volumetric capture data The information processing apparatus according to (6). (8) The information processing apparatus further includes a sensor for sensing the person The information processing apparatus according to (6) or (7). (9) The generation unit extracts feature point information of the person based on the video in which the person appears The information processing apparatus according to (5). (10) The generation unit Based on the feature point information of the person and the RGB data sensing the person, removes the background of the person in the RGB data Generates the virtual object based on the RGB data with the background removed and the feature point information of the person The information processing apparatus according to any one of (5) to (9). (11) The generation unit generates a virtual object of a type corresponding to the purpose of the body movement of the person The information processing apparatus according to any one of (5) to (10). (12) The information processing apparatus further includes a video generation unit that generates a superimposed video by superimposing the first virtual object and the adjusted second virtual object The information processing apparatus according to any one of (1) to (11). (13) The information processing apparatus further includes an evaluation unit that generates an evaluation value of the first virtual object by comparing the first virtual object and the adjusted second virtual object The information processing apparatus according to (12). (14) The evaluation value includes at least any one of the difference in three-dimensional contour information, the difference in acceleration information, and the difference in predetermined fitting points between the first virtual object and the adjusted second virtual object The information processing apparatus according to (13). (15) Further comprising an effect generation unit that generates an effect video for the superimposed video based on the evaluation value The information processing apparatus according to (13) or (14). (16) The effect generation unit generates the effect video according to the purpose of the body movement of the first person The information processing apparatus according to (15). (17) Further comprising a display control unit that displays the superimposed video and the effect video on a display unit The information processing apparatus according to (15) or (16). (18) The display control unit switches the effect video displayed on the display unit according to the operation of the first person The information processing apparatus according to (17). (19) Further comprising an instruction information generation unit that generates instruction information for the second person to give a predetermined instruction to the body movement of the first person based on the evaluation value The information processing apparatus according to (17) or (18). (20) The display control unit further displays the instruction information on the display unit The information processing apparatus according to (19). (21) The virtual object includes the feature point information of a person and a 3D model based on the feature point information, The feature point information and the 3D model are transmitted via different network slices The information processing apparatus according to any one of (1) to (20). (22) The feature point information is transmitted via a low-latency network slice, The 3D model is transmitted via a large-capacity network slice The information processing apparatus according to (21). (23) The functional unit including the adjustment unit is realized by MEC (Mobile Edge Computing). The information processing apparatus according to any one of (1) to (22). (24) The second virtual object is managed based on the copyright information including the person ID that identifies the second person. The information processing apparatus according to any one of (1) to (23). (25) The information processing apparatus Based on the feature point information of the first person included in the first virtual object in which the body movement of the first person is reflected, by adjusting the second virtual object in which the body movement of the second person is reflected and that is superimposed on the first virtual object, a second virtual object after adjustment is generated. Information processing method. (26) Causing a computer Based on the feature point information of the first person included in the first virtual object in which the body movement of the first person is reflected, by adjusting the second virtual object in which the body movement of the second person is reflected and that is superimposed on the first virtual object, a second virtual object after adjustment is generated. A program for causing execution of processing.

Explanation of Signs

[0252] 10, 10TE, 10ST MEC server, 20 cloud server, 100 device, 150 sensor unit, 160 control unit, 161 digital twin generation unit, 162 instruction information generation unit, 200 device, 250 sensor unit, 260 control unit, 261 digital twin generation unit, 262 digital twin adjustment unit, 263 superimposed image generation unit, 264 evaluation unit, 265 effect generation unit, 266 display control unit, 310 storage device, 320 instruction information generation unit, 312 storage unit, 313 control unit, 1001 CPU

Claims

1. An adjustment unit that generates an adjusted second virtual object by adjusting a second virtual object reflecting the body movement of a second person, which is superimposed on the first virtual object, based on the feature point information of the first person included in the first virtual object reflecting the body movement of the first person comprising The virtual object includes feature point information of a person and a 3D model based on the feature point information The feature point information and the 3D model are transmitted via different network slices An information processing apparatus

2. The adjustment unit changes the feature point information of the second person included in the second virtual object based on the feature point information of the first person included in the first virtual object The information processing apparatus according to claim 1

3. The feature point information includes at least any one of skeleton information, left - right information, and 3D contour information The information processing apparatus according to claim 2

4. further comprising a generation unit that generates a virtual object reflecting the body movement of the person based on the feature point information of the person The information processing apparatus according to claim 2

5. The generation unit extracts the feature point information of the person based on the sensor data that senses the person The information processing apparatus according to claim 4

6. The sensor data includes at least any one of ToF data, RGB data, and volumetric capture data The information processing apparatus according to claim 5

7. The generation unit extracts the feature point information of the person based on the video in which the person appears The information processing apparatus according to claim 4

8. The generation unit generates a virtual object of a type corresponding to the purpose of the body movement of the person The information processing apparatus according to claim 4

9. further comprising a video generation unit that generates a superimposed video by superimposing the first virtual object and the adjusted second virtual object The information processing apparatus according to claim 1

10. further comprising an evaluation unit that generates an evaluation value of the first virtual object by comparing the first virtual object and the adjusted second virtual object The information processing apparatus according to claim 9

11. The evaluation value includes at least any one of the difference in three-dimensional contour information, the difference in acceleration information, and the difference in predetermined fitting points between the first virtual object and the adjusted second virtual object. The information processing apparatus according to claim 10.

12. The information processing apparatus further includes an effect generation unit that generates an effect video for the superimposed video based on the evaluation value. The information processing apparatus according to claim 10.

13. The information processing apparatus further includes a display control unit that displays the superimposed video and the effect video on a display unit. The information processing apparatus according to claim 12.

14. The information processing apparatus further includes an instruction information generation unit that generates instruction information for the second person to give a predetermined instruction to the body movement of the first person based on the evaluation value. The information processing apparatus according to claim 13.

15. The feature point information is transmitted via a low-latency network slice, The three-dimensional model is transmitted via a large-capacity network slice. The information processing apparatus according to claim 1.

16. The functional unit including the adjustment unit is realized by MEC (Mobile Edge Computing). The information processing apparatus according to claim 1.

17. The second virtual object is managed based on copyright information including a person ID that identifies the second person. The information processing apparatus according to claim 1.

18. An information processing apparatus, An information processing method for generating an adjusted second virtual object by adjusting a second virtual object that reflects the body movement of a second person and is superimposed on the first virtual object based on the feature point information of the first person included in the first virtual object that reflects the body movement of the first person. The virtual object includes the feature point information of a person and a three-dimensional model based on the feature point information. The feature point information and the three-dimensional model are transmitted via different network slices. Information processing method.

19. A computer, A program for causing the computer to execute a process of generating an adjusted second virtual object by adjusting a second virtual object that reflects the body movement of a second person and is superimposed on the first virtual object based on the feature point information of the first person included in the first virtual object that reflects the body movement of the first person. The virtual object includes the characteristic point information of a person and a 3D model based on the characteristic point information, and the characteristic point information and the 3D model are transmitted via different network slices program.

Images