Information processing method and information processing apparatus

Abstract

The present disclosure relates to an information processing method and an information processing apparatus. The present disclosure provides an information processing method and an information processing apparatus capable of improving visual recognition of a display in which a user and another person are superimposed. The information processing method includes: acquiring a first model linked to an action of a first user; acquiring a second model linked to an action of a second user; and displaying the first model and the second model at mutually different reference positions during a period in which the first user or the second user continuously performs a prescribed action.

Description

Technical Field

[0001] This disclosure relates to information processing methods and information processing apparatus. Background Technology

[0002] Methods for superimposing stereoscopic images of other people onto a user's body are known (e.g., see Patent Document 1).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2000-353252

[0006] When stereoscopic images of other people are superimposed on the user's body, the visual recognizability of the user's body or the stereoscopic images of other people may be reduced. Summary of the Invention

[0007] The purpose of this disclosure, which was made in view of such circumstances, is to improve the visual recognizability of displays that overlay users and others.

[0008] One embodiment of the information processing method disclosed herein includes: an information processing device acquiring a first model linked to the actions of a first user; the information processing device acquiring a second model linked to the actions of a second user; and the information processing device displaying the first model and the second model at different reference positions during a period when the first user or the second user continuously performs a predetermined action.

[0009] One embodiment of the information processing apparatus disclosed herein includes a control unit. The control unit is configured to: acquire a first model linked to the actions of a first user; acquire a second model linked to the actions of a second user; and display the first model and the second model at different reference positions while the first user or the second user continuously performs a predetermined action.

[0010] Invention Effects

[0011] According to one embodiment of the information processing method and information processing apparatus of this disclosure, the visual recognizability of a display superimposed with that of a user and others can be improved. Attached Figure Description

[0012] Figure 1 This is a block diagram illustrating an example of the configuration of an information processing system in one implementation.

[0013] Figure 2 This is a schematic diagram illustrating an example of a structure for a first user to practice calligraphy.

[0014] Figure 3 This is a flowchart illustrating an example of an information processing method in one implementation.

[0015] Explanation of reference numerals in the attached figures:

[0016] 1: Information Processing System

[0017] 10: First information processing device (11: Control unit; 12: Storage unit; 13: Interface; 14: Imaging unit)

[0018] 20: Second information processing device (21: Control unit; 22: Storage unit; 23: Interface; 24: Imaging unit)

[0019] 30: User (31: Hand; 32: Pen; 33: Paper; 34: Template)

[0020] 40-42, 50-53: Display image. Detailed Implementation

[0021] (Example of the structure of information processing system 1)

[0022] like Figure 1 As shown, one embodiment of the information processing system 1 includes a first information processing device 10 and a second information processing device 20. The first information processing device 10 and the second information processing device 20 are simply referred to as information processing devices unless further distinction is required. The information processing system 1 is designed to provide a service that allows students to learn by imitating actions demonstrated by an instructor. The objects of learning may include, for example, calligraphy, dance, pottery, or cooking. The information processing system 1 of this embodiment is designed to provide calligraphy lessons. The first information processing device 10 is used by students learning calligraphy while looking at calligraphy models. The second information processing device 20 is used by instructors demonstrating calligraphy models. Among the users of the information processing system 1, the student using the first information processing device 10 is also referred to as the first user. The instructor using the second information processing device 20 is also referred to as the second user.

[0023] The number of the first information processing device 10 and the second information processing device 20 in the information processing system 1 is not limited to one, and there can be two or more. The first information processing device 10 or the second information processing device 20 can be configured to include a PC such as a laptop PC, desktop PC, or tablet PC. The first information processing device 10 or the second information processing device 20 can also be configured to include a portable terminal such as a smartphone or tablet computer held by a user. The first information processing device 10 or the second information processing device 20 is not limited to these examples and can be configured to include a wide variety of devices.

[0024] The first information processing device 10 and the second information processing device 20 can be communicatively connected to each other via a network in a wired or wireless manner. Alternatively, the first information processing device 10 and the second information processing device 20 can also be communicatively connected to each other via wired or wireless communication without a network.

[0025] The first information processing device 10 includes a control unit 11, a storage unit 12, and an interface 13. The second information processing device 20 includes a control unit 21, a storage unit 22, and an interface 23. The control unit 11 is also referred to as the first control unit. The control unit 21 is also referred to as the second control unit. The interface 13 is also referred to as the first interface. The interface 23 is also referred to as the second interface.

[0026] Control unit 11 controls at least one component of the first information processing device 10. Control unit 21 controls at least one component of the second information processing device 20. Control unit 11 and control unit 21 may be configured to be the same or similar. Control unit 11 or control unit 21 may be configured to include more than one processor. In this embodiment, "processor" can be a general-purpose processor, a dedicated processor for a specific process, etc., but is not limited thereto. Control unit 11 or control unit 21 may also be configured to include more than one dedicated circuit. Dedicated circuits may include, for example, FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit). Control unit 11 or control unit 21 may be configured to include dedicated circuits instead of a processor, or may be configured to include both a processor and dedicated circuits.

[0027] Storage unit 12 stores arbitrary information for the operation of the first information processing device 10. Storage unit 22 stores arbitrary information for the operation of the second information processing device 20. Storage unit 12 or storage unit 22 can be configured to be the same or similar. Storage unit 12 or storage unit 22 can also store system programs or application programs, etc. Storage unit 12 or storage unit 22 can be configured to include, for example, semiconductor memory, magnetic memory, or optical memory, but is not limited thereto, and can also be configured to include various other types of memory. Storage unit 12 or storage unit 22 can function as, for example, a main storage device, an auxiliary storage device, or a cache memory. Storage unit 12 or storage unit 22 can also be configured to include electromagnetic storage media such as a magnetic disk. Storage unit 12 or storage unit 22 can also be configured to include a non-transitory computer-readable medium. Storage unit 12 can also be included in control unit 11. Storage unit 22 can also be included in control unit 21.

[0028] Interface 13 in the first information processing device 10 outputs or inputs information or data from the control unit 11. Interface 23 in the second information processing device 20 outputs or inputs information or data from the control unit 21. Interface 13 may also include a communication module configured to communicate with interface 23 of the second information processing device 20 via a network or without a network. Interface 23 may also include a communication module configured to communicate with interface 13 of the first information processing device 10 via a network or without a network. The communication module may, for example, correspond to mobile communication standards such as 4G (4th Generation) or 5G (5th Generation). The communication module may also correspond to communication standards such as LAN (Local Area Network). The communication module may also correspond to wired or wireless communication standards. The communication module is not limited to these and may correspond to a wide variety of communication standards. Interface 13 or interface 23 can also be configured to connect to an external communication module.

[0029] The network can be configured to include the Internet, at least one WAN (Wide Area Network), at least one MAN (Metropolitan Area Network), or any combination thereof. The network can also be configured to include at least one wireless network, at least one optical network, or any combination thereof. Wireless networks can be configured, for example, to include point-to-point networks, cellular networks, wireless LANs (Local Area Networks), satellite communication networks, or terrestrial microwave networks.

[0030] Interface 13 or Interface 23 can also be configured to correspond to standards such as USB (Universal Serial Bus) or Bluetooth (registered trademark).

[0031] Interface 13 or Interface 23 can be configured to include an input device that receives information or data from the user. The input device can be configured, for example, to include a touch panel or touch sensor, or a pointing device such as a mouse. The input device can also be configured to include physical keys. The input device can also be configured to include a voice input device such as a microphone. Interface 13 or Interface 23 can also be configured to connect to an external input device.

[0032] Interface 13 or Interface 23 can be configured to include an output device that outputs information or data to a user. The output device may include, for example, a display device that outputs visual information such as images, text, or graphics. The display device may be configured to include, for example, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, an inorganic EL display, or a PDP (Plasma Display Panel). The display device is not limited to these displays and may be configured to include various other types of displays. The display device may be configured to include light-emitting devices such as LEDs (Light Emitting Diodes) or LDs (Laser Diodes). The display device may be configured to include various other devices. The output device may include, for example, a sound output device such as a speaker that outputs auditory information such as sound. The output device may include, for example, a vibrator that outputs tactile information such as vibration. The output device is not limited to these examples and may include various other devices. Interface 13 or Interface 23 can also be configured to connect to an external output device. The output device may be configured, for example, like a wristband, for a first or second user to wear. Output devices, such as pens used in calligraphy or knives or cutting boards used in cooking, can be configured as props for use by a first or second user when practicing movements.

[0033] Although not strictly necessary, the first information processing device 10 also includes an imaging unit 14. Similarly, the second information processing device 20 also includes an imaging unit 24. Imaging unit 14 is also referred to as the first imaging unit. Imaging unit 24 is also referred to as the second imaging unit. Imaging unit 14 or imaging unit 24 may include imaging devices such as a camera that captures RGB images. Imaging unit 14 or imaging unit 24 may include ranging devices such as a depth sensor or a stereo camera that acquires depth images. The first information processing device 10 or the second information processing device 20 may also acquire images or ranging data (depth data) from external imaging devices or ranging devices via interface 13 or interface 23. The functions of the imaging device or ranging device in imaging unit 14 or imaging unit 24 may also be included in the functions of interface 13 or interface 23.

[0034] (An example of an action in Information Processing System 1)

[0035] Hereinafter, in the information processing system 1 of this embodiment, an example of a first user practicing calligraphy while watching a demonstration by a second user will be described.

[0036] <Acquisition of Action Information>

[0037] The control unit 11 of the first information processing device 10 acquires information related to the actions of the first user. This information related to the actions of the first user is also referred to as first action information. Let's say the first user's actions include the action of writing characters through calligraphy practice. That is, let's say the first action information includes information related to the actions of the first user practicing calligraphy as a student. The control unit 21 of the second information processing device 20 acquires information related to the actions of the second user. This information related to the actions of the second user is also referred to as second action information. Let's say the second user's actions include the action of writing characters as a calligraphy model. That is, let's say the second action information includes information related to the actions of the second user demonstrating a calligraphy model as an instructor.

[0038] Control unit 11 can acquire RGB images or depth images obtained by the imaging unit 14 of the first information processing device 10 from a first user, and use them as first action information. Control unit 21 can acquire RGB images or depth images obtained by the imaging unit 24 of the second information processing device 20 from a second user, and use them as second action information. Control unit 11 or control unit 21 can acquire RGB images or depth images from external imaging devices or ranging devices via interface 13 or interface 23.

[0039] <Model Generation>

[0040] The control unit 11 of the first information processing device 10 generates a first model that is linked to the actions of the first user based on the first motion information. The control unit 11 can extract the skeletal information of the first user from the first motion information and generate the first model based on the skeletal information. The control unit 11 can generate the first model based on the skeletal information of the first user and an RGB image.

[0041] The control unit 21 of the second information processing device 20 generates a second model that is linked to the actions of the second user based on the second motion information. The control unit 21 can extract the skeletal information of the second user from the second motion information and generate the second model based on the skeletal information. The control unit 21 can generate the second model based on the skeletal information of the second user and an RGB image.

[0042] Control unit 11 or control unit 21 can generate a first model or a second model as a 3D model.

[0043] Skeletal information may include information indicating the location of joints corresponding to a person's joints. Skeletal information may also include information indicating the location and orientation of the bones connecting the joints. Control unit 11 may extract the skeletal information of the first user based on the depth image of the first user. Control unit 21 may extract the skeletal information of the second user based on the depth image of the second user.

[0044] <Model Display>

[0045] The control unit 11 of the first information processing device 10 displays the first model and the second model superimposed on a display device serving as an interface 13. The control unit 11 can acquire the second model from the second information processing device 20. The control unit 11 can also generate the second model by acquiring second action information from the second information processing device 20. The control unit 21 of the second information processing device 20 can display the first model and the second model superimposed on a display device serving as an interface 23. The control unit 21 can acquire the first model from the first information processing device 10. The control unit 21 can also generate the first model by acquiring first action information from the first information processing device 10.

[0046] The first user practices calligraphy while viewing a display created by superimposing the first and second models. For example, as shown... Figure 2 As shown, user 30, corresponding to the first user as a student, can practice calligraphy while looking at the display on the display device, which is interface 13. Figure 2 In one scenario, user 30 holds a pen 32 in their hand 31 and writes on paper 33 while looking at a template 34. Meanwhile, a second user, assuming the role of a lecturer, writes in another setting to demonstrate calligraphy templates.

[0047] The control unit 11 can acquire first action information related to the user 30 as the first user, generate a first model, and display the display image 40 of the first model on the interface 13. The display image 40 of the first model includes a display image 41 corresponding to the user 30's hand 31 and a display image 42 corresponding to the pen 32. The control unit 11 configures the first model in accordance with a reference position set in the virtual space and displays the display image 40 as an image obtained by observing the virtual space from a predetermined viewpoint.

[0048] On the other hand, the control unit 21 can acquire second action information related to the second user, generate a second model, and output it to the first information processing device 10. The control unit 11 acquires the second model from the second information processing device 20 and displays the display image 50 of the second model on the interface 13. The display image 50 of the second model includes a display image 51 corresponding to the second user's hand and a display image 52 corresponding to the pen. The control unit 11 configures the second model in accordance with a reference position set in the virtual space and displays the display image 50 as an image obtained by observing the virtual space from a predetermined viewpoint.

[0049] The control unit 11 can display the display image 40 of the first model or the display image 50 of the second model as a stereoscopic image obtained by compositing an RGB image and a depth image on the interface 13, or it can display only the RGB image. The control unit 11 can also generate polygons of the first model or the second model based on the depth image and draw the RGB image onto the polygons, thereby generating a stereoscopic image.

[0050] Control unit 11 Figure 2 In interface 13, each display image is shown as a mirror image (reversed horizontally). This makes it easy for the first user to imitate the actions of the second user. The control unit 11 can also display each display image as a non-reversed image.

[0051] Here, with the first user's display image 40 and the second user's display image 50 completely overlapping, it is difficult for the first user to visually recognize the second user's actions. Therefore, the control unit 11 can make the reference position set as the origin of the coordinate system when the first model is placed in the virtual space and the reference position set as the origin of the coordinate system when the second model is placed in the virtual space different from each other. The reference position set as the origin of the coordinate system when the first model is placed in the virtual space is also called the first reference position. The reference position set as the origin of the coordinate system when the second model is placed in the virtual space is also called the second reference position. That is, the control unit 11 can make the first reference position and the second reference position different from each other. By making the reference positions different from each other, the display image 40 of the first model and the display image 50 of the second model can be displayed alternately in the interface 13.

[0052] To facilitate observation by the first user of the second user's demonstration, the control unit 11 displays the hands of the first and second users in a manner that is slightly offset from each other and does not completely overlap. For example, the control unit 11 can differentiate between the first and second reference positions during the period when the second user is demonstrating the action, or during the period when the first user is practicing the action by imitating the demonstration. In other words, the control unit 11 can differentiate between the first and second reference positions during the period when either the first or second user continuously performs a prescribed action. The prescribed action may include the action of demonstrating calligraphy performed by the second user. The prescribed action may also include the action of practicing calligraphy performed by the first user.

[0053] The control unit 11 can determine whether the first user is continuously performing a prescribed action based on the first action information. For example, the control unit 11 can compare the joint points or bone movements of the first user's skeletal information with the joint points or bone movements in the prescribed action to determine whether the first user is continuously performing the prescribed action. The control unit 11 can also determine whether the first user is continuously performing the prescribed action based on the first user's image. The control unit 11 can compare the first user's image with the image of the prescribed action to determine whether the first user is continuously performing the prescribed action. The control unit 11 can determine that the first user is continuously performing the prescribed action when the first user records information such as text on a medium such as paper. The control unit 11 can determine that the first user is continuously performing the prescribed action when the writing tool such as a pen held by the first user is in contact with the medium such as paper.

[0054] Control unit 11 can acquire second action information from second information processing device 20 and determine whether the second user is continuously performing a prescribed action based on the second action information. Control unit 11 can also determine whether the second user is continuously performing a prescribed action based on the image of the second user. Control unit 11 can determine whether the second user is continuously performing a prescribed action in the same way as it does with the first user. Control unit 11 can also acquire a determination result from second information processing device 20 regarding whether the second user is continuously performing a prescribed action. In this case, control unit 21 can determine whether the second user is continuously performing a prescribed action based on the second action information. Control unit 21 can determine whether the second user is continuously performing a prescribed action in the same way as control unit 11.

[0055] The control unit 11 can differentiate between the first and second reference positions in a way that prevents the parts characteristic of a given action from overlapping. For example, in a calligraphy action characterized by the movement of the right hand holding the pen, the second reference position can be set such that the display image 51 of the second user's right hand moves to the right on the screen compared to the display image 41 of the first user's right hand. Conversely, if the display image 51 of the second user's right hand moves to the left, the display image 51 of the second user's right hand may be hidden by the torso of the display image 40 of the first user. By determining the direction in which the reference position is moved, the display image 51 of the second user's right hand is less likely to be hidden by the display image 40 of the first user.

[0056] As described above, by making the first reference position and the second reference position different during the period when the first user or the second user continuously performs the prescribed action, the visual recognition of the second user's action by the first user can be improved.

[0057] The control unit 11 can set the first reference position and the second reference position to the same position even when the first user and the second user are not continuously performing the prescribed actions. This makes it easier for the first user to match their posture with that of the second user. The control unit 11 can also save the actions of the second user's display image 50 in a way that allows for later display reproduction, even when the second user's display image 50 is hidden by the first user's display image 40. Similarly, the control unit 11 can save the actions of the first user's display image 40 in a way that allows for later display reproduction, even when the second user's display image 40 is hidden by the first user's display image 40. This allows each user to visually distinguish the differences in their actions from those of the other user later.

[0058] The control unit 11 can also acquire information related to the text written on paper by the second user and display it as display image 53 on the interface 13. The text written on paper by the second user can be obtained as a result of the second user's action. The information related to the text written on paper by the second user is also referred to as the result of the second user's action. In other words, the action of the second user writing on paper is an action of recording information such as text on a medium such as paper. When the specified action is an action of recording information on a medium, the control unit 11 can display the information recorded on the medium by the second user (the text written on paper) on the interface 13 in a way that the first user can visually recognize. In this way, the first user can confirm the text written by the second user, who is acting as a lecturer. The control unit 11 can... Figure 2 As illustrated in the example, the text representing the result of the second user's action is displayed in a manner corresponding to the location where the second user actually performed the action. The control unit 11 can display the text representing the result of the second user's action on the interface 13 in a frontal view. The control unit 11 can also display or project an image of the text representing the result of the second user's action onto the paper 33 where the first user writes. The control unit 11 can display the text representing the result of the second user's action on the interface 13 in various formats.

[0059] <Example of an information processing method>

[0060] As described above, in the information processing system 1 of this embodiment, the control unit 11 of the first information processing device 10 displays the actions of the second user superimposed on the actions of the first user, so that the first user can imitate the demonstration of the second user to practice. The control unit 11 of the first information processing device 10 may, for example, perform actions including... Figure 3 The flowchart illustrated in the example illustrates an information processing method. This information processing method can also be implemented as an information processing program executed by the control unit 11. The information processing program can be stored on a non-transitory computer-readable medium.

[0061] The control unit 11 acquires first action information (step S1). The control unit 11 generates a first model based on the first action information (step S2). The control unit 11 acquires a second model from the second information processing device 20 (step S3). The control unit 11 determines whether the first user or the second user is continuously performing a prescribed action (step S4).

[0062] When the first user or the second user continuously performs the prescribed action (step S4: Yes), the control unit 11 displays the first model and the second model at different reference positions (step S5). The control unit 11 displays the action result obtained by the second user continuously performing the prescribed action (step S6). After the process in step S6 is completed, the control unit 11 terminates. Figure 3 The execution of the flowchart process.

[0063] If the first user or the second user does not continuously perform the prescribed action (step S4: No), the control unit 11 displays the first model and the second model at the same reference position (step S7). After the process in step S7 is completed, the control unit 11 terminates. Figure 3 The execution of the flowchart process.

[0064] Control unit 11 can be repeatedly executed Figure 3 The process of the flowchart is used to continuously perform and display the actions of the first model and the second model. The control unit 11 may also obtain second action information from the second information processing device 20 to generate the second model during step S3. The control unit 11 may also obtain a determination result from the second information processing device 20 regarding whether the second user is continuously performing a prescribed action during the determination process in step S4. The control unit 11 may also save the actions of the first user's display image 40 or the second user's display image 50 in a manner that allows for future reproduction of the display after the display process in step S7.

[0065] <Summary>

[0066] As described above, according to the information processing system 1 of this embodiment, during the period when either the first user or the second user continuously performs a predetermined action, the first user and the second user are displayed at different reference positions. This improves the visual recognition of the second user's actions when the first user is practicing the predetermined action. As a result, the practice efficiency of the first user is improved.

[0067] (Other implementation methods)

[0068] Hereinafter, an example of the configuration of information processing system 1 in another embodiment will be described.

[0069] The control unit 11 of the first information processing device 10 can display images with the transparency of either the first model display image 40 or the second model display image 50 at a predetermined value or higher during the period when either the first user or the second user continuously performs a prescribed action. For example, the control unit 11 can set the transparency of the first model display image 40 to a predetermined value or higher and display it superimposed on the second model display image 50. In this way, the first user can easily identify both the first user's own action and the second user's action. That is, it can improve the visual recognition of the second user's action when the first user practices the prescribed action. The control unit 11 can also set the transparency of the second model display image 50 to a predetermined value or higher and display it superimposed on the first model display image 40 during the period when the second user demonstrates the action.

[0070] As described above, the information processing system 1 can be used not only for calligraphy practice, but also for practicing various movements such as dance movements, pottery movements, and cooking movements. When the information processing system 1 is used for dance practice, the prescribed movements correspond to the movements of different parts of the body or the movements of props used in the dance. When the information processing system 1 is used for pottery practice, the prescribed movements correspond to the movements of the hands in pottery making. The result of the movement corresponds to the shaping of the clay. When the information processing system 1 is used for cooking practice, the prescribed movements correspond to the movements of the hands or props such as knives in cooking. The result of the movement corresponds to the state of the processed ingredients.

[0071] The information processing system 1 can also enable one of the first information processing device 10 or the second information processing device 20 to function as a host server, and the other to function as a client server.

[0072] The embodiments of this disclosure have been described with reference to the accompanying drawings and examples. However, it should be noted that those skilled in the art can make various modifications and changes based on this disclosure. Therefore, it should be understood that these modifications and changes are included within the scope of this disclosure. For example, the functions included in each unit or step can be reconfigured in a logically consistent manner, and multiple units or steps can be combined into one or divided.

Claims

1. An information processing method, executed by an information processing device, comprising: Obtain the first model that is linked to the actions of the first user; Obtain the second model that is linked to the actions of the second user; The determination is made based on the image of the first user whether the first user is continuously performing the prescribed action, or based on the image of the second user whether the second user is continuously performing the prescribed action. During the period when the first user or the second user continuously performs the prescribed action, the first model and the second model are displayed at staggered reference positions. as well as If neither the first user nor the second user continues to perform the prescribed action, the first model and the second model are displayed at the same reference position. The reference position refers to the position set as the origin of the coordinate system when the model is configured in virtual space.

2. The information processing method according to claim 1 further includes: When the specified action is to record information on a medium, the information recorded by the second user on the medium is displayed in a manner that the first user can visually recognize.

3. The information processing method according to claim 1, comprising: The actions of the first model or the second model during the period when the first user and the second user did not continuously perform the specified actions are saved in a manner that can reproduce the display.

4. An information processing device comprising a control unit, wherein, The control unit is configured to: Obtain the first model that is linked to the actions of the first user; Obtain the second model that is linked to the actions of the second user; The determination is made based on the image of the first user whether the first user is continuously performing the prescribed action, or based on the image of the second user whether the second user is continuously performing the prescribed action. During the period when the first user or the second user continuously performs the prescribed action, the first model and the second model are displayed at staggered reference positions. as well as If neither the first user nor the second user continues to perform the prescribed action, the first model and the second model are displayed at the same reference position. The reference position refers to the position set as the origin of the coordinate system when the model is configured in virtual space.

Images