Information processing device, information processing system, and information processing method
The system efficiently reflects users' situations on avatars in virtual spaces by prioritizing specific body parts for motion and skeletal information, addressing the inefficiencies of existing systems with reduced data requirements and enhanced realism.
Patent Information
- Application Number
- JP2023080228
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2024-11-27
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing systems require large-scale equipment and significant data processing to reflect multiple users' real-time situations in avatars within a three-dimensional virtual space, leading to inefficiencies and high data demands.
An information processing device and system that acquires motion and image information from a user, prioritizing specific body parts for reflection on avatars, using a combination of wide-angle and telephoto imaging modes to generate and transmit motion and skeletal information, reducing the need for full image transmission.
Facilitates easy reflection of users' situations on avatars in virtual spaces with reduced data communication requirements, enhancing realism and efficiency by focusing on high-priority body parts like the face, thus minimizing data transmission.
Smart Images

Figure 2024164617000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device, an information processing system, and an information processing method. [Background technology]
[0002] There is known a technique for generating a three-dimensional model based on an image of a user. For example, Patent Document 1 discloses a technique in which a user is displayed as if transformed into a character and motion capture is performed. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2022-123941 A Summary of the Invention [Problem to be solved by the invention]
[0004] Recently, there has been an increase in cases where multiple users communicate through avatars in a three-dimensional virtual space. In such cases, when multiple users communicate in a virtual space through avatars, a large-scale device and a huge amount of data may be required to reflect the real-time situation of each user in the avatar.
[0005] An object of the present invention is to provide an information processing device, an information processing system, and an information processing method that can easily reflect a user's situation in an avatar in a virtual space. [Means for solving the problem]
[0006] The information processing device of the present invention comprises a target information acquisition unit that acquires movement information indicating the movement of a target, an image acquisition unit that acquires an image of a part of the target identified based on a priority, and an image processing unit that reflects the movement information acquired by the target information acquisition unit and the image acquired by the image acquisition unit in a virtual space in a pre-generated avatar of the target.
[0007] The information processing system of the present invention includes an information processing device of the present invention and at least one imaging device that photographs the object, wherein the imaging device is equipped with an image control unit that controls the imaging unit to capture an image of the entire object and an image of a portion of the object identified based on a priority, a motion information generation unit that generates motion information of the object based on the image of the entire object, and a communication control unit that transmits the motion information of the object and the image of the identified portion of the object to the information processing device.
[0008] The information processing method of the present invention includes the steps of acquiring motion information indicating the motion of an object, acquiring an image capturing a portion of the object identified based on a priority, and reflecting the motion information and the image in a virtual space in an avatar of the object that has been generated in advance. Effect of the Invention
[0009] According to the present invention, the user's situation can be easily reflected in an avatar in a virtual space. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an information processing system according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of the imaging device according to the first embodiment. [Diagram 3] FIG. 3 is a diagram for explaining a method for capturing an image of a target according to the first embodiment. [Figure 4]FIG. 4 is a block diagram showing the configuration of the information processing device according to the first embodiment. [Diagram 5] FIG. 5 is a flowchart showing the flow of processing in the imaging device according to the first embodiment. [Figure 6] FIG. 6 is a flowchart showing the process of the information processing device according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to this embodiment, and in the following embodiments, the same components are designated by the same reference numerals, and duplicated explanations will be omitted.
[0012] [First embodiment] (Information Processing System) An example of the configuration of an information processing system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of an information processing system according to the first embodiment.
[0013] 1, the information processing system 1 includes at least one imaging device 10 and an information processing device 12. The imaging device 10 and the information processing device 12 are connected via a network N so as to be able to communicate with each other.
[0014] The information processing system 1 provides a virtual space. The information processing system 1 generates, for example, an avatar of a target in the virtual space. The information processing system 1 reflects, for example, a real-time situation of the target in an avatar of the target that is generated in advance in the virtual space. Specifically, the imaging device 10 captures an image of the target, generates motion information, skeletal information, and an image of a high-priority part of the target, and transmits the generated information to the information processing device 12. The information processing device 12 updates the avatar of the target that is generated in advance based on the motion information, skeletal information, and an image of a high-priority part of the target acquired from the imaging device 10.
[0015] (Imaging device) An example of the configuration of the imaging device according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of the imaging device according to the first embodiment.
[0016] As shown in FIG. 2, the imaging device 10 includes an imaging section 20, an input section 22, a display section 24, a storage section 26, a communication section 28, and a control section 30.
[0017] The imaging unit 20 captures an image of an object. For example, the imaging unit 20 captures an image of the object to generate a still image or a moving image. For example, the imaging unit 20 is a camera including an optical element and an imaging element. For example, the imaging unit 20 is a visible light camera or an infrared camera. In this embodiment, for example, the imaging unit 20 captures an image of the entire object at a wide angle in wide-angle mode. For example, the imaging unit 20 captures an image of a high-priority part of the object by zooming in in telephoto mode. For example, the imaging unit 20 captures an image of the entire object and a high-priority part simultaneously.
[0018] The input unit 22 accepts various operations for the imaging device 10. The input unit 22 is realized by a microphone, a switch, a button, a touch panel, or the like.
[0019] The display unit 24 displays various images. The display unit 24 is, for example, a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like.
[0020] The storage unit 26 stores various types of information. The storage unit 26 stores information such as the contents of calculations performed by the control unit 30 and programs. The storage unit 26 includes at least one of a RAM (Random Access Memory), a main storage device such as a ROM (Read Only Memory), and an external storage device such as a HDD (Hard Disk Drive).
[0021] The communication unit 28 is a communication interface that executes communication between the imaging device 10 and an external device. The communication unit 28 executes communication between the imaging device 10 and the information processing device 12, for example.
[0022] The control unit 30 controls each unit of the imaging device 10. The control unit 30 has, for example, an information processing device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and a storage device such as a RAM or a ROM. The control unit 30 may be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 30 may be realized by a combination of hardware and software.
[0023] The control unit 30 includes an imaging control unit 40, a movement information generating unit 42, a skeleton information generating unit 44, a priority determining unit 46, and a communication control unit 48.
[0024] The imaging control unit 40 controls the imaging unit 20 to capture an image of the target and acquire an image of the target. FIG. 3 is a diagram for explaining a method of capturing an image of a target according to the first embodiment. As shown in FIG. 3, the imaging control unit 40 controls, for example, the imaging unit 20 to capture an image of a user U who is a target. The imaging control unit 40 acquires the image of the user U captured by the imaging unit 20.
[0025] The imaging control unit 40, for example, controls the imaging unit 20 to a wide-angle mode to capture an image of the entire user U. The imaging control unit 40, for example, controls the imaging unit 20 to a wide-angle mode to capture an image of the entire user U, including buildings and objects around the user U.
[0026] The imaging control unit 40, for example, controls the imaging unit 20 to a telephoto mode to increase the zoom magnification and focus on and image (zoom image) a specific part of the user U. The imaging control unit 40, for example, controls the imaging unit 20 to a telephoto mode to zoom image a high priority part of the user U. The imaging control unit 40, for example, controls the imaging unit 20 to a telephoto mode to zoom image the face region R1.
[0027] The motion information generating unit 42 generates motion information indicating the motion of the target based on the image of the target acquired by the imaging control unit 40. For example, the motion information generating unit 42 detects a motion vector of the user U using the entire image of the user U acquired by the imaging control unit 40, and generates motion information of the user U based on the detected motion vector. For example, the motion information generating unit 42 may detect the motion vector using the current frame of the image of the user U and the previous frame.
[0028] The skeleton information generating unit 44 generates skeleton information (skeletal data) indicating the skeleton of the target based on the image of the target acquired by the imaging control unit 40. The skeleton information generating unit 44 generates the skeleton information of the user U, for example, based on the whole image of the user U, using a skeleton estimation technique.
[0029] The priority determination unit 46 determines which of the parts of the target has a high priority based on the image of the target acquired by the imaging control unit 40. For example, in the example shown in FIG. 3, the priority determination unit 46 determines that the face area R1 of the user U is the part with a high priority. The part with a high priority may be determined in advance. There may be one or more parts with a high priority. The part with a high priority is, for example, a part of the avatar of the user U in the virtual space in which the situation of the user U is desired to be expressed in more detail. The part with a high priority may be set arbitrarily by the user of the information processing system 1.
[0030] The communication control unit 48 controls the communication unit 28 to transmit and receive various information to and from the information processing device 12. The communication control unit 48 controls the communication unit 28 to transmit the target movement information, skeletal information, and an image of a high priority part to the information processing device 12. The communication control unit 48 transmits, for example, the movement information of the user U, skeletal information, and an image of the face region R1 to the information processing device 12.
[0031] (Information processing device) An example of the configuration of the information processing device according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the information processing device according to the first embodiment.
[0032] As shown in FIG. 4, the information processing device 12 includes a storage unit 50, a communication unit 52, and a control unit .
[0033] The storage unit 50 stores various types of information. The storage unit 50 stores information such as the contents of calculations performed by the control unit 54 and programs. The storage unit 50 includes at least one of a RAM, a main storage device such as a ROM, and an external storage device such as a HDD.
[0034] The storage unit 50 stores various information necessary for generating a virtual space. For example, the storage unit 50 stores information on a target avatar and various virtual objects generated in advance. That is, the storage unit 50 stores asset data for generating a virtual space. The target avatar and virtual objects may be generated in advance using a well-known method. Specifically, the target avatar and virtual object may be generated using a well-known method based on an image of the target and an image of a real object, respectively.
[0035] The communication unit 52 is a communication interface that executes communication between the information processing device 12 and an external device. The communication unit 52 executes communication between the information processing device 12 and the imaging device 10, for example.
[0036] The control unit 54 controls each unit of the information processing device 12. The control unit 54 has, for example, an information processing device such as a CPU or an MPU, and a storage device such as a RAM or a ROM. The control unit 54 may be realized by, for example, an integrated circuit such as an ASIC or an FPGA. The control unit 54 may be realized by a combination of hardware and software.
[0037] The control unit 54 includes a virtual space generation unit 60 , a target information acquisition unit 62 , an image acquisition unit 64 , and an image processing unit 66 .
[0038] The virtual space generation unit 60 generates a virtual space including a target avatar and various virtual objects. That is, the virtual space generation unit 60 generates the virtual space using asset data stored in the storage unit 50.
[0039] The target information acquisition unit 62 acquires target information indicating a real-time situation of the target. The target information acquisition unit 62 acquires motion information of the target from the imaging device 10 via the communication unit 52. The target information acquisition unit 62 acquires skeletal information of the target from the imaging device 10 via the communication unit 52. The target information acquisition unit 62 acquires, for example, motion information and skeletal information of the user U. The target information acquisition unit 62 may acquire only one of the motion information and the skeletal information, or may acquire both.
[0040] The image acquisition unit 64 acquires an image of the target. For example, the image acquisition unit 64 acquires an image of a part of the target that is specified based on the priority. For example, the image acquisition unit 64 acquires an image of a part of the target that has a high priority. For example, the image acquisition unit 64 acquires an image of the face region R1 of the user U from the imaging device 10 via the communication unit 52.
[0041] The image processing unit 66 executes various image processing on the avatar and the virtual object in the virtual space. The image processing unit 66 reflects the motion information of the target acquired by the target information acquisition unit 62 on the avatar of the target. The image processing unit 66 reflects the skeletal information of the target acquired by the target information acquisition unit 62 on the avatar of the target. The image processing unit 66 reflects the image of the target acquired by the image acquisition unit 64 on the avatar of the target. For example, the image processing unit 66 reflects the motion information and skeletal information of the user U acquired by the target information acquisition unit 62 on the avatar of the user U. For example, the image processing unit 66 reflects the image of the face region R1 of the user U acquired by the image acquisition unit 64 on the avatar of the user U. That is, in the virtual space, the motion of the avatar of the user U is reflected based on the motion of the user U, etc. A part with high priority, such as the face of the avatar, is reflected based on the face image of the user U, etc. The image processing unit 66 reflects different information for each part of the target on the avatar in the virtual space based on the priority.
[0042] (Processing of imaging device) Processing in the imaging device according to the first embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the flow of processing in the imaging device according to the first embodiment.
[0043] The process shown in FIG. 5 is executed by the imaging device in order to reflect the real-time appearance of the target in the avatar of the target generated in the virtual space.
[0044] The control unit 30 determines whether the subject of the image capture is an animal including a human (step S10). Here, the subject of the image capture is assumed to be a user U shown in FIG. 3. If the subject is determined to be an animal (step S10; Yes), the process proceeds to step S12. If the subject is not determined to be an animal (step S10; No), the process of FIG. 5 ends.
[0045] If the result of the determination in step S10 is Yes, the priority determination unit 46 determines a high priority part of the target (step S12). Specifically, the priority determination unit 46 determines, for example, the face area R1 as a high priority part. Then, the process proceeds to step S14.
[0046] The imaging control unit 40 controls the imaging unit 20 to capture an image of the entire target (step S14). Specifically, the imaging control unit 40 controls the imaging unit 20 to a wide-angle mode to capture an image of the entire user U. Then, the process proceeds to step S16.
[0047] The imaging control unit 40 controls the imaging unit 20 to capture an image of a high priority part of the target (step S16). Specifically, the imaging control unit 40 controls the imaging unit 20 to a telephoto mode and captures an image of the face area R1 of the user U by zooming in. Then, the process proceeds to step S18.
[0048] The motion information generating unit 42 generates motion information of the target based on the entire image of the target (step S18). Specifically, the motion information generating unit 42 generates motion information of the user U by detecting a motion vector of the user U based on the entire image of the user U. Then, the process proceeds to step S20.
[0049] The skeleton information generating unit 44 generates skeleton information of the target based on the overall image of the target (step S20). Specifically, the skeleton information generating unit 44 estimates the skeleton of the user U based on the overall image of the user U and generates skeleton information. Then, the process proceeds to step S22.
[0050] The communication control unit 48 transmits the movement information generated by the movement information generation unit 42, the skeleton information generated by the skeleton information generation unit 44, and an image of a high priority part of the target to the information processing device 12 via the communication unit 28 (step S22). Specifically, the communication control unit 48 transmits, for example, the movement information of the user U, the skeleton information, and an image of the face region R1 to the information processing device 12. Then, the process of FIG. 5 ends.
[0051] (Processing of information processing device) The processing of the information processing device according to the first embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the processing of the information processing device according to the first embodiment.
[0052] The virtual space generating unit 60 generates a virtual space including a target avatar and a virtual object (step S30). The avatar is, for example, a three-dimensional virtual object such as a character generated to resemble a user U. For example, the avatar corresponding to each user can move freely in the virtual space. Each user can perform various activities in the virtual space through each avatar of each user. For example, in the virtual space, users can communicate with each other by text chat or voice chat. In addition, each user can participate in an event held in the virtual space. Events in the virtual space include, for example, games, concerts, etc. In the virtual space, a virtual reality including the activities of each user progresses. Then, the process proceeds to step S32.
[0053] The target information acquisition unit 62 acquires target information of the target (step S32). Specifically, the target information acquisition unit 62 acquires motion information of the user U from the imaging device 10, for example, via the communication unit 52. The target information acquisition unit 62 acquires skeletal information of the user U from the imaging device 10, for example, via the communication unit 52. Then, the process proceeds to step S34.
[0054] The image acquisition unit 64 acquires an image of the high priority part of the target (step S34). Specifically, the image acquisition unit 64 acquires an image of the face region R1 of the user U from the imaging device 10 via, for example, the communication unit 52. Then, the process proceeds to step S36.
[0055] The image processing unit 66 reflects the target information acquired by the target information acquisition unit 62 in the target avatar in the virtual space (step S36). Specifically, the image processing unit 66 reflects, for example, the movement information of the user U acquired by the target information acquisition unit 62 in the avatar of the user U in the virtual space. The image processing unit 66 reflects, for example, the skeletal information of the user U acquired by the target information acquisition unit 62 in the avatar of the user U in the virtual space. Then, the process proceeds to step S38.
[0056] The image processing unit 66 reflects the image of the high priority part acquired by the image acquisition unit 64 in the target avatar in the virtual space (step S38). Specifically, the image processing unit 66 reflects, for example, the image of the face region R1 of the user U acquired by the image acquisition unit 64 in the avatar of the user U in the virtual space. Then, the processing of FIG. 6 ends.
[0057] As described above, in the first embodiment, when reflecting the real-time situation of a target in an avatar in a virtual space, an image of the target as a whole is not transmitted, but only the movement information and skeletal information of the target is transmitted. Then, in the first embodiment, the movement of the target avatar in the virtual space is reflected in real-time based on the movement information and skeletal information. In this way, the first embodiment can easily reflect the user's situation in the avatar in the virtual space.
[0058] In addition, in the first embodiment, the avatar's movement in the virtual space is reflected in the user's state based on the user's movement information and skeletal information. This allows the first embodiment to reflect the user's movement in the avatar's movement without being based on an image. This allows the first embodiment to reduce the amount of communication data.
[0059] Furthermore, in the first embodiment, when reflecting the real-time situation of the target in the avatar of the target in the virtual space, an image is transmitted for the high priority part of the target. Then, in the first embodiment, the real-time situation is reflected based on the image for the high priority part such as the face of the target avatar in the virtual space. In this way, the first embodiment can reflect the detailed appearance based on the image for the high priority part such as the face in the avatar. For example, in the virtual space, the face of the avatar can realize a more realistic expression of the original user of the avatar.
[0060] [Variations] A modified example of the first embodiment will be described below. In the first embodiment, an image of the user U is captured in a wide-angle mode and a telephoto mode by a single imaging device 10, but the present invention is not limited to this.
[0061] For example, in a modification of the first embodiment, the user U may be captured by a plurality of imaging devices, including an imaging device 10 that captures the entire user U in wide-angle mode and an imaging device 10 that captures a high-priority part of the user U in telephoto mode. In the modification of the first embodiment, the processing load of each imaging device 10 can be reduced by using an imaging device 10 dedicated to the wide-angle mode and an imaging device 10 dedicated to the telephoto mode.
[0062] Each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various additions and usage conditions. This distribution and integration configuration may be performed dynamically.
[0063] Although the embodiments of the present invention have been described above, the present invention is not limited to the contents of these embodiments. The above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments. [Explanation of symbols]
[0064] 1. Information Processing Systems 10. Imaging device 12 Information processing equipment 20 Imaging unit 22 Input section 24 Display section 26,50 Storage part 28,52 Communications Department 30,54 Control section 40 Imaging control section 42 Motion information generation unit 44 Skeleton information generation unit 46 Priority judgment section 48 Communication control section 60 Virtual Space Generation Section 62 Target information acquisition unit 64 Image acquisition unit 66 Image Processing Unit
Claims
1. An object information acquisition unit that acquires motion information indicating a motion of an object; an image acquisition unit that acquires an image of a portion of the target that is identified based on the priority; an image processing unit that reflects the movement information acquired by the target information acquisition unit and the image acquired by the image acquisition unit in a virtual space on an avatar of the target that has been generated in advance; An information processing device comprising:
2. The object information acquisition unit acquires skeletal information of the object. The information processing device according to claim 1 .
3. the identified portion is a portion having a priority higher than a predetermined level and includes a face of the subject; 3. The information processing device according to claim 1 or 2.
4. The information processing device according to claim 1 ; At least one imaging device that captures an image of the object; The imaging device includes: an image control unit that controls an imaging unit to capture an image of the entire object and an image of a portion of the object that is specified based on a priority; a motion information generating unit that generates motion information of the object based on an entire image of the object; a communication control unit that transmits the motion information of the object and an image of the specified part of the object to the information processing device, Information processing system.
5. acquiring motion information indicative of motion of the object; acquiring an image of a portion of the target identified based on the priority; In a virtual space, the motion information and the image are reflected in a pre-generated avatar of the target; An information processing method comprising:
Citation Information
Patent Citations
Motion capture display system and program thereof
JP2022123941A
