Information processing device and computer program

Abstract

To prevent an unnatural motion of three-dimensional model data when a target is imaged to generate a video in which a three-dimensional model is arranged in a virtual space using the three-dimensional model data of the imaged target.SOLUTION: An information processing device includes: scan control means that controls a position and an angle of a scanner and scans an outer appearance of a subject placed on a turn table to generate a scan image; data generation means that generates three-dimensional model data from the scan image; video generation means that generates a video in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; and receiving means that receives an input to designate joint positions for the three-dimensional model data. The data generation means generates the tree-dimensional model data in association with joint information based on the joint positions designated by the designation input. The video generation means generates the video of the three-dimensional model in which a joint of the three-dimensional model data is operated based on the joint information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device and a computer program. [Background technology]

[0002] Patent document 1 describes a technology in which a texture generated from a group of captured images of an object taken from multiple angles by a photographing unit is mapped onto a primitive in a virtual space to generate a virtual space image as seen from a virtual camera in the virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-107251 A Summary of the Invention [Problem to be solved by the invention]

[0004] In generating virtual space images using the above technology, it is expected that unnatural movement of the three-dimensional model data can be prevented when photographing an object and generating an image in which the three-dimensional model is placed in a virtual space using the three-dimensional model data of the photographed object.

[0005] Therefore, the objective of the present invention is to prevent unnatural movement of three-dimensional model data when photographing an object and generating an image in which a three-dimensional model is placed in a virtual space using the three-dimensional model data of the photographed object. [Means for solving the problem]

[0006] One aspect of the present invention to solve the above problem is an information processing device, a scan control means for controlling the position and angle of the scanner to scan the external appearance of an object placed on a rotating table to generate a scanned image; a data generating means for generating three-dimensional model data from the scanned image; an image generating means for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; Equipped with a receiving unit for receiving a designated input of a joint position for the three-dimensional model data; the data generating means generates the three-dimensional model data in association with joint information based on the designated and input joint positions; The image generating means generates an image of a three-dimensional model by operating the joints of the three-dimensional model data based on the joint information. Effect of the Invention

[0007] When an object is photographed and a video in which a three-dimensional model is placed in a virtual space is generated using three-dimensional model data of the photographed object, unnatural movement of the three-dimensional model data can be prevented. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the arrangement of an image processing system 10 according to an embodiment. [Diagram 2] 1 is a diagram showing an example of a hardware configuration of an information processing device 100 and an example of a hardware configuration of a headset 150 according to an embodiment. [Diagram 3] FIG. 1 is a diagram showing an example of the appearance of a doll-type model according to an embodiment. [Figure 4A] FIG. 1 is a diagram showing an example of an implementation of an image processing system 10. [Figure 4B] FIG. 2 is a diagram showing an example of installation of a model 300 according to an embodiment. [Figure 4C] 5A and 5B are diagrams for explaining an operating range of a scanner according to the embodiment. [Figure 5A] 4 is a flowchart corresponding to an example of processing in the image processing system 10 according to the embodiment. [Figure 5B] 11 is a flowchart corresponding to an example of a joint setting process according to the embodiment. [Figure 6A] FIG. 4 is a diagram showing an example of a configuration of an indicator according to the embodiment. [Figure 6B] 11A and 11B are diagrams illustrating an example of an indicator superimposed on three-dimensional model data according to an embodiment. [Figure 6C] 11A to 11C are diagrams for explaining a method of adjusting an indicator according to the embodiment. [Figure 7] 6A to 6C are diagrams for explaining joint setting processing according to the embodiment. [Figure 8] 11A and 11B are diagrams showing an example of setting an associated motion range for three-dimensional model data according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. Also, the same reference numbers are used for identical or similar configurations, and duplicated descriptions are omitted. Also, in each figure, the up, down, left, right, front and back directions relative to the paper surface are used in the description in the text as the up, down, left, right, front and back directions of the parts (or components) in this embodiment.

[0010] First, the configuration of an image processing system corresponding to this embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of an image processing system 10 corresponding to this embodiment. The image processing system 10 is configured by connecting an information processing device 100 to a scanner 110, a support arm 120, a model support device 130, a display device 140, a headset 150, a depth sensor 160, and the like. Note that the system configuration is not limited to that shown in FIG. 1, and the information processing device 100 may be further connected to an external server, a cloud server, and the like via a network. The external server, etc., can execute at least a part of the processing of the embodiment described below.

[0011] The information processing device 100 controls the operation of at least one of the scanner 110, the support arm 120, the model support device 130, and the depth sensor 160, photographs the object to be photographed from any angle to generate multiple images, and generates three-dimensional model data (this data) from the multiple images. If the object to be photographed is configured to be separable into multiple constituent objects, images may be taken of each constituent object to generate three-dimensional model data, and these may be integrated to generate the three-dimensional model data of the object. The information processing device 100 is not limited to being a single device, and may be configured to distribute functions to multiple devices and integrate the whole to achieve the functions of this embodiment.

[0012] The information processing device 100 can also function as an image generating device that generates images to be displayed in a virtual space by using the generated three-dimensional model data as virtual space data. In this embodiment, the subjects to be photographed are articles such as an assemblable plastic model, an action figure (a figure with movable joints), a toy, a doll, etc., which are collectively referred to as "models" below.

[0013] Next, the scanner 110 is a three-dimensional scanner device that captures (scans) the three-dimensional shape of a model to be captured under the control of the information processing device 100, and outputs the three-dimensional shape and color information of the captured object. In this embodiment, the scan signals output from the scanner 110 are collectively called "scanned images." For example, Space Spider manufactured by Artec can be used as the scanner 110. In this embodiment, for example, 500 to 800 frames of scanned images can be acquired to obtain 3D scan data of the entire toy body. In addition, for example, a smartphone with a camera on which an application for capturing three-dimensional shapes is installed may be used as the scanner 110.

[0014] The support arm 120 is a position and orientation control device that moves the scanner 110 to a predetermined photographing position and orientation under the control of the information processing device 100. The support arm 120 may be configured to be capable of manually changing the photographing position and orientation, and to be capable of fixing the changed position and orientation, or may be configured to be controllable by the information processing device 100. When the support arm 120 is configured to be controllable by the information processing device 100, for example, xArm 7 manufactured by UFACTORY can be used. The xArm 7 is made of seven joints and can move in the same manner as a human arm. Note that instead of using the support arm 120, the scanner 110 may be positioned manually. For photographing positions and orientations that cannot be covered by the support arm 120, the scanner 110 may be manually operated to perform scanning.

[0015] The model support device 130 is a support base (or a mounting base on which the model is mounted) that supports a model with a fixed pose. The model support device 130 may be configured to be rotatable, for example, when the model is mounted on the support base or at the tip of a support rod. In this embodiment, the scanner 110 is positioned at an arbitrary photographing position and photographing angle by the support arm 120, and then the model support device 130 is rotated to perform photographing. The model support device 130 can rotate both clockwise and counterclockwise. In addition, the model support device 130 can be stopped at an arbitrary rotation angle within the range of 0 degrees to 360 degrees, and the model support device 130 can be stopped at a specific rotation angle and positioned at an arbitrary photographing position and photographing angle within the movable range of the scanner 110 to perform photographing. By performing this at a plurality of rotation angles, photographing positions, and photographing angles, an image of the entire model can be obtained. In addition, by driving the support arm 120 and the model support device 130 synchronously, the photographing process can be performed more easily and with high accuracy. Also, instead of using the model support device 130, the scanner 110 may be manually moved around the model to scan from any desired imaging position and angle.

[0016] The display device 140 is a display device such as a liquid crystal display (LCD), and can display the processing results of the information processing device 100. Specifically, it can display an image acquired by the scanner 110, display the main data of the three-dimensional model data generated from the captured image, superimpose an indicator for setting a joint position described below on the main data of the three-dimensional model data, and display an image (VR image) reconstructed using the main data. The display device 140 can include an operation unit 140A that accepts an operation from a user who is an observer of the displayed image, and the user can operate the operation unit 140A to perform an operation input according to the content of the image displayed on the display device 140.

[0017] The headset 150 is composed of a head mounted display 150A and a controller 150B, which will be described later. In particular, the headset 150 may be configured as a VR headset capable of providing moving images corresponding to the posture and inclination of a user who is an observer. A specific application is installed in the headset 150, and application data executed in the application can be downloaded from the information processing device 100 and executed. In this embodiment, the headset 150, like the information processing device 100, can function as an image generating device that uses three-dimensional model data as virtual space data to generate an image to be displayed in a virtual space and provides the generated image as a VR image. The application data includes display data for displaying the VR image.

[0018] The depth sensor 160 is a sensor capable of measuring the distance to a subject or object present in a predetermined space. The depth information acquired by the depth sensor 160 is output to the information processing device 100, and is used to identify the position information of the model to be photographed by the scanner 110 and control the scanning range of the scanner 110 and the operating range of the support arm 120. The depth sensor 160 can be installed, for example, on the stand of the support arm 120. However, the installation position is not limited thereto, and the depth sensor 160 can be installed in any location as long as it can measure all objects to be scanned installed on the model support device 130 and does not interfere with the movement of the support arm 120. For example, a depth camera of the RealSense series manufactured by Intel Corporation can be used as the depth sensor 160.

[0019] Next, the hardware configuration of the information processing device 100 corresponding to this embodiment will be described. FIG. 2(A) shows an example of the hardware configuration of the information processing device 100. The CPU 101 is a device that performs overall control of the information processing device 100 and performs calculation, processing, and management of data. For example, the CPU 101 controls the timing of image capture and the number of images captured by the scanner 110, and controls the arm joint of the support arm 120 to place the scanner 110 at an arbitrary image capture position and image capture angle. For example, after the image capture position and image capture angle of the scanner 110 are determined, the model support device 130 can be rotated to perform an image capture operation by the scanner 110. In addition, after the model support device 130 is stopped at a predetermined rotation angle, the scanner 110 can perform an image capture operation at an arbitrary image capture position and image capture angle. In addition, by controlling the operation of the scanner 110 and the support arm 120 based on the depth information from the depth sensor 160, the image capture position and image capture angle of the scanner 110 can be narrowed down to a specific range.

[0020] CPU 101 can also function as an image processing unit that processes images output from scanner 110. Specifically, CPU 101 can perform processing to generate an image that displays a three-dimensional model in a virtual space by using three-dimensional model data generated based on a scan signal obtained by scanner 110 as virtual space data.

[0021] The RAM 102 is a volatile memory and is used as a temporary storage area such as the main memory and work area of ​​the CPU 101. The ROM 103 is a non-volatile memory, and image data and other data, various programs for the operation of the CPU 101, and the like are stored in respective predetermined areas. The CPU 101 controls each part of the information processing device 100 using the RAM 102 as a work memory according to, for example, a program stored in the ROM 103. Note that the program for the operation of the CPU 101 is not limited to being stored in the ROM 103, and may be stored in the storage device 104.

[0022] The storage device 104 is composed of a magnetic disk such as a HDD or a flash memory. Application programs, an OS, control programs, related programs, game programs, etc. are stored in the storage device 104. The storage device 104 can read and write data under the control of the CPU 101. The storage device 104 may be used in place of the RAM 102 or the ROM 103.

[0023] The communication device 105 is a communication interface for communicating with the scanner 110, the support arm 120, the model support device 130, the display device 140, the headset 150, and the depth sensor 160 under the control of the CPU 101. The communication device 105 may further be configured to be able to communicate with an external server or the like. The communication device 105 may include a wireless communication module, which may include known circuitry including an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module card, a memory, and the like. Here, the communication between the information processing device 100 and the scanner 110, the support arm 120, the model support device 130, the display device 140, the headset 150, and the depth sensor 160 may be performed by wireless communication.

[0024] The communication device 105 may also include a wired communication module for wired connection. The wired communication module enables communication with other devices including the display device 140 via one or more external ports. The communication device 105 may also include various software components for processing data. The external ports are connected to other devices directly via Ethernet, USB, IEEE1394, etc., or indirectly via a network. Note that software that realizes the same functions as the above devices may be used to replace the hardware devices.

[0025] The operation unit 106 is composed of, for example, a button, a keyboard, a touch panel, a controller, etc., and accepts operation input from a user. The operation unit 106 may be shared with the operation unit 140A, or may be independent of the operation unit 140A. For example, when the operation unit 140A is assumed to be an operation unit composed of a keyboard and a mouse, etc., it can be shared with the operation unit 106. On the other hand, when the operation unit 140A is assumed to be an operation unit composed of a touch panel, a controller, etc., it can be an operation unit separate from the operation unit 106.

[0026] The display control unit 107 functions as a control unit for displaying information on the display device 140 connected to the information processing device 100, and controls the operation of the display device 140. A part of the functions of the operation unit 106 may be provided in the display device 140. For example, the display device 140 may be configured as a device equipped with a touch panel such as a tablet terminal.

[0027] Next, a hardware configuration of the headset 150 corresponding to this embodiment will be described. FIG. 2B shows an example of the hardware configuration of the headset 150. The headset 150 is composed of a head mounted display (HMD) 150A and a controller 150B. The HMD 150A provides a right eye image and a left eye image to the left and right eyes of the user, respectively, and enables a virtual reality (VR) experience by creating a sense of stereoscopic effect due to the parallax between them. The controller 150B is provided in a housing independent of the HMD 150A. The controller 150B is assumed to be configured as a pair of two controllers to be held and operated by the user's left and right hands, respectively, but may be a single controller.

[0028] The CPU 151 is a device that performs overall control of the headset 150 and calculates, processes, and manages data. For example, it can execute application data downloaded from the information processing device 100 to display VR video on the display 157. In addition, based on an operation accepted via the controller 156 or information detected by the detection unit 158, it can switch the VR video itself to be displayed, switch the viewpoint within the VR space, or change the position within the VR space.

[0029] RAM 152 is a volatile memory and is used as a temporary storage area such as the main memory and work area of ​​CPU 151. ROM 153 is a non-volatile memory, and image data and other data, various programs for CPU 151 to operate, and the like are stored in respective predetermined areas. CPU 151 controls each part of headset 150 using RAM 152 as a work memory according to a program stored in ROM 153, for example. Note that the program for CPU 151 to operate is not limited to being stored in ROM 153, and may be stored in storage device 154.

[0030] The storage device 154 is composed of a magnetic disk such as a HDD or a flash memory. The storage device 154 stores application programs, an OS, control programs, related programs, game programs, application data downloaded from the information processing device 100, display data, and the like. The storage device 154 can read and write data based on the control of the CPU 151. The storage device 154 may be used instead of the RAM 152 or the ROM 153.

[0031] The communication device 155 is a communication interface for communicating with the information processing device 100 and the controller 150B under the control of the CPU 151. The communication device 155 includes a wireless communication module that enables wireless communication based on Bluetooth or WiFi (IEEE802.11). The headset 150 can connect to the information processing device 100 via wireless communication and download display data for VR images. In addition, the headset 150 communicates with the controller 150B and receives information on user operation instructions for the controller 150B.

[0032] The display 156 is configured to provide the right eye image and the left eye image generated by the CPU 151 to the right eye and the left eye of the user, respectively. The detection unit 157 is a mechanism for detecting the gaze direction of the left and right eyes of the user and the tilt of the head mounted display. The detection unit 157 includes, for example, a sensor for detecting the gaze direction or the gaze direction of the user wearing the HMD 150A. It also includes a gyroscope, a magnetometer, an accelerometer, a global positioning system (GPS), a compass, etc., and can specify the position, attitude, tilt, etc. of the HMD 150A according to the detection information. The detection unit 157 detects information for determining the gaze direction and action of the user in the HMD 150A and transmits it to the CPU 151. The CPU 151 determines the gaze direction and the action based on the received detection information, and adjusts the image presented on the display 156 of the HMD 150A so as to match the determined gaze direction and the action.

[0033] The controller 150B is configured with, for example, a plurality of buttons, a cross key, etc., and receives operation inputs from the user such as selection operations, direction instruction operations, etc. The controller 150B is connected to the HMD 150A via a communication device 155 via wireless communication.

[0034] Next, referring to FIG. 3, an example of a model that is a subject of the photographing target in this embodiment will be described. The model 300 is a model that has a doll-like (robot or human) appearance. The model can be, for example, a plastic model that has been assembled and painted. Alternatively, it may be a completed model such as a figure (action figure) with movable joints. The model in FIG. 2 is merely an example for the purpose of explanation, and the shape of the model is not limited to a doll-like appearance, but can be any shape such as a general vehicle, a racing vehicle, a military vehicle, an aircraft, a ship, an animal, a virtual life form, etc. It is to be noted that the object to be photographed is not limited to a model as long as the object can be photographed three-dimensionally by the scanner 110.

[0035] The model 300 has each of the parts of a head 301, a chest 302, a right arm 303, a left arm 304, a right torso 305, a left torso 306, a right leg 307, a left leg 308, a right foot 309, and a left foot 310, which are joined together to form the model. At least a part of each of the parts 301 to 310 is supported so as to be rotatable (or swingable) relative to the adjacent parts. For example, the head 301 is supported so as to be rotatable relative to the chest 302, and the right arm 303 and the left arm 304 are supported so as to be rotatable relative to the chest 302. In this way, each part of the model 300 is provided with a joint structure, so that the model 300 can take any posture.

[0036] Next, an overall implementation example of the image processing system 10 in this embodiment will be described with reference to Fig. 4A. Fig. 4A is a diagram showing a generalized implementation example capable of generating the main data generated by photographing a model.

[0037] 4A, case 401 contains information processing device 100, a drive system for driving support arm 402, a drive system for driving turntable 406, and the like. Note that support arm 402 can also be manually adjusted in shooting direction and position. The surface of case 401 has a flat structure so that a promotional poster can be attached.

[0038] The support arm 402 corresponds to the support arm 120, and can support and fix the position of a terminal 403 functioning as the scanner 110 under the control of the information processing device 100 or manually. The support arm 402 can also operate to control the inclination of the terminal 403.

[0039] The terminal 403 is a touch panel terminal with a built-in camera that can be used as the scanner 110, and can be, for example, a smartphone, a tablet terminal, or a digital camera. Instead of these terminals, Space Spider manufactured by Artec can be used. FIG. 4A is merely a diagram showing a generalized example of the system configuration, and the configuration can be changed according to the type of device used as the scanner 110. The terminal 403 can take an image of the model 300 and transmit it to the information processing device 100. The ring light 404 is a lighting device used when the terminal 403 takes an image of the model 300, and can apply light evenly to the model 300 to make it difficult to cast a shadow. In addition to the ring light 404, a top light or auxiliary lights on the left, right, and bottom may be installed as an additional light source. Instead of the terminal 403, a three-dimensional scanner device as described above may be used.

[0040] The background sheet 405 is a background sheet for photography, and for example, a white sheet can be used. The turntable 406 can mount the model 300 and rotate the model. The turntable 406 and its configuration including its drive system correspond to the model support device 130, and the turntable 406 rotates clockwise and counterclockwise and stops under the control of the information processing device 100. A plurality of predetermined markers 410 may be arranged on the turntable 406. The markers 410 can be used to adjust the orientation and position of the photographed model.

[0041] In FIG. 4A, the model is placed on a semi-transparent (transparent) base, but other support devices such as an "Action Base" (registered trademark) may be used. The Action Base has a support pole configured to be bent into an "L" shape on a base, and the model can be attached to the tip of the pole. In this case, the above-mentioned marker 410 may be placed on the tip of the pole or on the base. The method of indicating the model can be changed depending on the pose. For example, in the case of an upright posture, the model can be placed on a transparent base and photographed. On the other hand, in the case of a flying posture, when it is desired to photograph the soles of the feet, an action base may be used. The action base may be used to photograph an upright posture.

[0042] The display device 407 is a device corresponding to the display device 140, and may have a touch panel function. The user can use the touch panel function to perform a predetermined selection operation. Also, operations can be accepted by a controller or mouse (not shown) that can function as the operation unit 140A instead of the touch panel. The VR headset 408 is composed of an HMD 408A and a controller 408B, and corresponds to the HMDs 150A and 150B, respectively. The user can perform operations while watching VR images by wearing the HMD 408A on the head and holding and operating the controller 408B in each hand.

[0043] 4B shows an example of installation of the model 300 using an action base. In this embodiment, as shown in FIG. 4B, four AR (Augmented Reality) markers 413 are placed on a base 412 of an action base 411, and the center and orientation of the action base are identified by the AR markers 413.

[0044] The AR marker 413 is used to adjust the orientation and position of the photographed model. Three-dimensional coordinates can be identified from the AR marker 413, and the inclination of the X-axis, Y-axis, and Z-axis can be identified. For example, if the three-dimensional coordinates identified from one AR marker 413 are used as a reference, the orientation and inclination of the data can be correctly corrected by matching the three-dimensional coordinates identified from the remaining AR marker 413 to the reference.

[0045] When the model 300 is installed using the action base 411, the connection relationship between the action base 411 and the model 300 is substantially uniquely fixed, so that the front of the action base 411 coincides with the front of the model 300. Therefore, if the front is determined by the center and direction of the action base 411, the front of the model 300 is also determined naturally. In this embodiment, the AR marker 413 may be a three-dimensional marker (a marker created by three-dimensional modeling) or a two-dimensional marker (for example, a marker created by printing on a sticker). In the example shown in FIG. 4B, a two-dimensionally created marker is used.

[0046] Next, an example of processing executed by the image processing system 10 according to the present embodiment will be described with reference to Fig. 5A. At least a part of the processing corresponding to the flowchart is realized by the CPU 101 of the information processing device 100 executing a program stored in the ROM 103 or the storage device 104.

[0047] First, in S501, CPU 101 accepts user registration. It accepts input of the user's name and contact information. It assigns a user identifier to each user to uniquely identify the individual user. CPU 101 stores the input user information in storage device 104 in association with the time the input was accepted and the user identifier. At this time, it may be possible to specify the type of model of oneself to be scanned (for example, a "doll-type and realistic model", a "doll-type and deformed model", or a "model with arms and legs omitted").

[0048] When user registration is complete, the user sets his / her own model on the model support device 130. The CPU 101 can determine whether or not the model 300 has been set on the model support device 130 based on the scanned image by the scanner 110. Alternatively, a switch that is turned on when the model 300 is set on the model support device 130 may be provided, and the CPU 101 can detect a signal from the switch to make the determination. Alternatively, a button for accepting an operation when the setting of the model 300 is completed may be displayed on the display device 140, and the CPU 101 can detect whether or not an operation in response to the button operation has been accepted. In S502, the CPU 101 detects that the model 300 has been set on the model support device 130 by any of the methods described above. In response to this detection, the process proceeds to S503.

[0049] In S503, the scanner 110, the support arm 120, the model support device 130, and the depth sensor 160 cooperate to execute a scan process (photographing process) to generate a scan image. Specifically, the CPU 101 controls the support arm 120 to move the scanner 110 to any of the registered photographing positions, and the scanner 110 can perform scanning while rotating the model support device 130 at the photographing position. In addition, after the model support device 130 is stopped at a predetermined rotation angle, the scanner 110 can perform a photographing operation at any photographing position and photographing angle. The photographed scan image is transmitted to the information processing device 100, and the CPU 101 stores the image in a table in the storage device 104 in association with a user identifier, etc.

[0050] Furthermore, in S503, the CPU 101 acquires depth information from the depth sensor 160 and controls the operations of the scanner 110 and the support arm 120. Specifically, the CPU 101 analyzes volume information of the space to be scanned acquired by the depth sensor 160, detects the position and range, and sets a predetermined range (called a "bounding box") to be the scanning target in the depth space acquired by the depth sensor 160.

[0051] In this case, if all the outputs from the depth sensor 160 are used, the amount of information is large, and even spatial information unnecessary for actual processing (for example, surrounding walls and the turntable 406) is included, so the processing target may be limited to depth information within a specific range (specified by width, height, and depth) in advance. The bounding box is specified by the width, height, and depth of the space surrounding the object included in the pre-specified range. This makes it possible to specify the volume or size of the scan target. In addition, since the scan target exists within the bounding box, the operating range (vertical and horizontal directions) of the support arm 120 may be set based on the size of the bounding box.

[0052] The bounding box can be common regardless of the type of the model 300. Therefore, the process of setting the bounding box based on the depth information from the depth sensor 160 may be performed in advance. In that case, it is not necessary to newly obtain the depth information in the process in S503.

[0053] Fig. 4C is a diagram for explaining the concept of a bounding box. In Fig. 4C, the model 300 is supported by an action base 411 and is disposed approximately at the center of the turntable 406. In Fig. 4C, the scanner device 413 is supported by a support arm 402, and it is assumed that Space Spider manufactured by Artec is used here. The drive system for driving the support arm 402, the drive system for driving the turntable 406, and the like are the same as those described in Fig. 4A, and therefore description thereof will be omitted.

[0054] In the space to be scanned on the turntable 406 shown in FIG. 4C, a bounding box 415 is set so as to include the model 300, the action base 411, and the pedestal 412. The action base 411 and its pedestal 412 (collectively referred to as the "installation base") on which the model is to be installed are included in the bounding box in order to obtain information for controlling the position and orientation of the three-dimensional model data. At this time, a depth sensor 414 is installed near the pedestal below the support arm 402. By setting the bounding box 415 in this way, the scan range can be limited, the amount of data to be processed can be reduced, and the processing speed can be increased while also achieving efficiency.

[0055] In the next S504, the CPU 101 executes post-scan processing on the scanned image acquired by the scan processing, generates each 3D model data of the model 300, and generates application data (display data) for video display. In the post-scan processing, an area defining the range of unnecessary data is created in advance, the data is read, and the surfaces included in the unnecessary area are deleted. This makes it possible to almost completely remove surfaces other than those of the model 300.

[0056] In addition, in post-scan processing, the front of the three-dimensional model data is identified based on the AR marker 413. By identifying the front of the three-dimensional model data, it becomes possible to improve the setting accuracy when setting the joint position by superimposing an indicator in the joint setting process at a later stage.

[0057] Specifically, the position and rotation direction of each AR marker 413 in three-dimensional space are obtained from the analysis results of each AR marker 413 in the scanned image. Next, the coordinates and rotation value that are considered to be the center of the action base 411 are calculated from the position of the AR marker 413. In this embodiment, the AR marker 413 is disposed at a certain distance from the center of the action base, so the position can be calculated based on this information. At this time, errors in the analysis of the AR marker 413 cause deviations in the center position, so the average value of the analysis results is determined as the center position of the action base.

[0058] When performing joint setting processing as described later, a movement amount for moving the center position to the origin of the three-dimensional space in which the joint setting processing is performed is determined, and the position of the three-dimensional model data of the model 300 is corrected based on the movement amount. This makes it possible to move the action base 411 to the origin of any space and to face a specific direction, and therefore to specify the front of the three-dimensional model data of the model 300. In addition, since the same processing can be performed on common scan data, the processing can be standardized regardless of differences in the model 300.

[0059] In the next step S505, the CPU 101 performs joint setting processing. This processing is for embedding joints in the three-dimensional model data generated in the processing of S504 so that the three-dimensional model data does not move unnaturally when moved in a virtual space. As a result, when the three-dimensional model data is moved in a virtual space, the joints of the three-dimensional model are expressed to bend and extend according to the posture, and other parts do not bend unnaturally, eliminating unnaturalness in the overall movement. Details of the processing in S505 will be described later with reference to FIG. 5B.

[0060] The application data generated as described above can play back images when executed in a specific application on the information processing device 100. The images include images of a three-dimensional model in which the joints of the three-dimensional model data are operated based on the joint information. Furthermore, when the specific application is executed on the head mounted display 150A, VR images can be played back. The generated three-dimensional model data and display data are stored in the storage device 104 in association with user information.

[0061] In the next S506, image display processing is executed. When the image display processing is performed in the information processing device 100, the CPU 101 executes a specific application corresponding to the display data, and displays an image on the display device 140 via the display control unit 107. In addition, in S506, the CPU 101 may control the communication device 105 to transmit the display data to the HMD 150A, and the CPU 151 of the HMD 150A may execute the received display data in a corresponding application to execute processing to display a VR image, thereby performing the image display processing. Note that, when the specific application is a game application, the three-dimensional model data that has been subjected to the joint setting processing may be used as an operational character operated by a player in the game.

[0062] Next, the details of the process in S505 in Fig. 5A will be described with reference to Fig. 5B. Fig. 5B is a flowchart showing an example of the details of the joint setting process in S505.

[0063] In S511, an indicator for setting a joint position is superimposed on the three-dimensional model data generated in S504 and displayed on the display device 140. An example of the indicator is shown in FIG.

[0064] Fig. 6A shows an indicator 600 for setting two-dimensional joint positions of a standard doll-type model, where indicators indicating each joint position are indicated by cross marks 601. The cross marks 601 are arranged at each joint position of the neck, shoulders, elbows, wrists, waist, thighs, knees, and ankles, and a total of 14 cross marks are arranged in Fig. 6A. The indicator 600 also has a contour line 602 showing a rough contour of the doll-type model to inform the operator which joint position each cross mark 601 indicates. This allows the operator to know which part of the doll-type model each cross mark 601 corresponds to.

[0065] The indicator 600 shown in FIG. 6 can be used as a common indicator that can be applied regardless of the type of model. However, the indicator is not limited to a common indicator, and multiple types of indicators corresponding to the type of model that is the source of the three-dimensional model data may be prepared in advance. In this case, the operator can specify the type of model. For example, the indicator 600 shown in FIG. 6A corresponds to a doll-type and realistic type model, but an indicator corresponding to a doll-type and deformed type model may be prepared separately from this. In addition, an indicator corresponding to a model in which arms and legs are omitted may be additionally prepared. In this case, one of the prepared indicators may be selected and displayed according to the type of the model of the user that has been scanned and specified in advance by the user (for example, specified at the time of user registration reception (S501)). Alternatively, a part of the configuration (for example, legs, etc.) may be deleted from the indicator 600.

[0066] As shown in FIG. 6B(A), the indicator 600 is displayed superimposed on the three-dimensional model data 610, resulting in a superimposed state 620 as shown in FIG. 6B(B). At this time, the size of the indicator 600 may be adjusted (enlarged or reduced) according to the overall height of the three-dimensional model data 610. Note that in this state, the joint positions of the indicator 600 and the three-dimensional model data 610 do not overlap appropriately, and a misalignment occurs. Therefore, in S512 of the flowchart in FIG. 5, an adjustment operation of the indicator is accepted, and the cross mark 601 is moved to the corresponding joint position of the three-dimensional model data 610.

[0067] FIG. 6C shows the process of moving the cross mark 601. FIG. 6C(A) shows a state in which the indicator 600 is simply superimposed on the three-dimensional model data 610, corresponding to the superimposed state 620 shown in FIG. 6B(B). In S512, in such a superimposed state 620, an operation for moving the position of the cross mark 601 is accepted so that the position of the cross mark 601 is located on the joint position of the three-dimensional model data 610. The operator can perform an operation for selecting and moving the cross mark 601 (for example, a drag-and-drop operation) by using the mouse of the operation unit 140A or the like. As a result, for example, the cross marks 601 placed on the left and right arms are moved outward, and move to the elbow joint position and wrist joint position of the arms of the three-dimensional model data 610 as shown in the superimposed state 630 of FIG. 6C(B). At this time, the contour line 602 is also changed to match the position of the cross mark 601 after it has been moved, so whether the contour line 602 is in an appropriate position with respect to the external shape of the three-dimensional model data 610 (for example, whether the contour line 602 follows the external shape) can be used as a guide for moving the cross mark 601. An operation to shift the position of the neck joint upward is also accepted. As a result, in the superimposed state 630 of FIG. 6C(B), the cross mark 601 moves to the neck joint position of the three-dimensional model data 610.

[0068] Next, from the superimposed state 630 in FIG. 6C(B), an operation to move each of the cross marks 601 arranged on the left and right legs outward is accepted, and an operation to lower the joint positions at the base of the thighs is accepted. This transitions to the superimposed state 640 in FIG. 6C(C). In the superimposed state 640, it can be seen that the cross marks 601 are arranged at each joint position. Note that in this embodiment, the position of the outline 602 of the indicator 600 is changed by moving the cross mark 601, but conversely, the cross mark 601 may be moved to an appropriate position in response to the deformation of the outline 602 of the indicator 600 so as to follow the external shape of the three-dimensional model data 610.

[0069] In this manner, it is possible to accept designation of a two-dimensional joint position in the three-dimensional model data 610. In the following S513, the CPU 101 sets a three-dimensional joint position (this is called a "joint reference position") in the three-dimensional model data 610 based on the position of the cross mark 601 accepted in S512 and the depth information of the three-dimensional model data 610. Setting of the joint reference position will be described with reference to FIG. 7.

[0070] FIG. 7 is a diagram for explaining the processing in S513 and S514. FIG. 7(A) is a diagram showing an example of a cross section in the depth direction of the three-dimensional model data 610 corresponding to the elbow joint surrounded by a dotted line 641 in the superimposed state 640 shown in FIG. 6C(C). A cross mark 601 is superimposed on the surface of the elbow joint 641. The cross mark 601 is placed on the surface of the elbow joint of the three-dimensional model data 610, but the reference position when the cross mark 601 actually operates as a joint is a position inside the elbow joint of the three-dimensional model data, and it is necessary to embed the cross mark 601 from the surface of the elbow joint into the inside of the elbow joint. The embedding amount at that time can be determined based on the depth information of the elbow joint of the three-dimensional model data 610.

[0071] If the depth of the elbow joint 641 is D in the position based on the cross mark 601, the joint reference position can be located at the position of D / 2, for example. The joint reference position is not set as a "point" but as a region (or range, for example, spherical, rectangular, etc.) having a predetermined three-dimensional size. Therefore, the position of D / 2 can be set as the center of gravity or center of the region, and the position can be finely adjusted in the three-dimensional model data 610. FIG. 7(B) shows a state in which a three-dimensional region 701 is set inside the elbow joint 641. By setting the joint reference position with a certain range in this way, it is possible to absorb some deviations regarding the set position. Based on the joint reference position, a bone structure (also called a skeletal structure or framework structure) can be placed in the three-dimensional model data when moving the three-dimensional model data.

[0072] 5B, in the next S514, the CPU 101 sets an associated motion range based on the joint reference position set in S513. The associated motion range is a range in which the movement is affected when the joint is moved based on the joint reference position, or a range in which the joint moves together with the joint.

[0073] In setting the relevant range of motion, the set joint reference positions (and bone structure) and the polygon mesh structure of the three-dimensional model data are referenced, and the parts located between the joint reference positions are classified (or grouped or divided into areas) so as to be included in each part such as the head, arms, legs, etc. This process makes it possible to process adjacent parts that are not actually connected (such as the arms and the chest and waist) as separate parts. Note that in order to reduce the processing load, this may be performed only on parts that have a large visual impact.

[0074] A joint is generally a structure that connects one part to another part in a manner that allows them to flex and extend (bend and stretch). For example, an elbow joint is a joint that connects the upper arm and the forearm, and the forearm can be flexed and extended relative to the upper arm when the elbow joint functions. In this case, it is necessary to identify the parts that belong to the upper arm or the forearm. S514 is a process for identifying the relevant motion range of this joint.

[0075] Specifically, in this embodiment, polygon vertices within a predetermined range from a certain joint reference position are classified to be included in the part corresponding to the joint reference position. For example, polygon vertices within a predetermined range in the head direction from a joint reference position located at the neck are classified to be included in the part called "head". As a result, these polygon vertices are set as the "relevant motion range of the bone structure of the head", so that when the bone structure of the head moves around the joint reference position of the neck, this "relevant motion range of the bone structure of the head" (polygon vertices classified to be included in the part called "head") can be made to move together with the bone structure of the head.

[0076] As another example, polygon vertices within a predetermined range in the forearm direction from the elbow joint reference position are classified as being included in the region called "forearm." As a result, these polygon vertices are set as the "relevant motion range of the forearm bone structure," so that when the forearm bone structure moves around the elbow joint reference position, this "relevant motion range of the forearm bone structure" (polygon vertices classified as being included in the region called "forearm") can be made to move together with the forearm bone structure.

[0077] As another example, polygon vertices within a predetermined range in the direction of the lower leg from the joint reference position at the knee are classified as being included in the region called "lower leg." As a result, these polygon vertices are set as the "relevant motion range of the bone structure of the lower leg," so that when the bone structure of the lower leg moves around the knee joint reference position, this "relevant motion range of the bone structure of the lower leg" (polygon vertices classified as being included in the region called "lower leg") can be made to move together with the bone structure of the lower leg.

[0078] If the range is set too finely, the processing load will be high and the processing time will be long. On the other hand, if the range is set too wide, even parts that are not actually related will be affected by the joints, which may give a sense of discomfort to the person watching the video. Therefore, it is necessary to set the relevant motion range appropriately. In addition, multiple settings for this relevant motion range may be prepared according to the type of the scanned model (for example, whether it is a "doll-type and realistic type model", a "doll-type and deformed type model", or a "model with arms and legs omitted"), and the setting may be selectable or automatically selected according to the type of model.

[0079] FIG. 8 shows an example of a state in which related motion ranges are set for the three-dimensional model data 800 of the model 300. Here, the head 801, chest 802, waist 803, left shoulder 804, left upper arm 805, left forearm 806, left hand 807, right shoulder 808, right upper arm 809, right forearm 810, right hand 811, left thigh 812, left lower leg 813, left foot 814, right thigh 815, right lower leg 816, and right foot 817 are color-coded into the same group as related motion ranges. The ranges included in these groups of the three-dimensional model data 710 move together with respect to the movement of the corresponding joints. For example, the left and right thighs move together in the case of a hip joint, and the left and right lower legs move together in the case of a knee joint. The relevant motion range shown in FIG. 8 includes a range specified as a range sandwiched between two joint reference positions, such as the left and right upper arms, the left and right forearms, the left and right thighs, and the left and right lower legs.

[0080] The CPU 101 performs the above-mentioned processes in S513 and S514 for all joint positions on which the cross marks 601 are superimposed in the superimposed state 640 in Fig. 6C(C), and stores the obtained position information of the joint reference position 701 and information of the associated motion range 702 as joint information in the storage device 104 in association with the user information and the three-dimensional model data. After that, the process proceeds to S506.

[0081] As described above, in this embodiment, when generating three-dimensional model data from an image obtained by photographing a model with a scanner, joint information for specifying the position and range of the joints of the three-dimensional model data can be associated. As a result, when the three-dimensional model data is placed in a virtual space and an image is displayed, the movement of the three-dimensional model data can be controlled according to the joint information set for the three-dimensional model data. At this time, the joints of the three-dimensional model data do not bend unnaturally, and the viewer does not feel uncomfortable.

[0082] In addition, in this embodiment, the depth sensor 160 can be used to narrow down the range photographed by the scanner, thereby reducing the amount of data to be processed, thereby making the processing more efficient and shortening the processing time. Furthermore, when photographing a model with a scanner, a marker capable of detecting the position and inclination can be placed on the installation stand on which the model is installed, and differences in the conditions during photographing can be eliminated by correction at a later stage. This makes it possible to use a common installation stand even when scanning multiple different models, and to relax the accuracy of the installation conditions to a certain extent.

[0083] The three-dimensional model data acquired in the above-mentioned embodiment can be used, for example, to play a competitive game in a virtual space or to create a promotional video. However, the use of the three-dimensional model data is not limited to these, and the model can also appear in events, concerts, sports, online conferences, etc. held in a virtual space. Furthermore, the technology of this embodiment can be applied to a video technology such as cross reality (XR) that combines the real world and the virtual world and makes it possible to perceive things that do not exist in the real space.

[0084] In this way, in this embodiment, three-dimensional model data is generated from images of the exterior of the model, and it becomes possible to view a video in which the three-dimensional model is placed in a virtual space. Some models, such as prefabricated plastic models, are finished as unique works by users through painting or other processes, and these individual characteristics can be reflected in the character representation in the video or virtual space, which can greatly improve the user's preference.

[0085] <Summary of the embodiment> The above embodiment discloses at least the following information processing device and computer program. (1) An information processing device, a scan control means for controlling the position and angle of the scanner to scan the external appearance of an object placed on a rotating table to generate a scanned image; a data generating means for generating three-dimensional model data from the scanned image; an image generating means for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; Equipped with a receiving unit for receiving a designated input of a joint position for the three-dimensional model data; the data generating means generates the three-dimensional model data in association with joint information based on the designated and input joint positions; The image generating means is an information processing device that generates an image of a three-dimensional model by operating the joints of the three-dimensional model data based on the joint information. (2) the receiving means includes an operation receiving means for receiving an operation from a user of the information processing device, and a display means; The receiving means is the display means displays an indicator including a plurality of indices indicating joint positions superimposed on the three-dimensional model data; The operation receiving means receives an operation for aligning a position of the index with a position of a predetermined joint of the three-dimensional model data, The information processing device according to (1), which accepts input specifying the joint position. (3) The information processing device according to (2), wherein the indicator is selected from among different types of indicators depending on the type of the subject. (4) The information processing device according to (2), wherein the indicator is used in common regardless of the type of the subject. (5) The data generating means determining an embedding amount of a joint reference position for operating a corresponding joint in the three-dimensional model data based on depth information of the three-dimensional model data at the designated and input joint position; determining the joint reference position based on the joint position and the embedding amount; The information processing device according to (2), wherein the joint reference position is associated with the three-dimensional model data as the joint information. (6) The data generating means determining a related motion range that moves in relation to the motion of the joint in the three-dimensional model data when the joint is moved in the three-dimensional model data based on the joint reference position; The information processing device according to (5), further comprising: the associated motion range being included in the joint information. (7) The information processing device according to (6), wherein the relevant motion range includes a range identified as a range enclosed by the joint reference positions. (8) A plurality of markers are arranged on a surface of a base of a mounting table on which the subject is mounted, and the markers are configured to be capable of acquiring positions and rotation directions in a three-dimensional space; The data generating means extracting information of the plurality of markers from the scanned image to identify a position of a stand on which the subject is placed; The information processing device according to any one of (2) to (7), further comprising: correcting a position and an orientation of the three-dimensional model data based on the position. (9) The information processing device according to (8), wherein the correction includes correcting an orientation of the three-dimensional model data so that the orientation of the indicator is aligned with the front of the indicator when the indicator is displayed superimposed on the three-dimensional model data. (10) The information processing device according to (8) or (9), wherein the marker is an AR marker. (11) A depth sensor is further provided to obtain depth information of a range scanned by the scanner, The information processing device according to any one of (1) to (10), wherein the scan control means controls the position and angle of the scanner based on depth information from the depth sensor. (12) The information processing device described in (11), wherein the scan control means sets a range that includes the subject placed on the rotating table and a mounting table on which the subject is placed, and controls the position and angle of the scanner to scan that range. (13) A computer program for causing a computer to function as the information processing device according to any one of (1) to (12).

[0086] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

Claims

1. An information processing device, a scan control means for controlling the position and angle of the scanner to scan the external appearance of an object placed on a rotating table to generate a scanned image; a data generating means for generating three-dimensional model data from the scanned image; an image generating means for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; Equipped with a receiving unit for receiving a designated input of a joint position for the three-dimensional model data; the data generating means generates the three-dimensional model data in association with joint information based on the designated and input joint positions; The image generating means is an information processing device that generates an image of a three-dimensional model by operating the joints of the three-dimensional model data based on the joint information.

2. the receiving means includes an operation receiving means for receiving an operation from a user of the information processing device, and a display means; The receiving means is the display means displays an indicator including a plurality of indices indicating joint positions superimposed on the three-dimensional model data; The operation receiving means receives an operation for aligning a position of the index with a position of a predetermined joint of the three-dimensional model data, The information processing apparatus according to claim 1 , further comprising: an input for designating the joint position.

3. The information processing apparatus according to claim 2 , wherein the indicator is selected from among different types of indicators depending on the type of the subject.

4. The information processing device according to claim 2 , wherein the indicator is commonly used regardless of the type of the subject.

5. The data generating means determining an embedding amount of a joint reference position for operating a corresponding joint in the three-dimensional model data based on depth information of the three-dimensional model data at the designated and input joint position; determining the joint reference position based on the joint position and the embedding amount; The information processing apparatus according to claim 2 , wherein the joint reference positions are associated with the three-dimensional model data as the joint information.

6. The data generating means determining a related motion range that moves in relation to the motion of the joint in the three-dimensional model data when the joint is moved in the three-dimensional model data based on the joint reference position; The information processing device according to claim 5 , wherein the relevant motion range is further included in the joint information.

7. The information processing device according to claim 6 , wherein the relevant motion range includes a range identified as a range sandwiched between the joint reference positions.

8. a plurality of markers are arranged on a surface of a base of a mounting table on which the subject is mounted, and the markers are configured to be capable of acquiring positions and rotation directions in a three-dimensional space; The data generating means extracting information of the plurality of markers from the scanned image to identify a position of a stand on which the subject is placed; The information processing apparatus according to claim 2 , further comprising: correcting a position and an orientation of the three-dimensional model data based on the position.

9. The information processing apparatus according to claim 8 , wherein the correction includes correcting an orientation of the three-dimensional model data so that the orientation of the indicator matches a front direction of the indicator when the indicator is displayed superimposed on the three-dimensional model data.

10. The information processing device according to claim 8 , wherein the marker is an AR marker.

11. a depth sensor for acquiring depth information of a range scanned by the scanner; The information processing apparatus according to claim 1 , wherein the scan control means controls a position and an angle of the scanner based on depth information from the depth sensor.

12. 12. The information processing apparatus according to claim 11, wherein the scan control means sets a range that includes the subject placed on the rotating table and a table on which the subject is placed, and controls a position and an angle of the scanner so as to scan the range.

13. A computer program for causing a computer to function as the information processing device according to any one of claims 1 to 12.

Images