Information processing device and computer program

Abstract

To generate a video corresponding to attribute information of an imaging object by associating generated three-dimensional model data with the attribute information when imaging an object and generating a video in which a three-dimensional model is disposed on a virtual space by using the three-dimensional model data of the imaged object.SOLUTION: An information processing device includes: rotation control means configured to control operation of a rotary table on which a subject is placed; scan control means configured to control a position and an angle of a scanner and scan an appearance of the subject to generate a scan image; data generation means configured to generate three-dimensional model data from the scan image; video generation means configured to generate a video in which a three-dimensional model based on the three-dimensional model data is disposed in a virtual space; and receiving means configured to receive an input of attribute information regarding the subject to be scanned. The video generation means is configured to generate the video so as to include effects for each subject in accordance with the attribute information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device and a computer program. [Background technology]

[0002] Patent document 1 describes a technology in which a texture generated from a group of captured images of an object taken from multiple angles by a photographing unit is mapped onto a primitive in a virtual space to generate a virtual space image as seen from a virtual camera in the virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-107251 Summary of the Invention [Problem to be solved by the invention]

[0004] In generating virtual space images using the above technology, when an object is photographed and three-dimensional model data of the photographed object is used to generate an image in which a three-dimensional model is placed in a virtual space, it is expected that the generated three-dimensional model data will be associated with attribute information of the photographed object, so that an image corresponding to the attribute information can be generated.

[0005] Therefore, when photographing an object and using the three-dimensional model data of the photographed object to generate an image in which a three-dimensional model is placed in a virtual space, the object is to associate the generated three-dimensional model data with attribute information of the photographed object, thereby making it possible to generate an image according to the attribute information. [Means for solving the problem]

[0006] One aspect of the present invention to solve the above problem is an information processing device, a rotation control means for controlling the operation of a rotation table on which the subject is placed; a scan control means for controlling the position and angle of a scanner to scan the appearance of an object placed on the rotary table and generate a scanned image; a data generation means for generating three-dimensional model data from the scanned image; an image generating means for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; an output means for outputting the image generated by the image generation means to a display device; a receiving means for receiving input of attribute information relating to the subject to be scanned; Equipped with The image generating means generates the image so as to include a rendition for each of the subjects according to the attribute information. [Effects of the Invention]

[0007] When photographing an object and using the three-dimensional model data of the photographed object to generate an image in which a three-dimensional model is placed in a virtual space, the three-dimensional model data can be generated in association with attribute information of the photographed object, and an image can be generated according to the attribute information. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the configuration of an image processing system 10 according to an embodiment. [Figure 2] 1 is a diagram showing an example of a hardware configuration of an information processing device 100 according to an embodiment, and an example of a hardware configuration of a headset 150. [Figure 3] 1A and 1B are diagrams showing an example of the appearance of a doll-type model according to an embodiment. [Figure 4A] FIG. 1 is a diagram showing an example of implementation of an image processing system 10. [Figure 4B] FIG. 1 is a diagram showing an example of the configuration of an image processing system 10 according to an embodiment when performing scan photography. [Figure 5] 4A and 4B are diagrams for explaining the relationship between the rotational positions of a scanner device 413 and a turntable 406 during scanning and photographing according to an embodiment. [Figure 6]3 is a flowchart corresponding to an example of processing in the image processing system 10 according to the embodiment. [Figure 7] FIG. 10 is a diagram showing an example of an attribute specification screen according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] The following embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be combined arbitrarily. Identical or similar components are given the same reference numerals, and redundant descriptions will be omitted. In each figure, the up, down, left, right, front, and back directions relative to the paper surface will be used in the description of the present embodiment as the up, down, left, right, front, and back directions of components (or parts) in the present embodiment.

[0010] First, the configuration of an image processing system corresponding to this embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of an image processing system 10 corresponding to this embodiment. The image processing system 10 is configured by connecting an information processing device 100 to a scanner 110, a support arm 120, a model support device 130, a display device 140, a headset 150, etc. Note that the system configuration is not limited to that shown in FIG. 1, and the information processing device 100 may be further connected to an external server, a cloud server, etc. via a network. The external server, etc., can execute at least a part of the processing of the embodiment described below.

[0011] The information processing device 100 controls the operation of at least one of the scanner 110, the support arm 120, and the model support device 130, photographs the object to be photographed from any angle to generate multiple images, and generates three-dimensional model data (the actual data) from the multiple images. If the object to be photographed is configured to be separable into multiple component parts, it is also possible to photograph each component part to generate three-dimensional model data, and then combine these to generate the three-dimensional model data of the target object.

[0012] The information processing device 100 can also function as an image generating device that generates images to be displayed in a virtual space by using the generated three-dimensional model data as virtual space data. In this embodiment, the subjects to be photographed are articles such as assemblable plastic models, action figures (figures with movable joints), toys, dolls, etc., which are hereinafter collectively referred to as "models."

[0013] Next, the scanner 110 is a three-dimensional scanner device that captures (scans) the three-dimensional shape of a model to be captured under the control of the information processing device 100 and outputs the three-dimensional shape and color information of the captured object. In this embodiment, the scan signals output from the scanner 110 are collectively referred to as "scanned images." For example, the scanner 110 may be a Space Spider manufactured by Artec. In this embodiment, for example, 3D scan data of the entire toy body can be obtained by capturing approximately 500 to 800 frames of scanned images. Alternatively, the scanner 110 may be a smartphone with a camera on which an application for capturing three-dimensional shapes is installed.

[0014] The support arm 120 is a position and orientation control device that moves the scanner 110 to a predetermined imaging position and orientation under the control of the information processing device 100. The support arm 120 may be configured to manually change the imaging position and orientation and to fix the changed position and orientation, or may be configured to be controllable by the information processing device 100. When the support arm 120 is configured to be controllable by the information processing device 100, for example, xArm 7 manufactured by UFACTORY Corporation can be used. The xArm 7 consists of seven joints and can move in the same manner as a human arm. Note that the scanner 110 may be positioned manually instead of using the support arm 120. For imaging positions and orientations that cannot be covered by the support arm 120, the scanner 110 may be manually operated to perform a scan.

[0015] The model support device 130 is a support stand (or a mounting stand on which the model is placed) that supports a model with a fixed pose. The model support device 130 may be configured to be rotatable, for example, with the model placed on the support stand or at the end of a support rod. In this embodiment, the scanner 110 is positioned at a desired imaging position and imaging angle using the support arm 120, and then the model support device 130 is rotated to perform imaging. The model support device 130 can rotate both clockwise and counterclockwise. The model support device 130 can also be stopped at any rotation angle between 0 and 360 degrees. It is also possible to stop the model support device 130 at a specific rotation angle and position it at any imaging position and imaging angle within the movable range of the scanner 110 to perform imaging. By performing this process at multiple rotation angles, imaging positions, and imaging angles, an image of the entire model can be acquired. Here, synchronous driving of the support arm 120 and the model support device 130 can facilitate imaging with high accuracy. Furthermore, instead of using the model support device 130, the scanner 110 may be manually moved around the model to scan from any desired imaging position and angle.

[0016] The display device 140 is a display device such as a liquid crystal display (LCD), and can display the processing results of the information processing device 100. Specifically, it can display images acquired by the scanner 110, display the original three-dimensional model data generated from the captured images, and display images (VR images) reconstructed using the original data. The display device 140 can include an operation unit 140A that accepts operations from a user who is an observer of the displayed images, and the user can operate the operation unit 140A to perform operation input according to the content of the image displayed on the display device 140.

[0017] The headset 150 is composed of a head-mounted display 150A and a controller 150B, which will be described later. In particular, the headset 150 may be configured as a VR headset capable of providing moving images corresponding to the posture and tilt of a user who is an observer. A predetermined application is installed in the headset 150, and application data executed by the application can be downloaded from the information processing device 100 and executed. In this embodiment, the headset 150, like the information processing device 100, can function as an image generating device that uses three-dimensional model data as virtual space data to generate images to be displayed in a virtual space and provides the generated images as VR images. Note that the application data includes display data for displaying the VR images.

[0018] Next, the hardware configuration of the information processing device 100 corresponding to this embodiment will be described. FIG. 2A shows an example of the hardware configuration of the information processing device 100. The CPU 101 is a device that performs overall control of the information processing device 100 and performs calculations, processing, and management of data. For example, the CPU 101 controls the timing and number of images captured by the scanner 110, and can control the arm joint of the support arm 120 to position the scanner 110 at any desired imaging position and imaging angle. For example, after the imaging position and imaging angle of the scanner 110 are determined, the model support device 130 can be rotated to perform an imaging operation using the scanner 110. Furthermore, after the model support device 130 is stopped at a predetermined rotation angle, the scanner 110 can perform an imaging operation at any desired imaging position and imaging angle.

[0019] The CPU 101 can also function as an image processing unit that processes images output from the scanner 110. Specifically, the CPU 101 can perform processing to generate an image that displays the three-dimensional model in a virtual space, using three-dimensional model data generated based on the scan signal obtained by the scanner 110 as virtual space data.

[0020] The RAM 102 is a volatile memory and is used as a temporary storage area such as the main memory or work area of the CPU 101. The ROM 103 is a non-volatile memory and stores image data and other data, various programs for the operation of the CPU 101, and the like, in predetermined areas. The CPU 101 controls each unit of the information processing device 100 using the RAM 102 as a work memory in accordance with, for example, a program stored in the ROM 103. Note that the program for the operation of the CPU 101 is not limited to being stored in the ROM 103, and may be stored in the storage device 104.

[0021] The storage device 104 is configured with a magnetic disk such as an HDD or flash memory. The storage device 104 stores application programs, an OS, control programs, related programs, game programs, etc. The storage device 104 can read and write data under the control of the CPU 101. The storage device 104 may be used in place of the RAM 102 or the ROM 103.

[0022] The communication device 105 is a communication interface for communicating with the scanner 110, the support arm 120, the model support device 130, the display device 140, and the headset 150 under the control of the CPU 101. The communication device 105 may further be configured to be able to communicate with an external server or the like. The communication device 105 may include a wireless communication module, which may include well-known circuitry including an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module card, a memory, etc. Here, communication between the information processing device 100 and the scanner 110, the support arm 120, the model support device 130, the display device 140, and the headset 150 may be performed via wireless communication.

[0023] The communication device 105 may also include a wired communication module for wired connection. The wired communication module enables communication with other devices, including the display device 140, via one or more external ports. It may also include various software components for processing data. The external ports are connected to other devices directly via Ethernet, USB, IEEE 1394, etc., or indirectly via a network. Note that software that realizes the same functions as each of the above devices may be configured as a substitute for the hardware devices.

[0024] The operation unit 106 is configured with, for example, buttons, a keyboard, a touch panel, a controller, etc., and accepts operation input from a user. The operation unit 106 may be shared with the operation unit 140A, or may be independent of the operation unit 140A. For example, if the operation unit 140A is intended to be an operation unit configured with a keyboard, a mouse, etc., the operation unit 106 can be shared with the operation unit 106. On the other hand, if the operation unit 140A is intended to be an operation unit configured with a touch panel, a controller, etc., the operation unit 106 can be a separate operation unit.

[0025] The display control unit 107 functions as a control unit for displaying information on the display device 140 connected to the information processing device 100, and controls the operation of the display device 140. Some of the functions of the operation unit 106 may be provided in the display device 140. For example, the display device 140 may be configured as a device equipped with a touch panel such as a tablet terminal.

[0026] Next, the hardware configuration of the headset 150 corresponding to this embodiment will be described. FIG. 2(B) shows an example of the hardware configuration of the headset 150. The headset 150 is composed of a head-mounted display (HMD) 150A and a controller 150B. The HMD 150A provides a right-eye image and a left-eye image to the user's left and right eyes, respectively, and enables a virtual reality (VR) experience by creating a sense of three-dimensionality due to the parallax between the images. The controller 150B is provided in a housing independent of the HMD 150A. The controller 150B is assumed to be configured as a pair of two controllers to be held and operated by the user's left and right hands, respectively, but may also be a single controller.

[0027] The CPU 151 is a device that performs overall control of the headset 150 and performs calculations, processing, and management of data. For example, it can execute application data downloaded from the information processing device 100 to display VR video on the display 157. Furthermore, based on operations received via the controller 156 or information detected by the detection unit 158, it can switch the VR video itself to be displayed, switch the viewpoint within the VR space, or change the position within the VR space.

[0028] RAM 152 is a volatile memory and is used as a temporary storage area such as the main memory or work area of CPU 151. ROM 153 is a non-volatile memory in which image data and other data, various programs for operating CPU 151, and the like are stored in predetermined areas. CPU 151 controls each unit of headset 150 using RAM 152 as a work memory in accordance with, for example, a program stored in ROM 153. Note that the program for operating CPU 151 is not limited to being stored in ROM 153, and may also be stored in storage device 154.

[0029] The storage device 154 is configured by a magnetic disk such as an HDD or flash memory. The storage device 154 stores application programs, an OS, control programs, related programs, game programs, application data downloaded from the information processing device 100, display data, and the like. The storage device 154 can read and write data based on the control of the CPU 151. The storage device 154 may be used instead of the RAM 152 or the ROM 153.

[0030] The communication device 155 is a communication interface for communicating with the information processing device 100 and the controller 150B under the control of the CPU 151. The communication device 155 includes a wireless communication module that enables wireless communication based on Bluetooth or WiFi (IEEE802.11). The headset 150 can connect to the information processing device 100 via wireless communication and download display data for VR images. The headset 150 also communicates with the controller 150B and receives information on user operation instructions for the controller 150B.

[0031] The display 156 is configured to provide the right-eye image and left-eye image generated by the CPU 151 to the user's right eye and left eye, respectively. The detection unit 157 is a mechanism for detecting the gaze direction of the user's left and right eyes and the tilt of the head-mounted display. The detection unit 157 includes, for example, a sensor for detecting the gaze direction or gaze direction of the user wearing the HMD 150A. It also includes a gyroscope, a magnetometer, an accelerometer, a global positioning system (GPS), a compass, etc., and can identify the position, posture, tilt, etc. of the HMD 150A based on the detection information. The detection unit 157 detects information for determining the user's gaze direction and action in the HMD 150A and transmits it to the CPU 151. The CPU 151 determines the gaze direction and action based on the received detection information and adjusts the image presented on the display 156 of the HMD 150A to match the determined gaze direction and action.

[0032] The controller 150B is configured with, for example, a plurality of buttons, a cross key, etc., and receives operation inputs from the user such as selection operations, direction instruction operations, etc. The controller 150B is connected to the HMD 150A via a communication device 155 via wireless communication.

[0033] Next, referring to FIG. 3 , an example of a model and its accessories that are the subject of the image capture in this embodiment will be described. The model 300 is a model with a doll-like (robot or human) appearance. The model may be, for example, a plastic model that has been assembled and painted. Alternatively, it may be a completed model such as a figurine (action figure) with movable joints. The model in FIG. 2 is merely an example for illustrative purposes, and the shape of the model is not limited to a doll-like appearance, and may be a model of any shape, such as a general vehicle, racing vehicle, military vehicle, aircraft, ship, animal, or virtual creature. It should be noted that the object to be captured is not limited to a model, as long as the object's three-dimensional shape can be captured by the scanner 110.

[0034] The model 300 has a head 301, a chest 302, a right arm 303, a left arm 304, a right torso 305, a left torso 306, a right leg 307, a left leg 308, a right foot 309, and a left foot 310, which are joined together. At least a portion of each of the parts 301 to 310 is supported so as to be rotatable (or swingable) relative to the adjacent parts. For example, the head 301 is supported so as to be rotatable relative to the chest 302, and the right arm 303 and the left arm 304 are supported so as to be rotatable relative to the chest 302. In this way, each part of the model 300 is provided with a joint structure, so the model 300 can assume any posture.

[0035] The model 300 also includes an accessory 311. The accessory 311 is an accessory part that is expected to be used with the model 300, and in FIG. 3, it is a rifle-shaped weapon. That is, it is expected that the model 300 will use the accessory 311 by holding it with both hands. The shape of the weapon is not limited to a rifle, and can include any weapon shape, such as a sword, a shield, a bomb (e.g., a grenade), or a bazooka. The accessory is not limited to a weapon, and may be anything that can be used with the model 300, such as a decorative item. For example, if the model 300 is a doll-type model of a specific character, the accessory may be any item that the character carries (e.g., a carry-on bag, a hat, a bag, an accessory, a smartphone, or any other item that the character can carry).

[0036] Next, an overall implementation example of the image processing system 10 in this embodiment will be described with reference to Fig. 4A. Fig. 4A is a diagram showing a generalized implementation example capable of generating the main data generated by photographing a model.

[0037] 4A, case 401 contains information processing device 100, a drive system for driving support arm 402, a drive system for driving turntable 406, etc. Note that support arm 402 can also be manually adjusted for shooting direction and position. The surface of case 401 has a flat structure so that a promotional poster can be attached.

[0038] The support arm 402 corresponds to the support arm 120, and can support and fix the position of the terminal 403 functioning as the scanner 110 under the control of the information processing device 100 or manually. The support arm 402 can also operate to control the tilt of the terminal 403.

[0039] The terminal 403 is a touch-screen terminal with a built-in camera that can be used as the scanner 110. For example, a smartphone, tablet, or digital camera can be used. Instead of these terminals, Space Spider manufactured by Artec can also be used. FIG. 4A is a diagram showing a generalized example of the system configuration, and the configuration can be changed depending on the type of device used as the scanner 110. The terminal 403 can capture images of the model 300 and transmit them to the information processing device 100. The ring light 404 is a lighting device used when the terminal 403 captures an image of the model 300. It can evenly illuminate the model 300 and reduce the appearance of shadows. In addition to the ring light 404, additional light sources such as a top light or auxiliary lights on the left, right, and bottom sides can be installed. Instead of the terminal 403, a three-dimensional scanner device such as the one described above can also be used.

[0040] The background sheet 405 is a background sheet for photography, and a white sheet can be used, for example. The turntable 406 can mount the model 300 and rotate the model. The turntable 406 and its configuration, including its drive system, correspond to the model support device 130, and the turntable 406 rotates clockwise and counterclockwise and stops under the control of the information processing device 100. A plurality of predetermined markers 410 may be arranged on the turntable 406. The markers 410 can be used to adjust the orientation and position of the photographed model.

[0041] In FIG. 4A, the model is placed on a translucent (transparent) base, but other support devices such as an "Action Base" (registered trademark) may also be used. The Action Base has a support pole on top of the base that can be bent into a "L" shape, and the model can be attached to the tip of the pole. In this case, the above-mentioned marker 410 may be placed on the tip of the pole. The method of indicating the model can be changed depending on the pose. For example, in the case of an upright posture, the model can be placed on a transparent base and photographed. On the other hand, if it is desired to photograph the soles of the feet, such as in a flying posture, an Action Base may be used. Note that an Action Base may also be used to photograph an upright posture.

[0042] The display device 407 is a device corresponding to the display device 140 and may have a touch panel function. The user can perform a predetermined selection operation using the touch panel function. Alternatively, operations can be accepted using a controller or mouse (not shown) that can function as the operation unit 140A instead of the touch panel. The VR headset 408 is composed of an HMD 408A and a controller 408B, and corresponds to the HMDs 150A and 150B, respectively. The user can perform operations while viewing VR images by wearing the HMD 408A on their head and holding and operating the controllers 408B in their left and right hands.

[0043] Next, referring to FIG. 4B , a more specific configuration of the image processing system 10 corresponding to this embodiment when simultaneously scanning the model 300 and the accessory 311 will be described. In FIG. 4B , the model 300 and the accessory 311 are placed on a turntable 406. The model 300 is supported by an action base 411 and is located approximately in the center of the turntable 406. The accessory 311 is supported by a support member 412A in a frame 412, and is located adjacent to the model 300 and near the end of the turntable 406. The support member 412A can be, for example, a clip-shaped member. This allows the accessory 411 to be supported within the frame 412 by being clipped by the clip.

[0044] The scanner device 413 is supported by a support arm 402, and in this example, it is assumed that a Space Spider manufactured by Artec is used. The drive system for driving the support arm 402, the drive system for driving the turntable 406, etc. are the same as those described in Fig. 4A, and therefore description thereof will be omitted. In this embodiment, the turntable 406 can rotate in both clockwise and counterclockwise directions.

[0045] In this embodiment, model 300 and accessory 311 are placed together on turntable 406 in the configuration shown in Fig. 4B, and are then photographed together by scanner device 413 to generate a scanned image. The scanned image includes both model 300 and accessory 311, but the three-dimensional model data is generated by separating them.

[0046] The frame 412 that supports the accessory 311 is configured as a frame that surrounds the accessory 311 and is used as an indicator for identifying the area in which the accessory 311 is located. In other words, a three-dimensional object that exists inside the frame 412 is recognized as the accessory 311. It is desirable that the thickness of the frame 412 be as thin as possible within a range that maintains strength so as not to obstruct the line of sight of the scanner device 413 as much as possible. For example, the thickness can be about 3 mm.

[0047] Furthermore, model 300 and frame 412 are spaced apart to ensure a certain distance. Depending on the rotation angle of turntable 406, frame 412 may be positioned in front of model 300 relative to scanner device 413, blocking the capture of model 300. This blocked portion needs to be complemented at a different rotation angle, but if model 300 and frame 412 are too close, the complement accuracy will decrease, so a certain distance is maintained to ensure complement accuracy. In this case, the sizes of model 300 and accessory 311 may be limited to a predetermined size or less so that a certain distance can be maintained between model 300 and frame 412 (or so that they are not too close). For example, the size of model 300 may be 15 cm in height, 15 cm in width, and 10 cm in depth or less, and the size of accessory 311 may be 15 cm in height, 5 cm in width, and 2 cm in thickness or less.

[0048] Furthermore, in this embodiment, the installation table 414 on which the action base 411 supporting the model 300 is installed is configured to be thicker than the installation table 415 on which the frame 412 supporting the accessory 311 is installed. That is, the model 300 is installed so as to be positioned relatively higher than the accessory 311. As an example, the model 300 can be installed so that at least the upper body or the upper half thereof is positioned higher than the frame 412. This reduces the proportion of the model 300 that is blocked by the accessory 311 when scanning the model 300, thereby increasing the success rate of scanning.

[0049] Next, referring to Fig. 5, the coordinated operation of the turntable 406 and support arm 120 during scanning according to this embodiment will be described. Fig. 5 is a diagram showing the relationship between the rotation direction and amount of the turntable 406 relative to the scanner device 413, and the installation stands 414 and 415. In Fig. 5, the model 300 installed on the installation stand 414 is positioned so that the direction marked "forward" is the front. The same can be said for the accessory 311 installed on the installation stand 415.

[0050] FIG. 5(A) shows the relationship between the scanner device 413 and the installation stands 414 and 415 at the start position of the scanning operation. The rotation angle of the turntable 406 at this point is set to 0 degrees, which serves as the reference. When the turntable 406 rotates 90 degrees to the right (clockwise) from this state, the state shown in FIG. 5(B) is reached. During the transition from FIG. 5(A) to FIG. 5(B), the scanner device 413 is positioned to the side of the turntable 406 and scans the model 300 and the accessory 311. Then, in the state shown in FIG. 5(B) where the fronts of the model 300 and the accessory 311 are positioned in front of the scanner device 413, the support arm 402 is moved to the limit of its range of motion to scan the front, left and right sides, top, and bottom of the model 300 and the accessory 311.

[0051] Next, from the state shown in Fig. 5(B), the turntable 406 is rotated rightward from the 90-degree position to the 270-degree position. During this time, the model 300 and the accessory 311 are scanned. Then, in the state shown in Fig. 5(C) where the backs of the model 300 and the accessory 311 are positioned in front of the scanning device 413, the support arm 402 is moved to the limit of its range of motion, and the back, left and right sides, top, and bottom of the model 300 and the accessory 311 are scanned.

[0052] Next, from the state shown in FIG. 5(C), the turntable 406 is further rotated to the right from the 270-degree position to the 330-degree position. During this time, the model 300 and the accessories 311 are scanned. Then, the state shown in FIG. 5(D) is reached, and the support arm 402 is moved to the limit of its range of motion from diagonally behind the right of the model 300 and the accessories 311, to scan the model 300 and the accessories 311. This scan allows for a complementary scan of the portions of the exterior of the model 300 that are obscured by the frame 412 and the accessories 311 (the surfaces of the model 300 on which the frame 412 and the accessories 311 are located). Note that although scanning from diagonally behind the right is described here, scanning may also be performed from diagonally in front of the right.

[0053] Next, from the state shown in FIG. 5(D), the turntable 406 is rotated leftward (counterclockwise) from the 330-degree position to the 30-degree position. During this time, the scanning device 413 is positioned so as to provide a bird's-eye view from diagonally above the model 300 and the accessory 311, and a scan is performed. This results in the state shown in FIG. 5(E). Then, from the state shown in FIG. 5(E), the turntable 406 is rotated rightward (clockwise) from the 30-degree position to the 360-degree position. During this time, the scanning device 413 is positioned so as to provide a sideways view from diagonally below the model 300 and the accessory 311, and a scan is performed. This results in the state shown in FIG. 5(F).

[0054] In the scanner device 413 assumed for use in this embodiment, scanning is performed at a position approximately midway (for example, when the distance between the scanner device 413 and the model 300 is approximately 130 mm) within the scannable range of the scanner device 413 (for example, a distance of 90 mm to 180 mm). In this case, if the accessory 311 passes between the scanner device 413 and the model 300, there is a risk that the accessory 311 will block the model 300 and cause the scan target to be lost. However, as described above, by rotating the turntable 406 clockwise and also counterclockwise to perform scanning, and by performing scanning to complement the blocked range, it is possible to reliably obtain three-dimensional model data for each of the model 300 and the accessory 311 without losing sight of the scan target.

[0055] Next, an example of processing executed by the image processing system 10 according to this embodiment will be described with reference to Fig. 6. At least a part of the processing corresponding to this flowchart is realized by the CPU 101 of the information processing device 100 executing a program stored in the ROM 103 or the storage device 104.

[0056] First, in S601, the CPU 101 accepts user registration. The CPU 101 accepts input of the user's name and contact information. Each user is assigned a user identifier to uniquely identify the individual user. The CPU 101 stores the input user information in the storage device 104 in association with the time the input was accepted and the user identifier.

[0057] Once user registration is complete, the user sets their own model on the model support device 130. The CPU 101 can determine whether the model 300 and accessories 311 have been set on the model support device 130 based on the scanned image by the scanner 110. Alternatively, a switch that turns on when the model 300 and accessories 311 are set on the model support device 130 may be provided, and the CPU 101 may detect a signal from the switch to make the determination. Alternatively, a button for accepting an operation when the setting of the model 300 and accessories 311 is complete may be displayed on the display device 140, and the CPU 101 may detect whether the operation in response to the button operation has been accepted. In S602, the CPU 101 detects, by one of the methods described above, that the model 300 and accessories 311 have been set on the model support device 130. In response to this detection, the processing proceeds to S603.

[0058] In S603, a process is executed to receive the specification of the attributes of the model 300 and the accessories 311. Here, for example, a screen 700 as shown in Fig. 7 is displayed, and it is possible to receive the selection of the attribute (type) of the model to be scanned, the attribute (type) of the accessories, and the color (effect color) to be used to effect the subject and accessories.

[0059] The screen 700 in FIG. 7 can be displayed on, for example, the display device 140 to accept operation input from the user. In the display of the screen 700, the user can select either realistic or deformed for the model attribute 710. Because the appearance characteristics of the model differ between realistic and deformed models, the scanning method can also be selected according to the selected model attribute. For example, in the case of deformed models, the operating range of the scanner device 413 may be set narrower than in the case of realistic models. Note that the model attribute 710 may further allow the user to specify a scale value of the model (1 / 144, 1 / 100, 1 / 60, etc.). This makes it possible to further set the operating range of the scanner device 413 according to the scale of the model.

[0060] Next, for the accessory attribute 720, assuming that the accessory is a weapon, one of rifle, machine gun, and sword can be selected. Note that the weapon attributes shown in FIG. 7 are merely examples, and more detailed attributes can be specified. Also, while FIG. 7 shows only offensive weapons, attributes of defensive weapons, shields, etc. may also be specified. For the effect color 730, one of red, yellow, yellow-green, blue, and pink can be selected. The effect colors shown in FIG. 7 are merely examples, and colors other than those shown here can be specified.

[0061] A scan start button 740 is displayed on the screen 700, and the user can instruct the start of scanning when the specification of the attributes from 710 to 730 above is completed. When the start of scanning is instructed, the process proceeds to S604. At this time, the attribute information input via the screen of Fig. 7 is stored in the storage device 104 in association with the user information.

[0062] In S604, as described with reference to FIG. 5, the scanner 110, support arm 120, and model support device 130 work together to perform a scan process (photographing process) to generate a scanned image. Specifically, the CPU 101 controls the support arm 120 to move the scanner 110 to one of the registered photographing positions, and the scanner 110 can perform a scan while rotating the model support device 130 at that photographing position. Furthermore, after stopping the model support device 130 at a predetermined rotation angle, the scanner 110 can perform a photographing operation at any photographing position and photographing angle. The photographed scanned image is sent to the information processing device 100, and the CPU 101 stores the image in a table in the storage device 104 in association with a user identifier, etc.

[0063] In the next step S605, the CPU 101 performs post-scanning processing on the scanned image obtained by the scanning process, generates three-dimensional model data for the model 300 and the accessories 311, and generates application data (display data) for displaying the image. In the post-scanning processing, an area defining the range of unnecessary data is created in advance, the data is read, and surfaces included in the unnecessary area are deleted. This makes it possible to remove almost all surfaces other than those of the model 300 and the accessories 311. The unnecessary data range includes areas where the action base 411, installation stand 414, frame 412, support member 412A, installation stand 415, etc. are located.

[0064] The generated application data can play back video when executed in a specific application on the information processing device 100. Also, when executed in the specific application on the head-mounted display 150A, it can play back VR video. The generated three-dimensional model data and display data are stored in the storage device 104 in association with the user information together with the attribute information received in S603.

[0065] In the following S606, video display processing is executed. When the video display processing is performed in the information processing device 100, the CPU 101 executes a specific application corresponding to the display data and displays the video on the display device 140 via the display control unit 107. Alternatively, in S606, the CPU 101 may control the communication device 105 to transmit the display data to the HMD 150A, and the CPU 151 of the HMD 150A may execute the received display data in the corresponding application to display the VR video, thereby performing the video display processing.

[0066] When displaying the image, the effects for the subject and accessories can be switched according to the attribute information. For example, if the accessory 311 has a rifle attribute, using the accessory 311 will result in a single shot, whereas if the accessory 311 has a machine gun attribute, using the accessory 311 will allow for rapid fire. In addition, the bullets will be displayed in the color specified by the effect color.

[0067] In FIG. 6, a case where the specification of attribute information is accepted before the start of the scanning process is described, but the embodiment is not limited to this, and the specification of attribute information may be accepted after the scanning process of S604 or the post-scanning process of S605.

[0068] As described above, in this embodiment, when three-dimensional model data generated from an image obtained by photographing a model with a scanner is placed in a virtual space and display data is generated for displaying an image, three-dimensional model data is generated by simultaneously photographing the model and its accessories, and this data can be used in the virtual space together with the three-dimensional model data of the model.

[0069] In addition, in this embodiment, attribute information for accessories can be specified, so that when a three-dimensional model of the accessory is used in a virtual space, it is possible to perform a presentation according to the received attributes. This allows the user to reflect their own image in the presentation of the accessory, so that they do not feel uncomfortable. Furthermore, attribute information can also be specified for the type of model, so that by controlling the scan range according to the type of model, more accurate scanned images can be obtained.

[0070] In the above-described embodiment, the accessories are described assuming weapons, but the type of accessories is not limited to weapons. For example, if the model is a specific humanoid character, the accessories may be clothing, accessories, etc. worn by the character. It is also possible to have the model appear in events, concerts, sports, online conferences, etc. held in a virtual space. Furthermore, the technology of this embodiment can also be applied to video technology such as cross reality (XR), which merges the real world with the virtual world and allows the perception of things that do not exist in real space.

[0071] In this way, in this embodiment, three-dimensional model data is generated from photographed images of the exterior of the model and its accessories, and it becomes possible to view a video in which the three-dimensional model is placed in a virtual space. Some models and accessories, such as prefabricated plastic models, are finished as unique works by the user through painting or other processes, and this individuality can be reflected in the video and character representation in the virtual space, greatly improving the user's preferences.

[0072] <Summary of the embodiment> The above-described embodiment discloses at least the following information processing device and computer program. (1) An information processing device, a rotation control means for controlling the operation of a rotation table on which the subject is placed; a scan control means for controlling the position and angle of a scanner to scan the appearance of an object placed on the rotary table and generate a scanned image; a data generation means for generating three-dimensional model data from the scanned image; an image generating means for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; a receiving means for receiving input of attribute information relating to the subject to be scanned; Equipped with The image generating means generates the image so as to include a rendition for each of the subjects according to the attribute information. (2) The subject includes a first subject and a second subject; the first object is placed at the center of the turntable; the second subject is placed near an end of the turntable; The information processing device described in (1). (3) The information processing device according to (2), wherein the second subject is an accessory of the first subject. (4) The information processing device according to (3), wherein the first subject is a doll-type model and the second subject is an accessory part of the model. (5) The first subject is placed on a first placement means; the second subject is placed on a second placement means different from the first placement means; The information processing device according to any one of (2) to (4), wherein the first installation means installs at least a part of the first subject at a higher position than the second installation means. (6) The information processing device according to (5), wherein at least a portion of the first subject includes half of the first subject. (7) The information processing device according to (5) or (6), wherein the data generation means excludes the area where the first placement means and the second placement means are placed from the scanned image to generate three-dimensional model data of the first subject and the second subject. (8) The information processing device according to any one of (5) to (7), wherein the second setting means includes a frame that surrounds the second subject. (9) The information processing device described in (8), wherein the first subject and the frame body are positioned at a distance from each other so that the first subject is not blocked by the frame body when scanning the diagonal surface of the first subject on the side where the frame body is positioned, or each has a size equal to or smaller than a predetermined size. (10) The information processing device according to (8) or (9), wherein the second installation means supports the second subject within the frame body by a support member attached to the frame body. (11) The information processing device according to any one of (2) to (10), wherein the first subject and the second subject are placed so that their front faces face the same direction. (12) The information processing device according to any one of (2) to (11), wherein the attribute information includes attribute information of a type of the first subject and a type of the second subject. (13) The information processing device according to (12), wherein the attribute information further includes information on a color used to render at least one of the first subject and the second subject. (14) The information processing device according to (12) or (13), wherein the image generating means includes, in the image, a staged image relating to the second subject based on the attribute information. (15) The information processing device according to any one of (12) to (14), wherein the scan control means controls the position and angle of the scanner in accordance with attribute information of the type of the first subject. (16) The information processing device according to any one of (1) to (15), wherein the rotation control means rotates the turntable clockwise and counterclockwise. (17) A computer program for causing a computer to function as the information processing device according to any one of (1) to (16).

[0073] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

Claims

1. An information processing device, a rotation control means for controlling the operation of a rotation table on which the subject is placed; a scan control means for controlling the position and angle of a scanner to scan the appearance of an object placed on the rotary table and generate a scanned image; a data generation means for generating three-dimensional model data from the scanned image; an image generating means for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; a receiving means for receiving input of attribute information relating to the subject to be scanned; Equipped with The image generating means generates the image so as to include a rendition for each of the subjects according to the attribute information.

2. the subject includes a first subject and a second subject; the first object is positioned at the center of the turntable; the second subject is placed near an end of the rotating table; The information processing device according to claim 1 .

3. The information processing device according to claim 2 , wherein the second subject is an accessory of the first subject.

4. The information processing apparatus according to claim 3 , wherein the first subject is a doll-type model, and the second subject is an accessory of the model.

5. the first subject is placed on a first placement means; the second subject is placed on a second placement means different from the first placement means; The information processing apparatus according to claim 2 , wherein the first setting means sets at least a part of the first subject at a higher position than the second setting means.

6. The information processing device according to claim 5 , wherein the at least part of the first subject includes half of the first subject.

7. 6. The information processing device according to claim 5, wherein the data generating means generates three-dimensional model data of the first subject and the second subject by excluding an area where the first setting means and the second setting means are arranged from the scanned image.

8. The information processing apparatus according to claim 5 , wherein the second installation means includes a frame that surrounds the second subject.

9. 9. The information processing device according to claim 8, wherein the first subject and the frame body are arranged at a distance from each other so that the first subject is not blocked by the frame body when scanning the diagonal surface of the first subject on the side where the frame body is arranged, or each of the first subject and the frame body has a size equal to or smaller than a predetermined size.

10. The information processing apparatus according to claim 8 , wherein the second installation means supports the second subject within the frame by a support member attached to the frame.

11. The information processing apparatus according to claim 2 , wherein the first subject and the second subject are placed so that their front faces face the same direction.

12. The information processing device according to claim 2 , wherein the attribute information includes attribute information of a type of the first subject and a type of the second subject.

13. The information processing device according to claim 12 , wherein the attribute information further includes information on a color used to render at least one of the first subject and the second subject.

14. The information processing device according to claim 12 , wherein the image generating means includes, in the image, a staged image relating to the second subject based on the attribute information.

15. The information processing apparatus according to claim 12 , wherein the scan control means controls the position and angle of the scanner in accordance with attribute information of the type of the first subject.

16. The information processing apparatus according to claim 1 , wherein the rotation control means rotates the rotary table clockwise and counterclockwise.

17. A computer program for causing a computer to function as the information processing device according to any one of claims 1 to 16.

Images