Information processing device and computer program

Abstract

To provide an information processing device and an information processing program configured to image a target and generate a video in which a three-dimensional model is arranged on a virtual space using three-dimensional model data of the imaged target, in accordance with attribute information, while associating the generated three-dimensional model data with the attribute information of the imaged target.SOLUTION: In an image processing system 10, an information processing device includes: rotation control means configured to control the motion of a turn table on which a subject is placed; scan control means which controls a position and an angle of the scanner and scans an outer appearance of the subject to generate a scan image; data generation means configured to generate three-dimensional model data from the scan image; video generation means configured to generate a video in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space so as to include an effect for each subject according to attribute information; and receiving means configured to receive an input of attribute information regarding the subject to be scanned.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device and a computer program. [Background technology]

[0002] Patent document 1 describes a technology in which a texture generated from a group of captured images of an object taken from multiple angles by a photographing unit is mapped onto a primitive in a virtual space to generate a virtual space image as seen from a virtual camera in the virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-107251 A Summary of the Invention [Problem to be solved by the invention]

[0004] In generating virtual space images using the above technology, when an object is photographed and three-dimensional model data of the photographed object is used to generate an image in which a three-dimensional model is placed in a virtual space, it is expected that the generated three-dimensional model data will be associated with attribute information of the photographed object, making it possible to generate an image corresponding to the attribute information.

[0005] Therefore, when photographing an object and using three-dimensional model data of the photographed object to generate an image in which a three-dimensional model is placed in a virtual space, the object is to associate the generated three-dimensional model data with attribute information of the photographed object, thereby making it possible to generate an image corresponding to the attribute information. [Means for solving the problem]

[0006] One aspect of the present invention to solve the above problem is an information processing device, A rotation control means for controlling the operation of a rotation platform on which a subject is placed; a scan control means for controlling a position and an angle of a scanner to scan the external appearance of a subject placed on the rotating table and generate a scanned image; a data generating means for generating three-dimensional model data from the scanned image; an image generating means for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; an output means for outputting the image generated by the image generating means to a display device; a receiving means for receiving input of attribute information relating to a subject to be scanned; Equipped with The image generating means generates the image so as to include a rendition for each of the subjects according to the attribute information. Effect of the Invention

[0007] When photographing an object and using three-dimensional model data of the photographed object to generate an image in which a three-dimensional model is placed in a virtual space, the three-dimensional model data can be generated in association with attribute information of the photographed object, and an image can be generated according to the attribute information. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the arrangement of an image processing system 10 according to an embodiment. [Diagram 2] 1 is a diagram showing an example of a hardware configuration of an information processing device 100 and an example of a hardware configuration of a headset 150 according to an embodiment. [Diagram 3] FIG. 1 is a diagram showing an example of the appearance of a doll-type model according to an embodiment. [Figure 4A] FIG. 1 is a diagram showing an example of an implementation of an image processing system 10. [Figure 4B] FIG. 2 is a diagram showing an example of the configuration of the image processing system 10 according to the embodiment when performing scan photography. [Diagram 5] 4A and 4B are diagrams for explaining the relationship between the rotational positions of a scanner device 413 and a turntable 406 during scanning and photographing according to the embodiment. [Figure 6]4 is a flowchart corresponding to an example of processing in the image processing system 10 according to the embodiment. [Figure 7] FIG. 13 is a diagram showing an example of an attribute designation screen according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. Also, the same reference numbers are used for identical or similar configurations, and duplicated descriptions are omitted. Also, in each figure, the up, down, left, right, front and back directions relative to the paper surface are used in the description in the text as the up, down, left, right, front and back directions of the parts (or components) in this embodiment.

[0010] First, the configuration of an image processing system corresponding to this embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of an image processing system 10 corresponding to this embodiment. The image processing system 10 is configured by connecting a scanner 110, a support arm 120, a model support device 130, a display device 140, a headset 150, and the like to an information processing device 100. Note that the system configuration is not limited to that shown in FIG. 1, and the information processing device 100 may be further connected to an external server, a cloud server, and the like via a network. The external server, etc., can execute at least a part of the processing of the embodiment described below.

[0011] The information processing device 100 controls the operation of at least one of the scanner 110, the support arm 120, and the model support device 130, photographs the object to be photographed from any angle to generate multiple images, and generates three-dimensional model data (main data) from the multiple images. If the object to be photographed is configured to be separable into multiple component objects, each component object may be photographed to generate three-dimensional model data, and the three-dimensional model data of the object may be generated by integrating the photographs.

[0012] The information processing device 100 can also function as an image generating device that generates images to be displayed in a virtual space by using the generated three-dimensional model data as virtual space data. In this embodiment, the subjects to be photographed are articles such as an assemblable plastic model, an action figure (a figure with movable joints), a toy, a doll, etc., which are collectively referred to as "models" below.

[0013] Next, the scanner 110 is a three-dimensional scanner device that captures (scans) the three-dimensional shape of a model to be captured under the control of the information processing device 100, and outputs the three-dimensional shape and color information of the captured object. In this embodiment, the scan signals output from the scanner 110 are collectively called "scanned images." For example, Space Spider manufactured by Artec can be used as the scanner 110. In this embodiment, for example, 500 to 800 frames of scanned images can be acquired to obtain 3D scan data of the entire toy body. In addition, for example, a smartphone with a camera on which an application for capturing three-dimensional shapes is installed may be used as the scanner 110.

[0014] The support arm 120 is a position and orientation control device that moves the scanner 110 to a predetermined photographing position and orientation under the control of the information processing device 100. The support arm 120 may be configured to be capable of manually changing the photographing position and orientation, and to be capable of fixing the changed position and orientation, or may be configured to be controllable by the information processing device 100. When the support arm 120 is configured to be controllable by the information processing device 100, for example, xArm 7 manufactured by UFACTORY can be used. The xArm 7 is made of seven joints and can move in the same manner as a human arm. Note that instead of using the support arm 120, the scanner 110 may be positioned manually. For photographing positions and orientations that cannot be covered by the support arm 120, the scanner 110 may be manually operated to perform scanning.

[0015] The model support device 130 is a support base (or a mounting base on which the model is mounted) that supports a model with a fixed pose. The model support device 130 may be configured to be rotatable, for example, when the model is mounted on the support base or at the tip of a support rod. In this embodiment, the scanner 110 is positioned at an arbitrary photographing position and photographing angle by the support arm 120, and then the model support device 130 is rotated to perform photographing. The model support device 130 can rotate both clockwise and counterclockwise. In addition, the model support device 130 can be stopped at an arbitrary rotation angle within the range of 0 degrees to 360 degrees, and the model support device 130 can be stopped at a specific rotation angle and positioned at an arbitrary photographing position and photographing angle within the movable range of the scanner 110 to perform photographing. By performing this at a plurality of rotation angles, photographing positions, and photographing angles, an image of the entire model can be obtained. In addition, by driving the support arm 120 and the model support device 130 synchronously, the photographing process can be performed more easily and with high accuracy. Also, instead of using the model support device 130, the scanner 110 may be manually moved around the model to scan from any desired imaging position and angle.

[0016] The display device 140 is a display device such as a liquid crystal display (LCD), and can display the processing results of the information processing device 100. Specifically, it can display an image acquired by the scanner 110, display the main data of the three-dimensional model data generated from the captured image, and display an image (VR image) reconstructed using the main data. The display device 140 can include an operation unit 140A that accepts operations from a user who is an observer of the displayed image, and the user can operate the operation unit 140A to perform an operation input according to the content of the image displayed on the display device 140.

[0017] The headset 150 is composed of a head mounted display 150A and a controller 150B, which will be described later. In particular, the headset 150 may be configured as a VR headset capable of providing moving images corresponding to the posture and inclination of a user who is an observer. A specific application is installed in the headset 150, and application data executed in the application can be downloaded from the information processing device 100 and executed. In this embodiment, the headset 150, like the information processing device 100, can function as an image generating device that uses three-dimensional model data as virtual space data to generate an image to be displayed in a virtual space and provides the generated image as a VR image. The application data includes display data for displaying the VR image.

[0018] Next, the hardware configuration of the information processing device 100 corresponding to this embodiment will be described. FIG. 2(A) shows an example of the hardware configuration of the information processing device 100. The CPU 101 is a device that performs overall control of the information processing device 100 and performs calculation, processing, and management of data. For example, the CPU 101 controls the image capture timing and number of images captured by the scanner 110, and controls the arm joint of the support arm 120 to place the scanner 110 at an arbitrary image capture position and image capture angle. For example, after the image capture position and image capture angle of the scanner 110 are determined, the model support device 130 can be rotated to perform an image capture operation by the scanner 110. In addition, after the model support device 130 is stopped at a predetermined rotation angle, the scanner 110 can perform an image capture operation at an arbitrary image capture position and image capture angle.

[0019] CPU 101 can also function as an image processing unit that processes images output from scanner 110. Specifically, CPU 101 can perform processing to generate an image that displays a three-dimensional model in a virtual space by using three-dimensional model data generated based on a scan signal obtained by scanner 110 as virtual space data.

[0020] The RAM 102 is a volatile memory and is used as a temporary storage area such as the main memory and work area of ​​the CPU 101. The ROM 103 is a non-volatile memory, and image data and other data, various programs for the operation of the CPU 101, and the like are stored in respective predetermined areas. The CPU 101 controls each part of the information processing device 100 using the RAM 102 as a work memory according to, for example, a program stored in the ROM 103. Note that the program for the operation of the CPU 101 is not limited to being stored in the ROM 103, and may be stored in the storage device 104.

[0021] The storage device 104 is composed of a magnetic disk such as a HDD or a flash memory. Application programs, an OS, control programs, related programs, game programs, etc. are stored in the storage device 104. The storage device 104 can read and write data under the control of the CPU 101. The storage device 104 may be used in place of the RAM 102 or the ROM 103.

[0022] The communication device 105 is a communication interface for communicating with the scanner 110, the support arm 120, the model support device 130, the display device 140, and the headset 150 under the control of the CPU 101. The communication device 105 may further be configured to be able to communicate with an external server or the like. The communication device 105 may include a wireless communication module, which may include known circuitry including an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module card, a memory, and the like. Here, the communication between the information processing device 100 and the scanner 110, the support arm 120, the model support device 130, the display device 140, and the headset 150 may be performed by wireless communication.

[0023] The communication device 105 may also include a wired communication module for wired connection. The wired communication module enables communication with other devices including the display device 140 via one or more external ports. The communication device 105 may also include various software components for processing data. The external ports are connected to other devices directly via Ethernet, USB, IEEE1394, etc., or indirectly via a network. Note that software that realizes the same functions as the above devices may be used to replace the hardware devices.

[0024] The operation unit 106 is composed of, for example, a button, a keyboard, a touch panel, a controller, etc., and accepts operation input from a user. The operation unit 106 may be shared with the operation unit 140A, or may be independent of the operation unit 140A. For example, when the operation unit 140A is assumed to be an operation unit composed of a keyboard and a mouse, etc., it can be shared with the operation unit 106. On the other hand, when the operation unit 140A is assumed to be an operation unit composed of a touch panel, a controller, etc., it can be an operation unit separate from the operation unit 106.

[0025] The display control unit 107 functions as a control unit for displaying information on the display device 140 connected to the information processing device 100, and controls the operation of the display device 140. A part of the functions of the operation unit 106 may be provided in the display device 140. For example, the display device 140 may be configured as a device equipped with a touch panel such as a tablet terminal.

[0026] Next, a hardware configuration of the headset 150 corresponding to this embodiment will be described. FIG. 2B shows an example of the hardware configuration of the headset 150. The headset 150 is composed of a head mounted display (HMD) 150A and a controller 150B. The HMD 150A provides a right eye image and a left eye image to the left and right eyes of the user, respectively, and enables a virtual reality (VR) experience by creating a sense of stereoscopic effect due to the parallax between them. The controller 150B is provided in a housing independent of the HMD 150A. The controller 150B is assumed to be configured as a pair of two controllers to be held and operated by the user's left and right hands, respectively, but may be a single controller.

[0027] The CPU 151 is a device that performs overall control of the headset 150 and calculates, processes, and manages data. For example, it can execute application data downloaded from the information processing device 100 to display VR video on the display 157. In addition, based on an operation accepted via the controller 156 or information detected by the detection unit 158, it can switch the VR video itself to be displayed, switch the viewpoint within the VR space, or change the position within the VR space.

[0028] RAM 152 is a volatile memory and is used as a temporary storage area such as the main memory and work area of ​​CPU 151. ROM 153 is a non-volatile memory, and image data and other data, various programs for CPU 151 to operate, and the like are stored in respective predetermined areas. CPU 151 controls each part of headset 150 using RAM 152 as a work memory according to a program stored in ROM 153, for example. Note that the program for CPU 151 to operate is not limited to being stored in ROM 153, and may be stored in storage device 154.

[0029] The storage device 154 is composed of a magnetic disk such as a HDD or a flash memory. The storage device 154 stores application programs, an OS, control programs, related programs, game programs, application data downloaded from the information processing device 100, display data, and the like. The storage device 154 can read and write data based on the control of the CPU 151. The storage device 154 may be used instead of the RAM 152 or the ROM 153.

[0030] The communication device 155 is a communication interface for communicating with the information processing device 100 and the controller 150B under the control of the CPU 151. The communication device 155 includes a wireless communication module that enables wireless communication based on Bluetooth or WiFi (IEEE802.11). The headset 150 can connect to the information processing device 100 via wireless communication and download display data for VR images. In addition, the headset 150 communicates with the controller 150B and receives information on user operation instructions for the controller 150B.

[0031] The display 156 is configured to provide the right eye image and the left eye image generated by the CPU 151 to the right eye and the left eye of the user, respectively. The detection unit 157 is a mechanism for detecting the gaze direction of the left and right eyes of the user and the tilt of the head mounted display. The detection unit 157 includes, for example, a sensor for detecting the gaze direction or the gaze direction of the user wearing the HMD 150A. It also includes a gyroscope, a magnetometer, an accelerometer, a global positioning system (GPS), a compass, etc., and can specify the position, attitude, tilt, etc. of the HMD 150A according to the detection information. The detection unit 157 detects information for determining the gaze direction and action of the user in the HMD 150A and transmits it to the CPU 151. The CPU 151 determines the gaze direction and the action based on the received detection information, and adjusts the image presented on the display 156 of the HMD 150A so as to match the determined gaze direction and the action.

[0032] The controller 150B is configured with, for example, a plurality of buttons, a cross key, etc., and receives operation inputs from the user such as selection operations, direction instruction operations, etc. The controller 150B is connected to the HMD 150A via a communication device 155 via wireless communication.

[0033] Next, referring to FIG. 3, an example of a model and its accessories that are the subject of the photographing target in this embodiment will be described. The model 300 is a model with a doll-like (robot or human) appearance. The model can be, for example, a plastic model that has been assembled and painted. Alternatively, it may be a completed model such as a figure (action figure) with movable joints. The model in FIG. 2 is merely an example for the purpose of explanation, and the shape of the model is not limited to a doll-like appearance, but can be any shape such as a general vehicle, a racing vehicle, a military vehicle, an aircraft, a ship, an animal, a virtual life form, etc. It is to be noted that the object to be photographed is not limited to a model as long as the object can be photographed three-dimensionally by the scanner 110.

[0034] The model 300 has each of the parts of a head 301, a chest 302, a right arm 303, a left arm 304, a right torso 305, a left torso 306, a right leg 307, a left leg 308, a right foot 309, and a left foot 310, which are joined together to form the model. At least a part of each of the parts 301 to 310 is supported so as to be rotatable (or swingable) relative to the adjacent parts. For example, the head 301 is supported so as to be rotatable relative to the chest 302, and the right arm 303 and the left arm 304 are supported so as to be rotatable relative to the chest 302. In this way, each part of the model 300 is provided with a joint structure, so that the model 300 can take any posture.

[0035] In addition, the model 300 is accompanied by an accessory 311. The accessory 311 is an accessory part that is expected to be used with the model 300, and in FIG. 3, it is a weapon having a rifle shape. That is, it is expected that the model 300 will use the accessory 311 by holding it with both hands. The shape of the weapon is not limited to the rifle shape, and may include any weapon shape such as a sword shape, a shield shape, a bomb shape (e.g., a grenade), a bazooka shape, etc. In addition, the accessory is not limited to a weapon, and may be something like an ornament, so long as it is an accessory part that can be used with the model 300. For example, if the model 300 is a doll-type model of a specific character, the accessory may be any item carried by the character (e.g., carry-on items, hats, bags, accessories, smartphones, or any other items that the character can carry).

[0036] Next, an overall implementation example of the image processing system 10 in this embodiment will be described with reference to Fig. 4A. Fig. 4A is a diagram showing a generalized implementation example capable of generating the main data generated by photographing a model.

[0037] 4A, case 401 contains information processing device 100, a drive system for driving support arm 402, a drive system for driving turntable 406, and the like. Note that support arm 402 can also be manually adjusted in shooting direction and position. The surface of case 401 has a flat structure so that a promotional poster can be attached.

[0038] The support arm 402 corresponds to the support arm 120, and can support and fix the position of a terminal 403 functioning as the scanner 110 under the control of the information processing device 100 or manually. The support arm 402 can also operate to control the inclination of the terminal 403.

[0039] The terminal 403 is a touch panel terminal with a built-in camera that can be used as the scanner 110, and can be, for example, a smartphone, a tablet terminal, or a digital camera. Instead of these terminals, Space Spider manufactured by Artec can be used. FIG. 4A is merely a diagram showing a generalized example of the system configuration, and the configuration can be changed according to the type of device used as the scanner 110. The terminal 403 can take an image of the model 300 and transmit it to the information processing device 100. The ring light 404 is a lighting device used when the terminal 403 takes an image of the model 300, and can apply light evenly to the model 300 to make it difficult to cast a shadow. In addition to the ring light 404, a top light or auxiliary lights on the left, right, and bottom may be installed as an additional light source. Instead of the terminal 403, a three-dimensional scanner device as described above may be used.

[0040] The background sheet 405 is a background sheet for photography, and for example, a white sheet can be used. The turntable 406 can mount the model 300 and rotate the model. The turntable 406 and its configuration including its drive system correspond to the model support device 130, and the turntable 406 rotates clockwise and counterclockwise and stops under the control of the information processing device 100. A plurality of predetermined markers 410 may be arranged on the turntable 406. The markers 410 can be used to adjust the orientation and position of the photographed model.

[0041] In FIG. 4A, the model is placed on a semi-transparent (transparent) base, but other support devices such as an "Action Base" (registered trademark) may be used. The Action Base has a support pole configured to be bent into an "L" shape on a base, and the model can be attached to the tip of the pole. In this case, the above-mentioned marker 410 may be placed on the tip of the pole. The method of indicating the model can be changed depending on the pose. For example, in the case of an upright posture, the model can be placed on a transparent base and photographed. On the other hand, in the case of a flying posture, when it is desired to photograph the soles of the feet, an action base may be used. The action base may be used to photograph an upright posture.

[0042] The display device 407 is a device corresponding to the display device 140, and may have a touch panel function. The user can use the touch panel function to perform a predetermined selection operation. Also, operations can be accepted by a controller or mouse (not shown) that can function as the operation unit 140A instead of the touch panel. The VR headset 408 is composed of an HMD 408A and a controller 408B, and corresponds to the HMDs 150A and 150B, respectively. The user can perform operations while watching VR images by wearing the HMD 408A on the head and holding and operating the controller 408B in each hand.

[0043] Next, referring to FIG. 4B, a more specific configuration of the image processing system 10 in the case where the model 300 and the accessory 311 are scanned simultaneously, which corresponds to this embodiment, will be described. In FIG. 4B, the model 300 and the accessory 311 are placed on the turntable 406. The model 300 is supported by the action base 411 and is disposed approximately in the center of the turntable 406. The accessory 311 is supported by a support member 412A in the frame 412, and is disposed adjacent to the model 300 and near the end of the turntable 406. The support member 412A can be, for example, a clip-shaped member. This allows the accessory 411 to be supported in the frame 412 by being clipped by the clip.

[0044] The scanner device 413 is supported by a support arm 402, and it is assumed here that a Space Spider manufactured by Artec is used. The drive system for driving the support arm 402 and the drive system for driving the turntable 406 are the same as those described in Fig. 4A, and therefore description thereof will be omitted. In this embodiment, the turntable 406 can rotate in both clockwise and counterclockwise directions.

[0045] In this embodiment, model 300 and accessories 311 are placed together on turntable 406 in the configuration shown in Fig. 4B, and are photographed together by scanner device 413 to generate a scanned image. The scanned image includes both model 300 and accessories 311, but the three-dimensional model data is generated by separating them.

[0046] The frame 412 supporting the accessory 311 is configured as a frame surrounding the accessory 311 and is used as an index for identifying the range in which the accessory 311 is arranged. That is, a three-dimensional object present inside the frame 412 is recognized as the accessory 311. It is desirable that the thickness of the frame 412 is as thin as possible within a range in which strength is maintained so as not to block the line of sight of the scanner device 413 as much as possible. For example, the thickness can be about 3 mm.

[0047] Furthermore, model 300 and frame 412 are spaced apart to ensure a certain distance. Depending on the rotation angle of turntable 406, frame 412 may be positioned in front of model 300 relative to scanner device 413, blocking the photography of model 300. The blocked portion needs to be complemented at a different rotation angle, but if model 300 and frame 412 are too close, the complementation accuracy decreases, so a certain distance is maintained to ensure complementation accuracy. In this case, the sizes of model 300 and accessory 311 may be limited to a predetermined size or less so that a certain distance can be secured between model 300 and frame 412 (or so that they are not too close). For example, the size of model 300 may be 15 cm long, 15 cm wide, and 10 cm deep or less, and the size of accessory 311 may be 15 cm long, 5 cm wide, and 2 cm thick or less.

[0048] In this embodiment, the installation table 414 on which the action base 411 supporting the model 300 is installed is configured to be thicker than the installation table 415 on which the frame 412 supporting the accessory 311 is installed. That is, the model 300 is installed so as to be located relatively higher than the accessory 311. As an example, at least the upper body or the upper half of the model 300 can be installed so as to be located higher than the frame 412. This reduces the proportion of the model 300 that is blocked by the accessory 311 when scanning the model 300, and increases the success rate of scanning.

[0049] Next, referring to Fig. 5, a description will be given of the coordinated operation between the turntable 406 and the support arm 120 during scanning according to this embodiment. Fig. 5 is a diagram showing the relationship between the rotation direction and amount of rotation of the turntable 406 relative to the scanner device 413, and the installation tables 414 and 415. In Fig. 5, the model 300 installed on the installation table 414 is arranged so that the direction marked "forward" is the front. The same can be said for the accessory 311 arranged on the installation table 415.

[0050] FIG. 5(A) shows the relationship between the scanner device 413 and the installation tables 414 and 415 at the start position of the scanning operation. The rotation angle of the turntable 406 at this point is set to 0 degrees, which is the reference. When the turntable 406 rotates 90 degrees to the right (clockwise) from this state, the state shown in FIG. 5(B) is reached. During the transition from FIG. 5(A) to FIG. 5(B), the scanner device 413 is positioned to the side of the turntable 406 and scans the model 300 and the accessories 311. Then, in the state shown in FIG. 5(B) where the fronts of the model 300 and the accessories 311 are positioned in front of the scanning device 413, the support arm 402 is operated to the limit of its range of motion to scan the front, left and right side surfaces, top surface, and bottom surface of the model 300 and the accessories 311.

[0051] Next, from the state shown in Fig. 5(B), turntable 406 is rotated rightward from the 90 degree position to the 270 degree position. During this time, model 300 and accessories 311 are scanned. Then, in the state shown in Fig. 5(C) where the backs of model 300 and accessories 311 are positioned in front of scanning device 413, support arm 402 is operated to the limit of its range of motion to scan the back, left and right side surfaces, top surface, and bottom surface of model 300 and accessories 311.

[0052] Next, from the state shown in FIG. 5(C), the turntable 406 is further rotated to the right from the 270° position to the 330° position. During this time, the model 300 and the accessories 311 are scanned. Then, the state shown in FIG. 5(D) is reached, and the support arm 402 is moved to the limit of its range of motion from the right rear of the model 300 and the accessories 311 to scan the model 300 and the accessories 311. This scan makes it possible to scan the parts of the exterior of the model 300 that are shielded by the frame 412 and the accessories 311 (the surfaces of the model 300 on the side where the frame 412 and the accessories 311 are arranged) so as to complement the parts. Note that, although the case of scanning from the right rear is described here, scanning from the right front may also be performed.

[0053] Next, from the state shown in FIG. 5(D), the turntable 406 is rotated leftward (counterclockwise) from the 330 degree position to the 30 degree position. During this time, the scanning device 413 is positioned so as to be a bird's-eye view from diagonally above the model 300 and the accessories 311, and scanning is performed. This results in the state shown in FIG. 5(E). Then, from the state shown in FIG. 5(E), the turntable 406 is rotated rightward (clockwise) from the 30 degree position to the 360 ​​degree position. During this time, the scanning device 413 is positioned so as to be a sideways view from diagonally below the model 300 and the accessories 311, and scanning is performed. This results in the state shown in FIG. 5(F).

[0054] In the scanner device 413 assumed for use in this embodiment, scanning is performed at a position approximately midway (for example, the distance between the scanner device 413 and the model 300 is about 130 mm) relative to the scanable range of the scanner device 413 (for example, a distance of 90 mm to 180 mm). In this case, if the accessory 311 passes between the scanner device 413 and the model 300, there is a risk of a problem in that the accessory 311 blocks the model 300 and the target to be scanned is lost. However, as described above, by rotating the turntable 406 in the right direction and also in the left direction in the reverse direction to perform scanning, and by performing scanning to complement the blocked range, it is possible to reliably obtain three-dimensional model data of the model 300 and the accessory 311 without losing sight of the target to be scanned.

[0055] Next, an example of processing executed by the image processing system 10 according to the present embodiment will be described with reference to Fig. 6. At least a part of the processing corresponding to the flowchart is realized by the CPU 101 of the information processing device 100 executing a program stored in the ROM 103 or the storage device 104.

[0056] First, in S601, the CPU 101 accepts user registration. The CPU 101 accepts input of the user's name and contact information. A user identifier is assigned to each user to uniquely identify each individual user. The CPU 101 stores the input user information in the storage device 104 in association with the time when the input was accepted and the user identifier.

[0057] When user registration is complete, the user sets his / her own model on the model support device 130. The CPU 101 can determine whether or not the model 300 and the accessories 311 have been set on the model support device 130 based on the scanned image by the scanner 110. Alternatively, a switch that is turned on when the model 300 and the accessories 311 are set on the model support device 130 may be provided, and the CPU 101 may detect a signal from the switch to make the determination. Alternatively, a button for accepting an operation when the setting of the model 300 and the accessories 311 is completed may be displayed on the display device 140, and the CPU 101 can detect whether or not the operation in response to the button operation has been accepted. In S602, the CPU 101 detects that the model 300 and the accessories 311 have been set on the model support device 130 by any of the above-mentioned methods. In response to this detection, the process proceeds to S603.

[0058] In S603, a process is executed to accept the specification of attributes of the model 300 and the accessories 311. Here, for example, a screen 700 as shown in Fig. 7 is displayed, and it is possible to accept the selection of the attribute (type) of the model to be scanned, the attribute (type) of the accessories, and the color (effect color) to be used to effect the subject and the accessories.

[0059] The screen 700 in FIG. 7 can be displayed on the display device 140, for example, to accept operation input from the user. In the display of the screen 700, the user can select either real or deformed for the model attribute 710. Since the external characteristics of the model differ between real and deformed models, the scanning method can also be selected according to the selected model attribute. For example, in the case of deformation, the operating range of the scanner device 413 may be set narrower than in the case of real. Note that the model attribute 710 may further allow the scale value of the model (1 / 144, 1 / 100, 1 / 60), etc. to be specified. This makes it possible to further set the operating range of the scanner device 413 according to the scale of the model.

[0060] Next, for the accessory attribute 720, assuming that the accessory is a weapon, one of a rifle, a machine gun, and a sword can be selected. Note that the weapon attributes shown in FIG. 7 are merely examples, and more detailed attributes can be specified. Also, while FIG. 7 shows only offensive weapons, it is also possible to specify attributes of defensive weapons, shields, and the like. For the effect color 730, one of red, yellow, light green, blue, and pink can be selected. The effect color shown in FIG. 7 is merely an example, and colors other than those shown here can be specified.

[0061] A start scan button 740 is displayed on the screen 700, and the user can instruct the start of scanning when the specification of attributes from 710 to 730 above is completed. When the start of scanning is instructed, the process proceeds to S604. At this time, the attribute information input via the screen of Fig. 7 is stored in the storage device 104 in association with the user information.

[0062] In S604, as described with reference to Fig. 5, the scanner 110, the support arm 120, and the model support device 130 cooperate to execute the scan process (photographing process) to generate a scan image. Specifically, the CPU 101 controls the support arm 120 to move the scanner 110 to any of the registered photographing positions, and the scanner 110 can perform scanning while rotating the model support device 130 at the photographing position. In addition, after the model support device 130 is stopped at a predetermined rotation angle, the scanner 110 can perform a photographing operation at any photographing position and photographing angle. The photographed scan image is transmitted to the information processing device 100, and the CPU 101 stores the image in a table in the storage device 104 in association with a user identifier, etc.

[0063] In the next S605, the CPU 101 executes post-scan processing on the scanned image acquired by the scan processing, generates three-dimensional model data for each of the model 300 and the accessories 311, and generates application data (display data) for video display. In the post-scan processing, an area defining the range of unnecessary data is created in advance, the data is read, and the surfaces included in the unnecessary area are deleted. This makes it possible to remove almost all surfaces other than those of the model 300 and the accessories 311. The unnecessary data range includes areas where the action base 411, the installation stand 414, the frame 412, the support member 412A, the installation stand 415, etc. are located.

[0064] The generated application data can play back images when executed in a specific application on the information processing device 100. Also, when executed in the specific application on the head mounted display 150A, it can play back VR images. The generated three-dimensional model data and display data are stored in the storage device 104 in association with the user information together with the attribute information received in S603.

[0065] In the next step S606, video display processing is executed. When the video display processing is executed in the information processing device 100, the CPU 101 executes a specific application corresponding to the display data, and displays the video on the display device 140 via the display control unit 107. In addition, in step S606, the CPU 101 may control the communication device 105 to transmit the display data to the HMD 150A, and the CPU 151 of the HMD 150A may execute the received display data in the corresponding application to execute processing for displaying the VR video, thereby executing the video display processing.

[0066] When displaying the image, the effects for the subject and accessories can be switched according to the attribute information. For example, if the accessory 311 has a rifle attribute, a single shot will be fired when the accessory 311 is used, whereas if the accessory 311 has a machine gun attribute, continuous fire will be possible when the accessory 311 is used. In addition, the bullets are displayed in a color specified by the effect color.

[0067] In FIG. 6, a case has been described in which the specification of attribute information is accepted before the start of scanning processing, but the embodiment is not limited to this, and the specification of attribute information may be accepted after the scanning processing of S604 or the post-scanning processing of S605.

[0068] As described above, in this embodiment, when generating display data for displaying an image by placing three-dimensional model data generated from an image obtained by photographing a model with a scanner in a virtual space, the model and its accessories are photographed at the same time to generate three-dimensional model data that can be used in the virtual space together with the three-dimensional model data of the model.

[0069] In addition, in this embodiment, since attribute information can be specified for accessories, when a three-dimensional model of the accessory is used in a virtual space, it is possible to perform a presentation according to the received attributes. This allows the user to reflect his or her own image in the presentation of the accessory, so that the user does not feel uncomfortable. Furthermore, since attribute information can be specified for the type of model, the scan range can be controlled according to the type of model, so that a more accurate scanned image can be obtained.

[0070] In the above embodiment, the description has been given assuming that the accessory is a weapon, but the type of accessory is not limited to a weapon. For example, if the model is a specific humanoid character, the accessory may be a costume or accessory worn by the character. It is also possible to have the model appear in events, concerts, sports, online conferences, etc. held in a virtual space. Furthermore, the technology of this embodiment can be applied to a video technology such as cross reality (XR) that fuses the real world with the virtual world and makes it possible to perceive things that do not exist in the real space.

[0071] In this way, in this embodiment, three-dimensional model data is generated from images of the exterior of the model and its accessories, and it becomes possible to view a video in which the three-dimensional model is placed in a virtual space. Some models and accessories, such as prefabricated plastic models, are finished as unique works by users through painting or the like, and their individuality can be reflected in the character representation in the video and virtual space, which can significantly improve the preference of the users.

[0072] <Summary of the embodiment> The above embodiment discloses at least the following information processing device and computer program. (1) An information processing device, A rotation control means for controlling the operation of a rotation platform on which a subject is placed; a scan control means for controlling a position and an angle of a scanner to scan the external appearance of a subject placed on the rotating table and generate a scanned image; a data generating means for generating three-dimensional model data from the scanned image; an image generating means for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; a receiving means for receiving input of attribute information relating to a subject to be scanned; Equipped with The image generating means generates the image so as to include a rendition for each of the subjects according to the attribute information. (2) the subject includes a first subject and a second subject; the first object is placed at the center of the turntable; the second subject is placed near an end of the rotating table; An information processing device as described in (1). (3) The information processing device according to (2), wherein the second subject is an accessory of the first subject. (4) The information processing device according to (3), wherein the first subject is a doll-type model and the second subject is an accessory of the model. (5) The first subject is placed on a first placement means; the second subject is placed on a second placement means different from the first placement means; The information processing device according to any one of (2) to (4), wherein the first setting means sets at least a part of the first subject at a higher position than the second setting means. (6) The information processing device according to (5), wherein the at least a portion of the first subject includes half of the first subject. (7) The information processing device described in (5) or (6), wherein the data generation means excludes an area in which the first installation means and the second installation means are positioned from the scanned image, and generates three-dimensional model data of the first subject and the second subject. (8) The information processing device according to any one of (5) to (7), wherein the second installation means includes a frame that surrounds the second subject. (9) An information processing device as described in (8), wherein the first subject and the frame body are positioned at a distance from each other so that the first subject is not blocked by the frame body when scanning the diagonal surface of the first subject on the side on which the frame body is positioned, or each has a size equal to or smaller than a predetermined size. (10) The information processing device according to (8) or (9), wherein the second installation means supports the second subject within the frame body by a support member attached to the frame body. (11) The information processing device according to any one of (2) to (10), wherein the first subject and the second subject are placed so that their front faces face the same direction. (12) The information processing device according to any one of (2) to (11), wherein the attribute information includes attribute information of a type of the first subject and a type of the second subject. (13) The information processing device according to (12), wherein the attribute information further includes color information used in rendering at least one of the first subject and the second subject. (14) The information processing device according to (12) or (13), wherein the image generating means includes, in the image, a staged image relating to the second subject based on the attribute information. (15) The information processing device according to any one of (12) to (14), wherein the scan control means controls a position and an angle of the scanner in accordance with attribute information of a type of the first subject. (16) The information processing device according to any one of (1) to (15), wherein the rotation control means rotates the turntable clockwise and counterclockwise. (17) A computer program for causing a computer to function as an information processing device according to any one of (1) to (16).

[0073] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

Claims

1. An information processing device, A rotation control means for controlling the operation of a rotation platform on which a subject is placed; a scan control means for controlling a position and an angle of a scanner to scan the external appearance of a subject placed on the rotating table and generate a scanned image; a data generating means for generating three-dimensional model data from the scanned image; an image generating means for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; a receiving means for receiving input of attribute information relating to a subject to be scanned; Equipped with The image generating means generates the image so as to include a rendition for each of the subjects according to the attribute information.

2. the subject includes a first subject and a second subject, the first object is positioned at the center of the turntable; The second object is disposed near an end of the rotating table. The information processing device according to claim 1 .

3. The information processing apparatus according to claim 2 , wherein the second subject is an accessory of the first subject.

4. The information processing apparatus according to claim 3 , wherein the first subject is a doll-type model, and the second subject is an accessory of the model.

5. the first subject is placed on a first placement means; the second subject is placed on a second placement means different from the first placement means; The information processing apparatus according to claim 2 , wherein the first setting means sets at least a part of the first subject at a higher position than the second setting means.

6. The information processing device according to claim 5 , wherein the at least a portion of the first subject includes half of the first subject.

7. 6. The information processing apparatus according to claim 5, wherein the data generating means generates three-dimensional model data of the first subject and the second subject by excluding an area in which the first setting means and the second setting means are arranged from the scanned image.

8. The information processing apparatus according to claim 5 , wherein the second installation means includes a frame surrounding the second subject.

9. 9. The information processing device of claim 8, wherein the first subject and the frame body are positioned at a distance from each other so that the first subject is not blocked by the frame body when scanning the diagonal surface of the first subject on the side on which the frame body is positioned, or each has a size equal to or smaller than a predetermined size.

10. The information processing apparatus according to claim 8 , wherein the second installation means supports the second subject within the frame by a support member attached to the frame.

11. The information processing apparatus according to claim 2 , wherein the first subject and the second subject are placed so that their front faces face in the same direction.

12. The information processing apparatus according to claim 2 , wherein the attribute information includes attribute information of a type of the first subject and a type of the second subject.

13. The information processing device according to claim 12 , wherein the attribute information further includes information on a color used in rendering at least one of the first subject and the second subject.

14. The information processing device according to claim 12 , wherein the image generating means includes, in the image, a staged image relating to the second subject based on the attribute information.

15. The information processing apparatus according to claim 12 , wherein the scan control means controls a position and an angle of the scanner in accordance with attribute information of a type of the first subject.

16. The information processing apparatus according to claim 1 , wherein the rotation control means rotates the rotating platform clockwise and counterclockwise.

17. A computer program for causing a computer to function as the information processing device according to any one of claims 1 to 16.

Images