Information processing apparatus, and computer program

Abstract

To provide an information processing device and an information processing program configured to image a target and generate a video in which a three-dimensional model is arranged on a virtual space using three-dimensional model data of the imaged target, in accordance with attribute information, while associating the generated three-dimensional model data with the attribute information of the imaged target.SOLUTION: In an image processing system 10, an information processing device includes: rotation control means configured to control the motion of a turn table on which a subject is placed; scan control means which controls a position and an angle of the scanner and scans an outer appearance of the subject to generate a scan image; data generation means configured to generate three-dimensional model data from the scan image; video generation means configured to generate a video in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space so as to include an effect for each subject according to attribute information; and receiving means configured to receive an input of attribute information regarding the subject to be scanned.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and a computer program.

Background Art

[0002] Patent Document 1 describes a technique for generating a virtual space image visible from a virtual camera in a virtual space by mapping a texture generated from a group of captured images of an object captured from a plurality of imaging directions by an imaging unit to a primitive in the virtual space.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In generating a virtual space image as in the above technology, when generating a video in which a three-dimensional model is arranged on a virtual space using three-dimensional model data of a photographed object, it is expected that the three-dimensional model data to be generated is associated with the attribute information of the photographed object so that a video corresponding to the attribute information can be generated.

[0005] Therefore, an object of the present invention is to be able to generate a video corresponding to the attribute information by associating the three-dimensional model data to be generated with the attribute information of the photographed object when generating a video in which a three-dimensional model is arranged on a virtual space using three-dimensional model data of a photographed object.

Means for Solving the Problems

[0006] One form for solving the above problems is an information processing apparatus, rotation control means for controlling the operation of a turntable on which a subject is installed, Scan control means for controlling the position and angle of a scanner to scan the appearance of a subject installed on the turntable and generate a scan image; Data generation means for generating three-dimensional model data from the scan image; Video generation means for generating a video in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; Output means for outputting the video generated by the video generation means to a display device; Reception means for receiving an input of attribute information regarding the subject to be scanned; Comprising; The video generation means generates the video so as to include an effect for each subject according to the attribute information. Another form for solving the above problems is an information processing apparatus, Scan control means for controlling the position and angle of a scanner to scan the appearance of a subject installed on an installation table and generate a scan image; Data generation means for generating three-dimensional model data from the scan image; Video generation means for generating a video in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; Reception means for receiving an input of attribute information regarding the subject to be scanned; Comprising; The video generation means generates the video so as to include an effect for each subject according to the attribute information. Another form for solving the above problems is an information processing apparatus, Scan control means for controlling the position and angle of a scanner to scan the appearance of a subject installed on an installation table and generate a scan image; Data generation means for generating three-dimensional model data from the scan image; Video generation means for generating a video in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; Reception means for receiving an input of attribute information regarding the subject to be scanned based on a selection operation of the attribute information from the user; Comprising; The video generation means generates the video so as to include the effects for each subject according to the attribute information. and the subject includes a first subject and a second subject, the first subject is installed on a first installation means at the center of the installation table, the second subject is installed on a second installation means different from the first installation means near an end of the installation table, at least a part of the first subject is installed at a position higher than the second installation means by the first installation means 。

Advantages of the Invention

[0007] When photographing an object and generating a video in which a three-dimensional model is arranged in a virtual space using the three-dimensional model data of the photographed object, the three-dimensional model data can be generated in association with the attribute information of the photographed object, and a video according to the attribute information can be generated.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7

Modes for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of the features described in the embodiments are essential for the invention. Two or more of the plurality of features described in the embodiments may be arbitrarily combined. Also, the same or similar configurations are assigned the same reference numerals, and redundant descriptions are omitted. In each figure, the up-down, left-right, front-back directions with respect to the paper surface are used as the up-down, left-right, front-back directions of the components (or parts) in the present embodiment when explaining in the text.

[0010] First, the configuration of the image processing system corresponding to the present embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of an image processing system 10 corresponding to the present embodiment. The image processing system 10 is configured by connecting a scanner 110, a support arm 120, a model support device 130, a display device 140, a headset 150, etc. to an information processing device 100. Note that the system configuration is not limited to that shown in FIG. 1, and the information processing device 100 may be further connected to an external server, a cloud server, etc. via a network. At least a part of the processing of the embodiments described below can be executed by the external server or the like.

[0011] The information processing device 100 controls the operation of at least one of the scanner 110, the support arm 120, and the model support device 130, captures an article to be photographed from an arbitrary angle to generate a plurality of images, and generates three-dimensional model data (this data) from the plurality of images. When the article to be photographed has a configuration that can be separated into a plurality of component articles, photographing may be performed for each component article to generate three-dimensional model data, and the three-dimensional model data of the target article may be generated by integrating them.

[0012] The information processing apparatus 100 can also function as a video generation apparatus that generates a video to be displayed on a virtual space by using the generated three-dimensional model data as virtual space data. In the present embodiment, the subject to be photographed is an article such as an assemblable plastic model, an action figure (a figure having movable joints), a toy, a doll, etc., and these are collectively referred to as "models" hereinafter.

[0013] Next, the scanner 110 is a three-dimensional scanner device that photographs (scans) the three-dimensional shape of the model of the subject to be photographed according to the control of the information processing apparatus 100 and outputs the three-dimensional shape and color information of the imaging object. In the present embodiment, the scan signals output from the scanner 110 are collectively referred to as "scan images". For the scanner 110, for example, Space Spider manufactured by Artec can be used. In the present embodiment, for example, 3D scan data of the entire toy can be obtained by acquiring about 500 to 800 frames of scan images. Also, as the scanner 110, for example, a smartphone with a camera installed with an application for photographing a three-dimensional shape may be used.

[0014] The support arm 120 is a position and orientation control device that moves the scanner 110 to a predetermined photographing position and orientation according to the control of the information processing apparatus 100. The support arm 120 may be configured such that the photographing position and orientation can be manually changed and the changed position and orientation can be fixedly maintained, or may be configured to be controllable by the information processing apparatus 100. When the support arm 120 is configured to be controllable by the information processing apparatus 100, for example, xArm 7 manufactured by UFACTORY can be used. xArm7 consists of seven joints and can move in the same way as a human arm. Note that instead of the support arm 120, the scanner 110 may be manually positioned. Also, for photographing positions and orientations that cannot be covered by the support arm 120, the scanner 110 may be manually operated to perform scanning.

[0015] The model support device 130 is a support base (or an installation base for installing the model) that supports the posed model. The model support device 130 may be configured to be rotatable, for example, when the model is installed on the support base or at the tip of the support rod. In this embodiment, after positioning the scanner 110 at an arbitrary shooting position and shooting angle by the support arm 120, the model support device 130 is rotated to perform shooting. The model support device 130 is rotatable in both the clockwise and counterclockwise directions. Also, the model support device 130 can be stopped at an arbitrary rotation angle within the range of 0 degrees to 360 degrees. The model support device 130 can be stopped at a specific rotation angle, and shooting can also be performed by positioning it at an arbitrary shooting position and shooting angle within the movable range of the scanner 110. By performing this at a plurality of rotation angles, shooting positions, and shooting angles, an image of the entire model can be obtained. Here, by driving the support arm 120 and the model support device 130 synchronously, the shooting process can be performed more simply and with high accuracy. Also, instead of the model support device 130, the scanner 110 can be manually moved around the model and scanned from an arbitrary shooting position and shooting angle.

[0016] The display device 140 is a display device such as a liquid crystal display (LCD), and can display the processing result in the information processing device 100. Specifically, it can display the image acquired by the scanner 110, display the original data of the three-dimensional model data generated from the captured image, or display a video (VR video) reconstructed using the original data. The display device 140 can be provided with an operation unit 140A that receives operations from the user who is the observer of the displayed video. The user can operate the operation unit 140A to perform operation inputs according to the content of the video displayed on the display device 140.

[0017] The headset 150 is composed of a head-mounted display 150A and a controller 150B, which will be described later. In particular, it may be configured to provide a moving image corresponding to the posture and tilt of a user as an observer as a VR headset. A predetermined application is installed in the headset 150, and application data executed in the application can be downloaded from the information processing device 100 and executed. In the present embodiment, similar to the information processing device 100, the headset 150 can function as a video generation device that uses three-dimensional model data as virtual space data to generate a video to be displayed on the virtual space and provides the generated video as a VR video. Note that the application data includes display data for displaying a VR video.

[0018] Next, the hardware configuration of the information processing device 100 corresponding to the present embodiment will be described. FIG. 2(A) shows an example of the hardware configuration of the information processing device 100. The CPU 101 is a device that performs overall control of the information processing device 100 and calculates, processes, and manages data. For example, it controls the imaging timing and the number of imaging of the scanner 110, and controls the joints of the arm of the support arm 120 to arrange the scanner 110 at an arbitrary imaging position and imaging angle. For example, after the imaging position and imaging angle of the scanner 110 are determined, the model support device 130 can be rotated to perform an imaging operation by the scanner 110. Also, after the model support device 130 is stopped at a predetermined rotation angle, an imaging operation can be performed by the scanner 110 at an arbitrary imaging position and imaging angle.

[0019] Also, the CPU 101 can also function as an image processing unit that processes the image output from the scanner 110. Specifically, the CPU 101 can perform a process of generating a video for displaying a three-dimensional model in the virtual space by using the three-dimensional model data generated based on the scan signal obtained by the scanner 110 as virtual space data.

[0020] The RAM 102 is a volatile memory and is used as a main memory of the CPU 101 and a temporary storage area such as a work area. The ROM 103 is a non-volatile memory, and image data, other data, various programs for the CPU 101 to operate, etc. are stored in respective predetermined areas. The CPU 101 controls each part of the information processing apparatus 100 by using the RAM 102 as a work memory according to a program stored in the ROM 103, for example. Note that the program for the CPU 101 to operate is not limited to being stored in the ROM 103 and may be stored in the storage device 104.

[0021] The storage device 104 is constituted by a magnetic disk such as an HDD or a flash memory, for example. Application programs, an OS, control programs, related programs, game programs, etc. are stored in the storage device 104. The storage device 104 can read and write data based on the control of the CPU 101. The storage device 104 may be used instead of the RAM 102 or the ROM 103.

[0022] The communication device 105 is a communication interface for communicating with the scanner 110, the support arm 120, the model support device 130, the display device 140, and the headset 150 based on the control of the CPU 101. The communication device 105 may be further configured to be communicable with an external server or the like. The communication device 105 can include a wireless communication module, and the module can include a well-known circuit mechanism including an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identification module card, a memory, etc. Here, the communication between the information processing apparatus 100 and the scanner 110, the support arm 120, the model support device 130, the display device 140, and the headset 150 may be performed by wireless communication.

[0023] In addition, the communication device 105 can also include a wired communication module for wired connection. The wired communication module enables communication with other devices including the display device 140 via one or more external ports. It can also include various software components for processing data. The external ports are coupled to other devices directly via Ethernet, USB, IEEE 1394, etc., or indirectly via a network. Note that it can also be configured as an alternative to a hardware device by software that realizes functions equivalent to those of each of the above devices.

[0024] The operation unit 106 is composed of, for example, buttons, a keyboard, a touch panel, a controller, etc., and receives operation inputs from the user. The operation unit 106 may be shared with the operation unit 140A or may be independent of the operation unit 140A. For example, when the operation unit 140A is assumed to be an operation unit configured as a keyboard and a mouse, etc., it can be shared with the operation unit 106. On the other hand, when the operation unit 140A is assumed to be an operation unit configured as a touch panel, a controller, etc., it can be a separate operation unit from the operation unit 106.

[0025] The display control unit 107 functions as a control unit for displaying information on the display device 140 connected to the information processing device 100 and controls the operation of the display device 140. Some functions of the operation unit 106 may be provided in the display device 140. For example, the display device 140 may be configured as a device equipped with a touch panel such as a tablet terminal.

[0026] Next, the hardware configuration of the headset 150 corresponding to the present embodiment will be described. FIG. 2(B) shows an example of the hardware configuration of the headset 150. The headset 150 includes a head-mounted display (HMD) 150A and a controller 150B. The HMD 150A provides a right-eye image and a left-eye image to the user's left and right eyes respectively, and enables a virtual reality (VR) experience by making the user feel a stereoscopic effect due to the parallax between them. The controller 150B is provided in a housing independent of the HMD 150A. The controller 150B assumes a configuration in which two controllers are paired for the user to hold and operate with their left and right hands respectively, but a single controller may also be used.

[0027] The CPU 151 is a device that performs overall control of the headset 150 and calculation, processing, and management of data. For example, it can execute application data downloaded from the information processing device 100 and display a VR image on the display 157. Also, based on operations received via the controller 156 and information detected by the detection unit 158, it can switch the VR image itself to be displayed, switch the viewpoint within the VR space, or change the position within the VR space.

[0028] The RAM 152 is a volatile memory and is used as a temporary storage area such as the main memory and work area of the CPU 151. The ROM 153 is a non-volatile memory, and image data, other data, and various programs for the CPU 151 to operate are stored in respective predetermined areas. The CPU 151 controls each part of the headset 150 using the RAM 152 as a work memory according to, for example, programs stored in the ROM 153. Note that the programs for the CPU 151 to operate are not limited to those stored in the ROM 153 and may be stored in the storage device 154.

[0029] The storage device 154 is composed of a magnetic disk such as an HDD or a flash memory. The storage device 154 stores application programs, an OS, control programs, related programs, game programs, application data downloaded from the information processing device 100, display data, and the like. The storage device 154 can read and write data based on the control of the CPU 151. The storage device 154 may be used instead of the RAM 152 or the ROM 153.

[0030] The communication device 155 is a communication interface for communicating with the information processing device 100 and the controller 150B based on the control of the CPU 151. The communication device 155 includes a wireless communication module that enables wireless communication based on Bluetooth or WiFi (IEEE802.11). The headset 150 can be connected to the information processing device 100 by wireless communication and download display data for VR images. Also, it communicates with the controller 150B and receives information on the user's operation instructions for the controller 150B.

[0031] The display 156 is configured to provide the right-eye image and the left-eye image generated by the CPU 151 to the user's right eye and left eye, respectively. The detection unit 157 is a mechanism for detecting the line-of-sight directions of the user's left and right eyes and the inclination of the head-mounted display. The detection unit 157 includes, for example, a sensor for detecting the line-of-sight direction or the gaze direction of a user wearing the HMD 150A. It also includes a gyroscope, a magnetometer, an accelerometer, a global positioning system (GPS), a compass, etc., and can identify the position, posture, inclination, etc. of the HMD 150A according to the detection information of these. The detection unit 157 detects information for determining the user's gaze direction and actions in the HMD 150A and transmits it to the CPU 151. The CPU 151 determines the gaze direction and actions based on the received detection information and adjusts the images presented on the display 156 of the HMD 150A to match the determined gaze direction and actions.

[0032] The controller 150B is composed of, for example, a plurality of buttons, a cross key, etc., and accepts operation inputs such as selection operations and direction indication operations from the user. The controller 150B is wirelessly communication-connected to the HMD 150A via the communication device 155.

[0033] Next, with reference to FIG. 3, an example of a model and its accessories that are the subjects to be photographed in this embodiment will be described. The model 300 is a model having the appearance of a humanoid type (robot or human). The model can be assembled, for example, as a plastic model and painted. Alternatively, it may be a completed model like an action figure with movable joints. The model in FIG. 2 is merely an example for explanation, and the shape of the model is not limited to having the appearance of a humanoid type. It can be a model of any shape such as a general vehicle, a racing vehicle, a military vehicle, an aircraft, a ship, an animal, a virtual life form, etc. Needless to say, the article to be photographed is not limited to a model as long as it is an article whose three-dimensional shape can be photographed by the scanner 110.

[0034] The model 300 has components such as a head 301, a chest 302, a right arm 303, a left arm 304, a right torso 305, a left torso 306, a right leg 307, a left leg 308, a right foot 309, and a left foot 310, and these are combined to form it. At least a part of each of the individual parts 301 to 310 is supported so as to be rotatable (or swingable) with respect to the adjacent part. For example, the head 301 is supported so as to be rotatable with respect to the chest 302, and the right arm 303 and the left arm 304 are supported so as to be rotatable with respect to the chest 302. Since joint structures are provided for each part of the model 300 in this way, the model 300 can take any posture.

[0035] In addition, an accessory 311 is attached to the model 300. The accessory 311 is an accessory part assumed to be used together with the model 300, and in FIG. 3, it is a weapon having a rifle shape. That is, it is assumed that the model 300 holds the accessory 311 with both hands and uses it. The form of the weapon is not limited to the form of a rifle, and can include weapons in any form such as a sword (knife) shape, a shield (shield) shape, a bomb shape (e.g., a grenade), a bazooka shape, etc. Also, the accessory is not limited to a weapon as long as it is an accessory part that can be used together with the model 300, and it may be something like a decoration. For example, when the model 300 is a doll model of a specific character, it may be various items carried by the character (e.g., carry-on items, hats, bags, accessories, smartphones, and any other items that the character can carry).

[0036] Next, referring to FIG. 4A, an overall implementation example of the image processing system 10 in the present embodiment will be described. FIG. 4A is a diagram showing a generalized implementation example capable of generating the present data generated by photographing a model.

[0037] In FIG. 4A, inside the case 401, there are included an information processing device 100, a drive system for driving the support arm 402, a drive system for driving the turntable 406, and the like. Note that the support arm 402 can also be manually adjusted in the photographing direction and position. The surface of the case 401 has a flat structure so that a promotional poster can be attached.

[0038] The support arm 402 corresponds to the support arm 120, and can support and fix the position of the terminal 403 that functions as the scanner 110 according to the control of the information processing device 100 or manually. The support arm 402 can also operate to control the inclination of the terminal 403.

[0039] The terminal 403 is a touch panel terminal with a built-in camera that can be used as the scanner 110. For example, a smartphone, a tablet terminal, a digital camera, etc. can be used. Instead of these terminals, Space Spider manufactured by Artec can also be used. FIG. 4A is a diagram showing a generalized example of the system configuration, and the configuration can be adjusted according to the type of device used as the scanner 110. The terminal 403 can capture an image of the model 300 and transmit it to the information processing device 100. The ring light 404 is an illumination device used when the terminal 403 captures an image of the model 300. It can evenly illuminate the model 300 and make it difficult to produce shadows. In addition to the ring light 404, as an additional light source, a top light or auxiliary lights can be installed on the left and right sides and the lower side. Instead of the terminal 403, a three-dimensional scanner device as described above may be used.

[0040] The background sheet 405 is a background sheet for photography. For example, a white sheet can be used. The turntable 406 can carry the model 300 and rotate the model. The configuration including the turntable 406 and its drive system corresponds to the model support device 130. According to the control of the information processing device 100, the turntable 406 rotates clockwise and counterclockwise and stops. A plurality of predetermined markers 410 may be arranged on the turntable 406. The marker 410 can be used to adjust the orientation and position of the photographed model.

[0041] In FIG. 4A, the model is placed on a semi-transparent (transparent) base, but in addition, for example, a support device called "Action Base" (registered trademark) may also be used. The Action Base has a support column configured to be bent in a "く" shape on the pedestal, and the model can be attached to the tip of the support column. At this time, the above-described marker 410 may be arranged at the tip of the support column. The method of instructing the model can be changed according to the pose. For example, in the case of an upright posture, it can be placed on a transparent base and photographed. On the other hand, when it is desired to photograph up to the soles of the feet in a flying posture, the Action Base may be used. Note that the Action Base may be used to photograph the upright posture.

[0042] The display device 407 is a device corresponding to the display device 140 and may have a touch panel function. The user can perform a predetermined selection operation using the touch panel function. Also, operations can be received by a controller or a mouse (not shown) that can function as the operation unit 140A instead of the touch panel. The VR headset 408 consists of an HMD 408A and a controller 408B, and corresponds to the HMDs 150A and 150B, respectively. The user can perform operations while viewing VR images by wearing the HMD 408A on the head and holding the controller 408B in each of the left and right hands.

[0043] Next, with reference to FIG. 4B, a more specific configuration of the image processing system 10 when simultaneously scanning the model 300 and the accessory 311 corresponding to the present embodiment will be described. In FIG. 4B, the model 300 and the accessory 311 are placed on the turntable 406. The model 300 is supported by the Action Base 411 and is arranged substantially at the center of the turntable 406. The accessory 311 is supported by a support member 412A within the frame 412, is adjacent to the model 300, and is arranged near the end of the turntable 406. The indicating member 412A can be, for example, a clip-shaped member. Thereby, the accessory 411 can be sandwiched by the clip and supported within the frame 412.

[0044] The scanner device 413 is supported by the support arm 402. Here, it is assumed that Space Spider manufactured by Artec is used. Since the drive system for driving the support arm 402, the drive system for driving the turntable 406, etc. are the same as those described in FIG. 4A, the description thereof is omitted. In this embodiment, the turntable 406 can rotate in both clockwise and counterclockwise directions.

[0045] In this embodiment, the model 300 and the accessory 311 are placed together on the turntable 406 in the form shown in FIG. 4B, and are collectively imaged by the scanner device 413 to generate a scan image. The scan image will include both the model 300 and the accessory 311, but the three-dimensional model data is generated by separating them.

[0046] The frame 412 that supports the accessory 311 is configured as a frame that surrounds the accessory 311 and is used as an indicator for identifying the range where the accessory 311 is arranged. That is, a three-dimensional object existing inside the frame 412 is recognized as the accessory 311. The thickness of the frame 412 is desirably as thin as possible within the range where the strength is maintained so as not to block the line of sight of the scanner device 413 as much as possible. For example, it can be about 3 mm thick.

[0047] Also, the model 300 and the frame 412 are arranged to be separated from each other to ensure a certain distance. Depending on the rotation angle of the turntable 406, the frame 412 may be located in front of the model 300 with respect to the scanner device 413 and block the photographing of the model 300. For the portion thus blocked, it is necessary to complement it at another rotation angle. However, if the model 300 and the frame 412 are too close to each other, the complement accuracy will decrease. Therefore, a certain distance is ensured to ensure the complement accuracy. At this time, a certain distance can be ensured (or not getting too close) between the model 300 and the frame 412. For example, the sizes of the model 300 and the accessory 311 may be limited to sizes equal to or smaller than predetermined sizes respectively. For example, the size of the model 300 can be 15 cm or less in length, 15 cm or less in width, and 10 cm or less in depth, and the size of the accessory 311 can be 15 cm or less in length, 5 cm or less in width, and 2 cm or less in thickness.

[0048] Also, in this embodiment, the installation base 414 on which the action base 411 that supports the model 300 is installed and the installation base 415 on which the frame 412 that supports the accessory 311 is installed are configured such that the installation base 414 is thicker. That is, the model 300 is installed so as to be positioned relatively higher than the accessory 311. As an example, at least the upper body or the upper half of the model 300 can be installed so as to be positioned above the frame 412. Thereby, when scanning the model 300, the ratio shielded by the accessory 311 is reduced, and the success rate of scanning can be increased.

[0049] Next, with reference to FIG. 5, the cooperative operation between the turntable 406 and the support arm 120 during scanning corresponding to this embodiment will be described. FIG. 5 is a diagram showing the relationship between the rotation direction and rotation amount of the turntable 406 with respect to the scanner device 413, and the installation bases 414 and 415. In FIG. 5, the model 300 installed on the installation base 414 is arranged such that the direction described as the front is the front. The accessory 311 arranged on the installation base 415 can be the same.

[0050] Figure 5(A) shows the relationship between the scanner device 413 and the installation bases 414 and 415 at the start position of the scanning operation. The rotation angle of the turntable 406 at this time is set as the reference 0 degrees. When the turntable 406 rotates 90 degrees to the right (clockwise) from this state, it becomes the state shown in Figure 5(B). While transitioning from Figure 5(A) to Figure 5(B), the scanner device 413 is positioned on the side of the turntable 406 and scans the model 300 and the accessory 311. Then, when it reaches the state shown in Figure 5(B) where the front sides of the model 300 and the accessory 311 are positioned on the front side of the scanning device 413, the support arm 402 is operated to the limit of its movable range to scan the front, left and right side surfaces, top surface, and bottom surface of the model 300 and the accessory 311.

[0051] Next, from the state of Figure 5(B), the turntable 406 is rotated from the 90-degree position to the 270-degree position to the right. During this period as well, the model 300 and the accessory 311 are scanned. Then, when it reaches the state shown in Figure 5(C) where the back sides of the model 300 and the accessory 311 are positioned on the front side of the scanning device 413, the support arm 402 is operated to the limit of its movable range to scan the back, left and right side surfaces, top surface, and bottom surface of the model 300 and the accessory 311.

[0052] Next, from the state of Figure 5(C), the turntable 406 is further rotated from the 270-degree position to the 330-degree position to the right. During this period as well, the model 300 and the accessory 311 are scanned. Then, when it reaches the state shown in Figure 5(D) where the support arm 402 is operated to the limit of its movable range from the right diagonally behind the model 300 and the accessory 311, the model 300 and the accessory 311 are scanned. By this scan, it is possible to scan so as to complement the portions of the appearance of the model 300 that are shielded by the frame 412 and the accessory 311 (the surfaces of the model 300 on the side where the frame 412 and the accessory 311 are arranged). Here, the case of scanning from the right diagonally behind is described, but scanning may also be performed from the right diagonally in front.

[0053] Next, starting from the state of FIG. 5(D), the turntable 406 is rotated counterclockwise (leftward) from the 330-degree position to the 30-degree position. During this time, the scanning device 413 is arranged so as to have an overhead view from diagonally above the model 300 and the accessory 311, and scanning is performed. Then, the state shown in FIG. 5(E) is reached. Then, starting from the state of FIG. 5(E), the turntable 406 is rotated clockwise (rightward) from the 30-degree position to the 360-degree position. During this time, the scanning device 413 is arranged so as to have an upward view from diagonally below the model 300 and the accessory 311, and scanning is performed. Then, the state shown in FIG. 5(F) is reached.

[0054] In the scanner device 413 assumed to be used in this embodiment, scanning is performed at a position approximately in the middle (for example, the distance between the scanner device 413 and the model 300 is about 130 mm) with respect to the scanable range of the scanner device 413 (for example, a distance from 90 mm to 180 mm). In this case, when the accessory 311 passes between the scanner device 413 and the model 300, there is a possibility that a problem may occur in that the model 300 is shielded by the accessory 311 and the scan target is lost. However, as described above, by rotating the turntable 406 rightward and also rotating it leftward in reverse rotation to perform scanning, and by performing scanning so as to complement the shielding range, it is possible to surely acquire the three-dimensional model data of each of the model 300 and the accessory 311 without losing the scan target.

[0055] Next, referring to FIG. 6, an example of the processing executed by the image processing system 10 corresponding to this embodiment will be described. At least a part of the processing corresponding to the flowchart is realized by the CPU 101 of the information processing device 100 executing a program stored in the ROM 103 or the storage device 104.

[0056] First, in S601, the CPU 101 accepts user registration. It accepts the input of the user's name and contact information. For each user, a user identifier for uniquely identifying the individual user is assigned. The CPU 101 stores the input user information in the storage device 104 in association with the time when the input was received and the user identifier.

[0057] When the user registration is completed, the user sets his / her model on the model support device 130. The CPU 101 can determine whether the model 300 and the accessory 311 are set on the model support device 130 based on the scan image by the scanner 110. Alternatively, a switch that turns on when the model 300 and the accessory 311 are set on the model support device 130 may be arranged, and the CPU 101 may detect and judge the signal from the switch. Alternatively, a button for accepting an operation when the setting of the model 300 and the accessory 311 is completed may be displayed on the display device 140, and the CPU 101 can detect whether it has received an operation for the button operation. In S602, the CPU 101 detects that the model 300 and the accessory 311 are set on the model support device 130 by any of the above methods. In response to the detection, the process proceeds to S603.

[0058] In S603, a process of accepting the specification of the attributes of the model 300 and the accessory 311 is executed. Here, for example, a screen 700 as shown in FIG. 7 is displayed, and the selection of the attributes (types) of the model to be scanned, the attributes (types) of the accessories, the subject, and the color (production color) used for the production of the accessories can be accepted.

[0059] The screen 700 in FIG. 7 can be displayed on, for example, the display device 140 to receive operation inputs from the user. In the display of the screen 700, the user can select either real or deformed for the model attribute 710. Since the appearance characteristics of the model differ between real and deformed, the scanning method can also be selected according to the selected model attribute. For example, in the case of deformed, the range in which the scanner device 413 operates may be set narrower than in the case of real. Note that for the model attribute 710, it may be possible to further specify the scale value (1 / 144, 1 / 100, 1 / 60, etc.) of the model. Thereby, it becomes possible to further set the operating range of the scanner device 413 according to the scale of the model.

[0060] Next, regarding the accessory attribute 720, here assuming the accessory is a weapon, it can be selected from any of a rifle, a machine gun, and a sword. Note that what is shown in FIG. 7 as the weapon attribute is merely an example, and more detailed attribute specifications are possible. Also, in FIG. 7, only attack weapons are shown, but it may be possible to specify attributes of defensive weapons, shields, etc. Regarding the effect color 730, any of red, yellow, yellow - green, blue, and pink can be selected. What is shown in FIG. 7 as the effect color is merely an example, and colors other than those shown here can be specified.

[0061] A scan start button 740 is displayed on the screen 700. When the specification of the attributes from 710 to 730 above is completed, the user can instruct the start of scanning. When the start of scanning is instructed, the process proceeds to S604. Also, at this time, the attribute information input via the screen in FIG. 7 is stored in the storage device 104 in association with the user information.

[0062] In S604, as described with reference to FIG. 5, the scanner 110, the support arm 120, and the model support device 130 cooperate to execute a scanning process (imaging process) to generate a scanned image. Specifically, the CPU 101 controls the support arm 120 to move the scanner 110 to any of the registered imaging positions, and the scanner 110 can perform scanning while rotating the model support device 130 at the imaging position. Further, after the model support device 130 is stopped at a predetermined rotation angle, the imaging operation can be performed by the scanner 110 at an arbitrary imaging position and imaging angle. The captured scanned image is transmitted to the information processing device 100, and the CPU 101 stores it in association with a user identifier or the like in the table of the storage device 104.

[0063] In the subsequent S605, the CPU 101 executes post-scanning processing on the scanned image obtained by the scanning process, generates three-dimensional model data of the model 300 and the accessory 311, and generates application data (display data) for video display. In the post-scanning processing, an area for defining the range of unnecessary data in advance is created, and the surfaces included in the area where the data is not to be read are deleted. As a result, the surfaces other than the model 300 and the accessory 311 can be substantially removed. The area of the unnecessary data range includes the areas where the action base 411, the installation base 414, the frame 412, the support member 412A, the installation base 415, etc. are located.

[0064] The generated application data can reproduce a video when executed in a specific application in the information processing device 100. Further, a VR video can be reproduced when executed in the specific application on the head-mounted display 150A. The generated three-dimensional model data and display data are stored in the storage device 104 in association with the user information together with the attribute information received in S603.

[0065] In subsequent S606, video display processing is executed. When the video display processing is performed in the information processing apparatus 100, the CPU 101 executes a specific application corresponding to the display data and displays a video on the display device 140 via the display control unit 107. Also, in S606, the CPU 101 controls the communication device 105 to transmit the display data to the HMD 150A, and the CPU 151 of the HMD 150A executes a process of displaying a VR video by executing the received display data in the corresponding application, thereby performing the video display processing.

[0066] During video display, the effects on the subject and accessories can be switched according to the attribute information. For example, when the accessory 311 has a rifle attribute, a single shot is fired when using the accessory 311, while in the case of a machine gun attribute, continuous firing is possible when using the accessory 311. Also, the display of the bullets at that time is performed in the color specified by the effect color.

[0067] In FIG. 6, the case where the designation of the attribute information is received before starting the scan process has been described, but the embodiment is not limited to this, and the designation of the attribute information may be received after the scan process of S604 or the post-scan process of S605.

[0068] As described above, in the present embodiment, when generating display data for displaying a video by arranging the three-dimensional model data generated from the image obtained by photographing the model with a scanner in the virtual space, the accessory is photographed simultaneously with the model to generate the three-dimensional model data, and it can be used in the virtual space together with the three-dimensional model data of the model.

[0069] In addition, in this embodiment, since it is possible to receive the designation of attribute information for accessories, when using the three-dimensional model of the accessory in the virtual space, it is possible to perform an effect according to the received attribute. As a result, the user can reflect their own image in the effect related to the accessory, so that they can avoid feeling uncomfortable. Furthermore, since it is possible to receive the designation of the type of model as attribute information, by controlling the scanning range according to the type of model, it is possible to obtain a more accurate scanned image.

[0070] In the above-described embodiment, the description has been made assuming a weapon as an accessory, but the type of accessory is not limited to a weapon. For example, when the model is a specific humanoid character, the accessory may be clothing, accessories, etc. worn by the character. In addition, it is also possible to make the model appear in events, concerts, sports, online meetings, etc. held in the virtual space. Furthermore, the technology of this embodiment can also be applied to video technologies that fuse the real world and the virtual world, such as cross-reality (XR), and make things that do not exist in the real space perceptible.

[0071] In this way, in this embodiment, it is possible to generate three-dimensional model data from an image obtained by photographing the appearance of a model and its accessories, and view a video in which the three-dimensional model is arranged in the virtual space. There are models and accessories that the user has painted and finished as their own works, such as assembled plastic models. Since their individuality can be reflected in videos and character expressions in the virtual space, the likability can be significantly improved.

[0072] <Summary of the Embodiment> The above embodiment discloses at least the following information processing apparatus and computer program. (1) An information processing apparatus, Rotation control means for controlling the operation of a turntable on which a subject is installed, Scan control means for controlling the position and angle of a scanner to scan the appearance of a subject placed on the turntable and generate a scan image; Data generation means for generating three-dimensional model data from the scan image; Video generation means for generating a video in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; Reception means for receiving input of attribute information regarding the subject to be scanned; Comprising; The video generation means generates the video so as to include an effect for each subject according to the attribute information, an information processing apparatus. (2) The subject includes a first subject and a second subject, The first subject is arranged at the center of the turntable, The second subject is arranged near an end of the turntable, The information processing apparatus according to (1). (3) The second subject is an accessory of the first subject, the information processing apparatus according to (2). (4) The first subject is a doll-shaped model, and the second subject is an accessory part of the model, the information processing apparatus according to (3). (5) The first subject is installed on a first installation means, The second subject is installed on a second installation means different from the first installation means, At least a part of the first subject is installed at a position higher than the second installation means by the first installation means, the information processing apparatus according to any one of (2) to (4). (6) At least a part of the first subject includes half of the first subject, the information processing apparatus according to (5). (7) The data generation means excludes an area where the first installation means and the second installation means are arranged from the scan image and generates three-dimensional model data of the first subject and the second subject, the information processing apparatus according to (5) or (6). (8) The second installation means is the information processing apparatus according to any one of (5) to (7), including a frame surrounding the second subject. (9) When scanning an oblique surface on the side where the frame of the first subject is disposed, the first subject and the frame are spaced apart so that the first subject is not shielded by the frame, or each has a size equal to or less than a predetermined size. The information processing apparatus according to (8). (10) The second installation means is the information processing apparatus according to (8) or (9), which supports the second subject within the frame by a support member attached to the frame. (11) The information processing apparatus according to any one of (2) to (10), wherein the front surfaces of the first subject and the second subject are arranged in the same direction. (12) The attribute information includes the attribute information of the type of the first subject and the type of the second subject. The information processing apparatus according to any one of (2) to (11). (13) The attribute information further includes information on a color used for at least one of the effects of the first subject and the second subject. The information processing apparatus according to (12). (14) The video generation means includes, in the video, an effect video related to the second subject based on the attribute information. The information processing apparatus according to (12) or (13). (15) The scan control means controls the position and angle of the scanner according to the attribute information of the type of the first subject. The information processing apparatus according to any one of (12) to (14). (16) The rotation control means rotates the turntable clockwise and counterclockwise. The information processing apparatus according to any one of (1) to (15). (17) A computer program for causing a computer to function as the information processing apparatus according to any one of (1) to (16).

[0073] The invention is not limited to the above embodiments, and various modifications and changes are possible within the scope of the gist of the invention.

Claims

1. An information processing apparatus, comprising: scan control means for controlling the position and angle of a scanner to scan the appearance of a subject placed on a mounting table and generate a scan image; data generation means for generating three-dimensional model data from the scan image; video generation means for generating a video in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; reception means for receiving input of attribute information regarding the subject to be scanned based on a selection operation of the attribute information by a user; wherein the video generation means generates the video so as to include an effect for each subject according to the attribute information; the subject includes a first subject and a second subject; the first subject is placed on first mounting means at the center of the mounting table; the second subject is placed on second mounting means different from the first mounting means near an end of the mounting table; the information processing apparatus, wherein at least a part of the first subject is placed at a position higher than the second mounting means by the first mounting means.

2. The information processing apparatus according to claim 1, wherein the second subject is an accessory of the first subject.

3. The information processing apparatus according to claim 2, wherein the first subject is a doll-type model and the second subject is an accessory part of the model.

4. The information processing apparatus according to claim 1, wherein at least a part of the first subject includes half of the first subject.

5. The information processing apparatus according to claim 1, wherein the data generation means generates three-dimensional model data of the first subject and the second subject by excluding from the scan image an area where the first mounting means and the second mounting means are arranged.

6. The information processing apparatus according to claim 1, wherein the second mounting means includes a frame surrounding the second subject.

7. The information processing apparatus according to claim 6, wherein the first subject and the frame are spaced apart so that the first subject is not shielded by the frame when scanning an oblique surface of the first subject on the side where the frame is arranged, or each has a size equal to or smaller than a predetermined size.

8. The information processing apparatus according to claim 6, wherein the second mounting means supports the second subject within the frame by a support member attached to the frame.

9. The information processing apparatus according to claim 1, wherein the first subject and the second subject are installed such that their fronts face the same direction.

10. The information processing apparatus according to claim 1, wherein the attribute information includes attribute information on the type of the first subject and the type of the second subject.

11. The information processing apparatus according to claim 10, wherein the attribute information further includes information on a color used for an effect of at least one of the first subject and the second subject.

12. The information processing apparatus according to claim 10, wherein the video generation means includes, in the video, an effect video related to the second subject based on the attribute information.

13. The information processing apparatus according to claim 10, wherein the scan control means controls the position and angle of the scanner according to the attribute information on the type of the first subject.

14. The information processing apparatus further includes operation control means for controlling the operation of the mounting table, The information processing apparatus according to claim 1, wherein the operation control means rotates the mounting table clockwise and counterclockwise.

15. A computer program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 14.

Images